Farshad's Note

Mastering Claude Code: Chrome Browser Automation

Thu, 22 Jan 2026 00:00:00 GMT

## The Browser Bridge Throughout this series, we've transformed Claude Code into a powerful engineering partner through configuration, commands, agents, plugins, and memory. But there's been one gap: the browser. Modern development constantly switches between terminal and browser—running tests, checking UI, debugging console errors, verifying deployments. What if Claude could see what you see in the browser? With the **Claude in Chrome** extension, it can. ## What Browser Automation Enables This integration unlocks workflows that were previously impossible: | Capability | Example | |------------|---------| | **Live debugging** | Claude reads console errors and DOM state, then fixes code | | **Design verification** | Build UI from mockups, verify directly in browser | | **Web app testing** | Test form validation, check visual regressions, verify user flows | | **Authenticated access** | Interact with Google Docs, Gmail, Notion without API setup | | **Data extraction** | Pull structured information from web pages | | **Task automation** | Automate form filling, data entry, multi-site workflows | | **Session recording** | Record browser interactions as GIFs for documentation | The key insight: Claude shares your browser's login state. No API keys, no OAuth setup—if you're logged in, Claude can interact with the page. ## Prerequisites Before starting, ensure you have: - **Google Chrome** browser (not Brave, Arc, or other Chromium variants—beta limitation) - **Claude in Chrome extension** v1.0.36+ from the Chrome Web Store - **Claude Code CLI** v2.0.73+ (`claude update` to upgrade) - **Paid Claude plan** (Pro, Team, or Enterprise) ## Setup ### Step 1: Update Claude Code ```bash claude update ``` Verify your version: ```bash claude --version # Should show v2.0.73 or higher ``` ### Step 2: Install the Chrome Extension Install the [Claude in Chrome extension](https://chromewebstore.google.com/detail/claude/fcoeoabgfenejglbffodgkkbkcdhcgfn) from the Chrome Web Store. ### Step 3: Launch with Chrome Flag ```bash claude --chrome ``` The first launch installs a native messaging host that enables communication between Claude Code and Chrome. ### Step 4: Verify Connection Run the `/chrome` command inside Claude Code: ```bash > /chrome ``` This shows connection status and lets you manage settings. If connected, you'll see the browser tools available. ## Architecture Understanding how the pieces connect helps troubleshoot issues: ```mermaid flowchart LR subgraph Terminal CC[Claude Code CLI] end subgraph "Native Messaging" NM[Native Host] end subgraph Chrome EXT[Claude Extension] TAB[Browser Tab] end CC <--> NM NM <--> EXT EXT <--> TAB ``` Key points: 1. Claude Code communicates via Chrome's **Native Messaging API** 2. The extension receives commands and executes them in the browser 3. A **visible browser window is required**—no headless mode 4. Claude reads page content, console output, and performs actions 5. Login state is shared—no re-authentication needed ## Available Tools Run `/mcp` and select `claude-in-chrome` to see all available tools: | Tool Category | Capabilities | |---------------|-------------| | **Navigation** | Go to URLs, forward/back in history | | **Interaction** | Click, type, scroll, hover, drag | | **Forms** | Fill inputs, select options, submit | | **Reading** | Extract text, read DOM, get accessibility tree | | **Console** | Read console logs, errors, warnings | | **Network** | Monitor XHR/Fetch requests | | **Tabs** | Create, manage, switch between tabs | | **Recording** | Capture interactions as animated GIFs | | **Screenshots** | Capture page state for analysis | ## Practical Workflows ### Debug a Local Web Application The most common workflow—something's broken in your UI: ``` I just updated the login form validation. Open localhost:3000, try submitting the form with invalid data, and check if the error messages appear correctly. ``` Claude will: 1. Navigate to your local dev server 2. Interact with the form 3. Read the resulting DOM and console 4. Report what it finds 5. Suggest or implement fixes ### Console Error Investigation When you see errors but don't know where they originate: ``` Open the dashboard page and check the console for any errors when the page loads. Tell me what's failing. ``` **Best practice:** Ask Claude to filter for specific patterns rather than dumping all console output: ``` Check the console for errors containing "API" or "fetch" ``` ### Automated Form Filling For repetitive data entry tasks: ``` I have a spreadsheet of customer contacts in contacts.csv. For each row, go to our CRM at crm.example.com, click "Add Contact", and fill in the name, email, and phone fields. ``` Claude iterates through your data, performing the same actions for each entry. ### Data Extraction Pull structured data from web pages: ``` Go to the product listings page and extract the name, price, and availability for each item. Save the results as a CSV file. ``` ### Working with Authenticated Services Since Claude shares your login state: ``` Draft a project update based on our recent commits and add it to my Google Doc at docs.google.com/document/d/abc123 ``` No API setup required—if you're logged into Google Docs, Claude can edit the document. ### Recording Demo GIFs Document workflows for team members or users: ``` Record a GIF showing how to complete the checkout flow, from adding an item to the cart through to the confirmation page. ``` The GIF recorder captures actions with visual indicators for clicks and scrolling. ### Quick Verification Fast checks during development: ``` Go to code.claude.com/docs, click on the search box, type "hooks", and tell me what results appear ``` ## Complete Workflow Example Let's walk through a real debugging session using browser automation with the tools from this series: **Scenario:** Users report that the signup form sometimes fails silently. ```bash # Start Claude Code with Chrome enabled claude --chrome ``` **Step 1: Investigate** ``` Open localhost:3000/signup and try submitting with an invalid email. Check the console and network tab for errors. ``` Claude navigates, interacts, and reports: - Console shows: "Uncaught TypeError: Cannot read property 'message' of undefined" - Network shows: 400 response from /api/signup with error details **Step 2: Find the Bug** ``` Based on that error, find where we're handling the signup API response incorrectly. ``` Claude searches the codebase (using skills from Part 2) and identifies the issue in `src/hooks/useSignup.ts`. **Step 3: Fix and Verify** ``` Fix the error handling. Then open the signup page again and verify the error message now displays correctly. ``` Claude: 1. Edits the file 2. Waits for hot reload 3. Navigates to signup 4. Submits invalid data 5. Confirms the error message appears **Step 4: Document** ``` Record a GIF showing the improved error handling for the PR. ``` Claude records the flow for documentation. ## Integration with Series Components Browser automation becomes more powerful combined with earlier configurations: | Component | Integration | |-----------|-------------| | **CLAUDE.md** | Add browser testing commands to project instructions | | **Commands** | Create `/test-ui` command that runs browser verification | | **Agents** | Use `principal-frontend-architect` for UI review in browser | | **Memory** | Store common test flows in project memory | Example command addition to `~/.claude/commands/`: ```markdown --- description: Test UI components in browser --- # /test-ui 1. Start the dev server if not running 2. Open the specified component/page in Chrome 3. Verify visual appearance matches expectations 4. Check console for errors 5. Test key interactions 6. Report findings ``` ## Best Practices ### 1. Modal Dialogs Block Everything JavaScript `alert()`, `confirm()`, and `prompt()` dialogs prevent Claude from receiving further commands. If your app uses these: ``` Avoid clicking buttons that trigger alerts, or warn me first so I can dismiss them manually. ``` If you get stuck, dismiss the dialog manually in the browser and tell Claude to continue. ### 2. Use Fresh Tabs Claude creates new tabs for each session. If something becomes unresponsive: ``` Create a new tab and navigate there instead ``` ### 3. Filter Console Output Console logs can be verbose. Always specify what you're looking for: ``` # Good Check console for errors containing "auth" # Bad Show me all console output ``` ### 4. Verify Before Automating For multi-step automations, verify the first iteration works: ``` First, show me the steps you'd take for the first contact. Don't actually do it yet. ``` Then proceed: ``` That looks right. Go ahead with all entries. ``` ### 5. Handle Dynamic Content For SPAs with dynamic loading: ``` Wait for the dashboard data to load before reading the values ``` Claude can wait for elements to appear before interacting. ## Troubleshooting ### Extension Not Detected | Check | Solution | |-------|----------| | Extension version | Verify v1.0.36+ in chrome://extensions | | Claude Code version | Run `claude --version`, update if < 2.0.73 | | Chrome running | Ensure Chrome is open (not just Chromium) | | Connection | Run `/chrome` → "Reconnect extension" | | Restart | Close both Claude Code and Chrome, restart both | ### Browser Not Responding | Symptom | Solution | |---------|----------| | No response to commands | Check for modal dialogs (alert/confirm/prompt) | | Tab seems stuck | Ask Claude to create a new tab | | Extension unresponsive | Disable/re-enable extension in chrome://extensions | ### Permission Errors The extension has site-level permissions. If Claude can't interact with a page: 1. Open extension settings in Chrome 2. Check which sites are allowed 3. Add the necessary site permissions ### First-Time Setup Issues On first use, Claude Code installs a native messaging host. If you see permission errors: 1. Restart Chrome completely 2. Run `claude --chrome` again 3. Check that the extension shows as connected ## Enabling by Default If you frequently use browser automation: ```bash > /chrome # Select "Enable by default" ``` **Note:** This increases context usage since browser tools are always loaded, even when not needed. ## Limitations Current beta limitations to be aware of: | Limitation | Details | |------------|---------| | Chrome only | Brave, Arc, and other Chromium browsers not supported | | WSL not supported | Windows Subsystem for Linux doesn't work yet | | Visible window required | No headless mode—browser must be visible | | Single browser | Can't control multiple Chrome profiles simultaneously | ## Security Considerations Browser automation has access to your authenticated sessions. Keep in mind: - Claude can see pages you're logged into - Don't use on sensitive banking or financial pages - Be cautious with automation on pages containing personal data - The extension respects site permissions you configure ## Conclusion Browser automation completes the Claude Code toolkit. Combined with everything we've built throughout this series: 1. **[Foundation](/2026/01/mastering-claude-code-foundation-and-philosophy/)** established the ground rules 2. **[Commands](/2026/01/mastering-claude-code-custom-commands-and-workflow/)** encoded reusable workflows 3. **[Agents & Skills](/2026/01/mastering-claude-code-specialized-agents-and-professional-skills/)** provided specialized expertise 4. **[MCP & Plugins](/2026/01/mastering-claude-code-mcp-connectors-plugins-and-complete-integration/)** connected external services 5. **[Memory](/2026/01/mastering-claude-code-memory-management-and-context-persistence/)** persisted context across sessions 6. **Browser Automation** (this post) bridges terminal and browser You now have a complete system where Claude can: - Read and write code in your codebase - Execute terminal commands - Access external documentation and services - Remember context across sessions - **See and interact with your browser** The gap between "AI assistant" and "AI pair programmer" continues to narrow. With browser automation, Claude doesn't just write code—it can verify that the code works, debug issues in real-time, and document the results. --- *This concludes the Mastering Claude Code series.* **Series Navigation:** - Part 1: [Foundation and Philosophy](/2026/01/mastering-claude-code-foundation-and-philosophy/) - Part 2: [Custom Commands and Workflow](/2026/01/mastering-claude-code-custom-commands-and-workflow/) - Part 3: [Specialized Agents and Professional Skills](/2026/01/mastering-claude-code-specialized-agents-and-professional-skills/) - Part 4: [MCP Connectors, Plugins, and Integration](/2026/01/mastering-claude-code-mcp-connectors-plugins-and-complete-integration/) - Part 5: [Memory Management and Context Persistence](/2026/01/mastering-claude-code-memory-management-and-context-persistence/) - Part 6: Chrome Browser Automation (this post)

Mastering Claude Code: Memory Management and Context Persistence

Wed, 21 Jan 2026 00:00:00 GMT

## The Memory Hierarchy In Part 4, we connected Claude Code to external services through MCP connectors and plugins. Now we'll explore the foundation that ties everything together: Claude Code's complete memory system—a hierarchical structure that enables context persistence across sessions, teams, and organizations. Memory in Claude Code isn't just configuration. It's **persistent context** that shapes how Claude understands your projects, preferences, and workflows. ## Memory Types and Locations Claude Code supports five levels of memory, each serving a distinct purpose: | Memory Type | Location | Purpose | Shared With | |-------------|----------|---------|-------------| | **Managed policy** | `/Library/Application Support/ClaudeCode/CLAUDE.md` (macOS) | Organization-wide instructions | All organization users | | **Project memory** | `./CLAUDE.md` or `./.claude/CLAUDE.md` | Team-shared project instructions | Team via source control | | **Project rules** | `./.claude/rules/*.md` | Modular, topic-specific instructions | Team via source control | | **User memory** | `~/.claude/CLAUDE.md` | Personal preferences for all projects | Just you (all projects) | | **Project local** | `./CLAUDE.local.md` | Personal project-specific preferences | Just you (this project) | **Priority Order:** Higher levels take precedence. Organization policies override everything, while local preferences are most specific. ## How Memory Discovery Works When Claude Code launches, it discovers memories through a recursive process: 1. Starts in the current working directory 2. Recursively checks parent directories up to (but not including) root 3. Loads any `CLAUDE.md` or `CLAUDE.local.md` files found 4. Discovers nested `CLAUDE.md` files in subtrees (loaded when those files are accessed) Example: Running Claude in `myproject/src/components/` discovers: - `myproject/CLAUDE.md` - `myproject/src/CLAUDE.md` (if it exists) - `myproject/src/components/CLAUDE.md` (if it exists) - `~/.claude/CLAUDE.md` This allows you to have general project instructions at the root and more specific instructions in subdirectories. ```mermaid flowchart TB subgraph "Memory Priority (Highest to Lowest)" direction TB ORG["🏢 Organization Policy
/Library/.../ClaudeCode/CLAUDE.md"] PROJ["📁 Project Memory
./CLAUDE.md"] RULES["📋 Project Rules
.claude/rules/*.md"] USER["👤 User Memory
~/.claude/CLAUDE.md"] LOCAL["🔒 Project Local
./CLAUDE.local.md"] end subgraph "Discovery Flow" direction LR CWD["Current Directory"] --> PARENT["Parent Directories"] PARENT --> HOME["Home Directory"] HOME --> SYSTEM["System Paths"] end ORG --> PROJ --> RULES --> USER --> LOCAL style ORG fill:#ffcdd2 style PROJ fill:#fff9c4 style RULES fill:#c8e6c9 style USER fill:#bbdefb style LOCAL fill:#e1bee7 ``` ## Setting Up Project Memory Initialize project memory with a single command: ```bash > /init ``` This creates a bootstrapped `CLAUDE.md` tailored to your codebase. Here's what to include: ```markdown # Project: My Application ## Commands - `npm run dev` - Start development server - `npm run test` - Run test suite - `npm run build` - Production build ## Architecture - React 18 with TypeScript - State management: Zustand - Styling: Tailwind CSS ## Conventions - Use functional components with hooks - Prefer named exports - Tests live next to source files (*.test.ts) ## Important Patterns - All API calls go through `src/lib/api.ts` - Authentication state in `src/stores/auth.ts` ``` ## Modular Rules with `.claude/rules/` For larger projects, modular rules provide organization: ``` your-project/ ├── .claude/ │ ├── CLAUDE.md # Main project instructions │ └── rules/ │ ├── code-style.md # Code style guidelines │ ├── testing.md # Testing conventions │ ├── security.md # Security requirements │ └── api/ │ ├── rest.md # REST API conventions │ └── graphql.md # GraphQL conventions ``` All `.md` files in `.claude/rules/` are automatically loaded with project-level priority. ### Path-Specific Rules Apply rules only to specific files using YAML frontmatter: ```markdown --- paths: - "src/api/**/*.ts" --- # API Development Rules - All API endpoints must include input validation - Use the standard error response format - Include OpenAPI documentation comments ``` Without a `paths` field, rules apply to all files unconditionally. ### Glob Patterns Supported patterns for path-specific rules: | Pattern | Matches | |---------|---------| | `**/*.ts` | All TypeScript files in any directory | | `src/**/*` | All files under `src/` | | `*.md` | Markdown files in project root only | | `src/components/*.tsx` | React components in specific directory | Multiple patterns and brace expansion: ```markdown --- paths: - "src/**/*.{ts,tsx}" - "{src,lib}/**/*.ts" - "tests/**/*.test.ts" --- ``` ### Organizing Rules with Subdirectories ``` .claude/rules/ ├── frontend/ │ ├── react.md │ └── styles.md ├── backend/ │ ├── api.md │ └── database.md └── general.md ``` ### Sharing Rules with Symlinks Link to shared rule sets across projects: ```bash # Link a shared rules directory ln -s ~/shared-claude-rules .claude/rules/shared # Link a specific rule file ln -s ~/company-standards/security.md .claude/rules/security.md ``` Circular symlinks are detected and handled gracefully. ## CLAUDE.md Imports Import additional files using `@path/to/import` syntax: ```markdown See @README for project overview and @package.json for npm commands. # Additional Instructions - Git workflow: @docs/git-instructions.md - API docs: @docs/api-reference.md ``` Supported paths: - Relative: `@docs/git-instructions.md` - Absolute: `@~/.claude/my-project-instructions.md` **Limitations:** - Maximum import depth: 5 hops - Imports not evaluated inside markdown code spans/blocks - Recursive imports supported ## User-Level Memory Personal preferences in `~/.claude/CLAUDE.md` apply to all projects: ```markdown # My Preferences ## Workflow - Always create a plan before implementation - Summarize results after completion - Don't commit unless asked directly ## Code Style - Use functional programming patterns where appropriate - Prefer immutable data structures - Use descriptive variable names ## Communication - Be concise in explanations - Include code examples - Reference file:line when discussing code ``` ### User-Level Rules Create personal rules in `~/.claude/rules/` that apply everywhere: ``` ~/.claude/rules/ ├── preferences.md # Personal coding preferences ├── workflows.md # Preferred workflows └── security.md # Personal security checklist ``` User-level rules load before project rules, so project rules take priority when there's a conflict. ## Project-Local Memory For personal project-specific preferences that shouldn't be shared: ```markdown # CLAUDE.local.md ## My Local Environment - Development server runs on port 3001 (not default 3000) - Using local PostgreSQL on port 5433 - Test database: myproject_test_local ## Personal Notes - Focus on the checkout module this sprint - Remember: the legacy API is being deprecated ``` `CLAUDE.local.md` files are automatically added to `.gitignore`. ## Managing Memory During Sessions ### View Loaded Memories ```bash > /memory ``` Shows all memory files currently loaded by Claude Code. ### Edit Memory Files ```bash > /memory ``` Opens memory files in your system editor for extensive additions or reorganization. ## Organization-Wide Policies For teams, deploy centrally managed policies: 1. Create the managed memory file at the **Managed policy** location: - macOS: `/Library/Application Support/ClaudeCode/CLAUDE.md` - Linux: `/etc/claude-code/CLAUDE.md` - Windows: `C:\Program Files\ClaudeCode\CLAUDE.md` 2. Deploy via configuration management (MDM, Ansible, Group Policy) 3. All developers automatically receive consistent instructions Example organization policy: ```markdown # Organization Standards ## Security Requirements - Never commit secrets or API keys - Use environment variables for configuration - Follow OWASP security guidelines ## Code Review - All changes require PR review - Include tests for new functionality - Update documentation when APIs change ## Compliance - Log all data access operations - Respect data retention policies - Follow GDPR requirements for user data ``` ## Best Practices ### Writing Effective Memory **Be specific:** ```markdown # Good - Use 2-space indentation - Maximum line length: 100 characters - Use camelCase for variables, PascalCase for components # Vague - Format code properly - Use good naming ``` **Use structure:** ```markdown ## API Development ### Endpoints - Use RESTful conventions - Version all APIs (/api/v1/...) ### Error Handling - Return appropriate HTTP status codes - Include error details in response body ``` **Keep rules focused:** ```markdown # testing.md - focused on one topic --- paths: - "**/*.test.ts" - "**/*.spec.ts" --- # Testing Standards - Use describe/it blocks for organization - Mock external dependencies - Aim for 80% coverage on critical paths ``` ### Organization Strategies | Project Size | Strategy | |--------------|----------| | **Small** | Single `CLAUDE.md` at root | | **Medium** | `CLAUDE.md` + 2-3 rule files | | **Large** | Full `.claude/rules/` with subdirectories | | **Monorepo** | Per-package CLAUDE.md + shared rules via symlinks | ### Review and Maintenance - **Review periodically** - Update memories as your project evolves - **Remove obsolete rules** - Outdated instructions cause confusion - **Test path patterns** - Verify rules apply to intended files - **Document rule purpose** - Future you will thank present you ## Memory vs. Commands vs. Agents Understanding when to use each: | Use Case | Solution | |----------|----------| | Project-specific context | Memory (CLAUDE.md) | | Personal preferences | User memory (~/.claude/CLAUDE.md) | | Reusable workflows | Commands (~/.claude/commands/) | | Specialized expertise | Agents (~/.claude/agents/) | | Persistent file-type rules | Rules (.claude/rules/) | Memory provides **context**. Commands provide **actions**. Agents provide **expertise**. ## Conclusion Claude Code's memory system is the connective tissue that makes everything else work. By understanding the five-level hierarchy—from organization policies to project-local preferences—you can build configurations that scale from personal projects to enterprise deployments. Key takeaways: - **Memory is hierarchical**: Organization → Project → Rules → User → Local - **Discovery is recursive**: Claude finds memories from current directory up to home - **Rules are modular**: Path-specific rules apply only to matching files - **Imports extend reach**: Reference external files with `@path/to/file` syntax Well-structured memory reduces the context you need to provide in every conversation. Your standards, conventions, and project knowledge persist automatically. ## Complete Memory Structure Here's the complete memory hierarchy for a well-configured setup: ``` /Library/Application Support/ClaudeCode/ └── CLAUDE.md # Organization policy ~/.claude/ ├── CLAUDE.md # Personal global preferences └── rules/ ├── preferences.md # Personal coding style └── workflows.md # Personal workflows your-project/ ├── CLAUDE.md # Project instructions (shared) ├── CLAUDE.local.md # Personal project prefs (not shared) └── .claude/ ├── CLAUDE.md # Alternative location └── rules/ ├── code-style.md # Team code standards ├── testing.md # Testing conventions ├── security.md # Security requirements └── frontend/ ├── react.md # React-specific rules └── styles.md # Styling conventions ``` ## What's Next With memory, commands, agents, and plugins in place, there's one frontier remaining: the browser. In Part 6, we'll explore **Chrome browser automation**: - **Live debugging** - Claude reads console errors and fixes code in real-time - **Design verification** - Build UI and verify it directly in the browser - **Web app testing** - Automated form validation and user flow testing - **Authenticated workflows** - Interact with Google Docs, Gmail, Notion without API setup - **Session recording** - Capture browser interactions as GIFs --- *Next in series: [Chrome Browser Automation](/2026/01/mastering-claude-code-chrome-browser-automation/) - Bridging terminal and browser*

Mastering Claude Code: MCP Connectors, Plugins, and Complete Integration

Tue, 20 Jan 2026 00:00:00 GMT

## Claude Code as Integration Hub Throughout this series, we've built: 1. [Foundation](/2026/01/mastering-claude-code-foundation-and-philosophy/) - CLAUDE.md files and terminal customization 2. [Commands](/2026/01/mastering-claude-code-custom-commands-and-workflow/) - 9 slash commands encoding best practices 3. [Agents & Skills](/2026/01/mastering-claude-code-specialized-agents-and-professional-skills/) - 10 agents and 19+ skills Now we connect Claude Code to your entire toolchain through the **Model Context Protocol (MCP)** and plugins. ## MCP Connectors MCP allows Claude Code to interact with external services through standardized protocols. Here's my current setup: ### Active Connectors | Connector | Purpose | |-----------|---------| | **Context7** | Up-to-date library documentation | | **Ralph Loop** | Automated iteration workflows | Context7 is particularly valuable—it provides current documentation for any library, eliminating outdated training data issues. ### Available MCP Servers The MCP ecosystem includes connectors for: | Category | Connectors | |----------|-----------| | **Version Control** | GitHub, GitLab | | **Project Management** | Linear, Jira, Confluence, Asana | | **Communication** | Slack | | **Databases** | Supabase, Firebase | | **Testing** | Playwright | ### Architecture ```mermaid flowchart LR CC[Claude Code] --> MCP[MCP Protocol] subgraph "External Services" MCP --> GH[GitHub] MCP --> C7[Context7] MCP --> RL[Ralph Loop] MCP --> SL[Slack] MCP --> LN[Linear] end subgraph "Local Services" MCP --> TS[TypeScript LSP] MCP --> KT[Kotlin LSP] MCP --> RS[Rust Analyzer] end ``` ## Language Server Plugins These plugins provide IDE-level intelligence: ```json { "enabledPlugins": { "typescript-lsp@claude-plugins-official": true, "kotlin-lsp@claude-plugins-official": true, "rust-analyzer-lsp@claude-plugins-official": true } } ``` Benefits: - **Type information** - Claude understands your types - **Go to definition** - Navigate codebase precisely - **Error detection** - Catch issues before running - **Refactoring support** - Safe automated changes ## Quality & Workflow Plugins Beyond language servers, I use several quality plugins: ### security-guidance Provides security best practices: ```json { "enabledPlugins": { "security-guidance@claude-code-plugins": true } } ``` Alerts Claude to: - OWASP Top 10 vulnerabilities - XSS prevention - SQL injection risks - Authentication best practices ### explanatory-output-style Enables educational insights during development: ```json { "enabledPlugins": { "explanatory-output-style@claude-code-plugins": true } } ``` Provides `Insight` blocks that explain implementation choices as Claude works. ### frontend-design Production-grade UI generation: ```json { "enabledPlugins": { "frontend-design@claude-code-plugins": true } } ``` Ensures generated interfaces avoid generic AI aesthetics and follow proper design principles. ### context7 Access to current library documentation: ```json { "enabledPlugins": { "context7@claude-plugins-official": true } } ``` ### ralph-loop Automated iteration workflow: ```json { "enabledPlugins": { "ralph-loop@claude-plugins-official": true } } ``` ## Complete Plugin Configuration Here's my full plugin configuration from `~/.claude/settings.json`: ```json { "enabledPlugins": { "security-guidance@claude-code-plugins": true, "explanatory-output-style@claude-code-plugins": true, "frontend-design@claude-code-plugins": true, "context7@claude-plugins-official": true, "rust-analyzer-lsp@claude-plugins-official": true, "ralph-loop@claude-plugins-official": true, "typescript-lsp@claude-plugins-official": true, "kotlin-lsp@claude-plugins-official": true } } ``` Eight plugins working together to enhance Claude's capabilities across: - Language understanding (3 LSP plugins) - Security awareness (1 plugin) - Design quality (1 plugin) - Documentation access (1 plugin) - Workflow automation (1 plugin) - Educational output (1 plugin) ## Complete Real-World Workflow Let's walk through implementing a feature end-to-end using everything we've built. ### Scenario: Add User Authentication **Day 1: Research & Planning** ```bash # Start with research > /research_codebase authentication patterns # Claude spawns parallel agents: # - codebase-locator finds auth-related files # - codebase-analyzer documents current patterns # Output saved to thoughts/shared/research/ ``` ```bash # Create implementation plan (enforced by CLAUDE.md) > Add JWT authentication to the API # Claude (following CLAUDE.md rules): # 1. Creates a complete plan and checklist # 2. Asks you to verify before implementing # 3. Waits for approval ``` ```bash # Deep dive on the plan > /interview plan.md # Claude asks about: # - Token refresh strategy # - Session handling # - Error responses # - Edge cases ``` **Day 2: Implementation** ```bash # After plan approval, Claude implements # Following the phased approach from the plan # Running tests after each phase > /fix-test auth.test.ts # Claude: # - Runs the specified test # - Fixes any failures # - Re-runs to verify ``` **Day 3: Verification & Commit** ```bash # Verify against specification > /verify-spec # Claude confirms implementation matches plan ``` ```bash # Commit with proper workflow > /commit-all # Claude: # 1. Creates feature branch (not on main) # 2. Stages all changes # 3. Writes conventional commit message # 4. No co-author tags ``` ### How Components Interact Throughout this workflow: | Component | Role | |-----------|------| | **CLAUDE.md** | Enforced plan-first approach | | **/research_codebase** | Documented existing patterns | | **/interview** | Explored edge cases | | **/fix-test** | Fixed failing tests | | **TypeScript LSP** | Type checking during implementation | | **security-guidance** | Flagged auth security concerns | | **/verify-spec** | Confirmed spec compliance | | **/commit-all** | Clean git workflow | ## Getting Started: Adoption Path You don't need everything at once. Here's a progressive adoption path: ### Level 1: Foundation (Day 1) ```bash # Create global CLAUDE.md mkdir -p ~/.claude cat > ~/.claude/CLAUDE.md << 'EOF' - create a plan before implementation - summarize results after completion - don't commit unless asked directly EOF ``` ```bash # Enable always thinking cat > ~/.claude/settings.json << 'EOF' { "alwaysThinkingEnabled": true } EOF ``` ### Level 2: Commands (Week 1) Start with three essential commands: 1. **commit-all** - Git workflow enforcement 2. **explain** - Code explanation with diagrams 3. **research_codebase** - Codebase documentation ```bash mkdir -p ~/.claude/commands # Create commit-all cat > ~/.claude/commands/commit-all.md << 'EOF' --- description: Commit all changes with conventional messages --- Create branch if on main, commit all staged and unstaged files using conventional commit format. EOF ``` ### Level 3: Agents & Skills (Month 1) Add agents as needs arise: 1. **principal-frontend-architect** - For frontend decisions 2. **data-model-auditor** - For database design 3. **product-team-orchestrator** - For cross-functional coordination ### Level 4: MCP & Plugins (Ongoing) Enable plugins incrementally: ```json { "enabledPlugins": { "typescript-lsp@claude-plugins-official": true, "security-guidance@claude-code-plugins": true } } ``` Add more as you discover needs. ## The Investment and Return Building this system takes time. Here's what I've invested: | Component | Files | Setup Time | |-----------|-------|------------| | Foundation | 3 | 1 hour | | Commands | 9 | 3 hours | | Agents | 10 | 6 hours | | Skills | 19+ | (community) | | Plugins | 8 | 30 minutes | **Total: ~11 hours of configuration.** The return: - **Consistency** - Every project follows the same quality standards - **Speed** - Complex workflows reduced to single commands - **Quality** - Built-in verification gates and best practices - **Knowledge** - Persistent documentation and research - **Expertise** - On-demand access to specialized agents ## Directory Structure Summary ``` ~/.claude/ ├── CLAUDE.md # Global principles ├── settings.json # Plugins, hooks, status line ├── statusline.sh # Custom terminal status ├── commands/ # 9 slash commands │ ├── research_codebase.md │ ├── explain.md │ ├── commit-all.md │ └── ... ├── agents/ # 10 specialized agents │ ├── principal-frontend-architect.md │ ├── product-team-orchestrator.md │ └── ... ├── skills/ # 19+ professional skills │ ├── frontend-design/ │ ├── prd-generator/ │ └── ... ├── output-styles/ # 3 output formats │ ├── enhanced-readability.md │ ├── mentor.md │ └── principal-architect.md └── plugins/ # Plugin cache and config ``` ## Conclusion Claude Code's default configuration is a starting point. With the right customization, it becomes a personalized engineering partner that: 1. **Understands your workflow** through CLAUDE.md files 2. **Enforces your standards** through custom commands 3. **Provides specialized expertise** through agents and skills 4. **Connects to your tools** through MCP and plugins The system I've shared here represents months of refinement. But you don't need to adopt it all at once. Start with the foundation, add commands as patterns emerge, and grow the system organically. The best configuration is the one that evolves with your needs. ## What's Next In Part 5, we'll explore Claude Code's complete memory system: - **Memory hierarchy** - From organization policies to personal preferences - **Modular rules** - Path-specific instructions with `.claude/rules/` - **Memory imports** - Referencing external files in your configuration - **Best practices** - Organizing and maintaining your memory structure --- *Next in series: [Memory Management and Context Persistence](/2026/01/mastering-claude-code-memory-management-and-context-persistence/) - The complete memory system*

Mastering Claude Code: Specialized Agents and Professional Skills

Mon, 19 Jan 2026 00:00:00 GMT

## Beyond Commands: Personas with Expertise In [Part 2](/2026/01/mastering-claude-code-custom-commands-and-workflow/), we built a library of 9 custom commands. But commands are just instructions—they tell Claude *what* to do, not *who* to be. Agents are different. They're **personas with deep domain expertise**. When you engage an agent, Claude adopts that persona's knowledge, communication style, and decision-making framework. I've built 10 specialized agents organized into four categories: | Category | Agents | |----------|--------| | **Product Team** | product-team-orchestrator, product-owner-manager, product-ux-designer | | **Architecture** | startup-architect-strategist, principal-frontend-architect | | **Data** | data-model-auditor, database-erd-evaluator, jpa-hibernate-modeler | | **Analysis** | side-project-analyst, medium-article-writer | ## Product Team Agents These agents simulate a cross-functional product team. ### product-team-orchestrator The central coordinator that brings other agents together: ```markdown --- name: product-team-orchestrator description: Coordinate collaboration between product design, architecture, and management teams model: opus color: green --- You are an expert Product Development Orchestrator specializing in facilitating seamless collaboration between cross-functional teams. Your deep experience in agile methodologies, stakeholder management, and technical product development enables you to bridge gaps between design, architecture, and business strategy. **Core Responsibilities:** You will coordinate collaboration between @agent-product-ux-designer, @agent-startup-architect-strategist, and @agent-product-owner-manager to ensure optimal project outcomes. ``` Key capabilities: 1. **Collaboration Facilitation** - Initiate structured discussions between agents - Translate technical concepts for non-technical stakeholders - Create clear action items and ownership assignments 2. **Conflict Resolution** - Identify potential conflicts early - Mediate disagreements by finding common ground - Escalate to product owner when business priorities must guide 3. **Progress Tracking** - Maintain clear view of project status across workstreams - Identify dependencies and blockers - Provide regular updates to stakeholders ### product-owner-manager Handles product requirements and prioritization decisions. ### product-ux-designer Focuses on user experience, interface design, and design systems. ## Architecture Agents For system design and technical decisions. ### principal-frontend-architect My most detailed agent—15+ years of frontend expertise encoded: ```markdown --- name: principal-frontend-architect description: Expert-level frontend architecture decisions, code reviews, and performance optimization model: opus color: blue --- You are a Principal Frontend Developer with 15+ years of experience architecting and building world-class web applications. Your expertise spans the entire frontend ecosystem, from vanilla JavaScript to cutting-edge frameworks. **Core Expertise:** - Modern JavaScript/TypeScript and advanced patterns - React ecosystem mastery including Next.js, Redux, React Query - Deep understanding of browser APIs and rendering pipelines - CSS architecture at scale (CSS-in-JS, Tailwind, design systems) - Build tools (Webpack, Vite, Turbopack, esbuild) - Testing strategies (unit, integration, E2E, visual regression) - Performance metrics and optimization (Core Web Vitals) - Accessibility standards (WCAG 2.1, ARIA) - Security best practices (CSP, XSS prevention) ``` The agent's approach is structured: ```markdown **Your Approach:** 1. **Architectural Thinking**: Always consider the bigger picture - Scalability and maintainability implications - Performance impact across devices and networks - Developer experience and team velocity - Technical debt and migration paths 2. **Code Review Methodology**: When reviewing code: - First assess overall architecture and design patterns - Identify performance bottlenecks - Check for accessibility and security vulnerabilities - Suggest specific, actionable improvements with examples - Explain the 'why' behind each recommendation 3. **Problem-Solving Framework**: - Gather context about business requirements - Propose multiple solutions with trade-offs articulated - Recommend optimal approach based on context - Anticipate future scaling needs ``` And includes project-aware behavior: ```markdown **Project Context Awareness:** When working within an existing project: - Adhere to established coding standards and patterns - Respect existing architectural decisions - Consider the team's current tech stack and expertise - Align recommendations with project-specific requirements ``` ### startup-architect-strategist For early-stage technical decisions where speed and scalability must balance. ## Data Agents Specialized in database design and data modeling. ### data-model-auditor Reviews and critiques database schemas: ```markdown --- name: data-model-auditor description: Evaluate, critique, and improve database schemas and data architecture designs model: opus --- ``` Examines: - Entity relationships - Normalization levels - Indexing strategies - Performance implications - Scalability concerns ### database-erd-evaluator Validates Entity-Relationship Diagrams: - Checks relationship correctness - Identifies missing constraints - Verifies normalization - Reviews design against best practices ### jpa-hibernate-modeler Specifically for JPA/Hibernate entity modeling: ```markdown --- name: jpa-hibernate-modeler description: Expert assistance with JPA/Hibernate entity modeling, Spring Data repository design model: opus --- ``` Expertise includes: - Entity relationship design - Mapping configurations - N+1 query optimization - Transaction management - Spring Data best practices ## Analysis Agents For research and content creation. ### side-project-analyst Strategic analysis for solo ventures: - Project feasibility evaluation - MVP definition - Tech stack selection - Solopreneur-specific constraints ### medium-article-writer Creates publication-ready technical articles: - Engaging hooks and structure - Technical accuracy with accessibility - Proper formatting for Medium ## The Orchestrator Pattern The orchestrator pattern is powerful for complex tasks requiring multiple perspectives: ```mermaid flowchart TD O[product-team-orchestrator] --> PO[product-owner-manager] O --> UX[product-ux-designer] O --> ARCH[startup-architect-strategist] PO <--> O UX <--> O ARCH <--> O subgraph "Conflict Resolution" O --> C{Conflict?} C -->|Yes| M[Mediate] M --> S[Solution] C -->|No| P[Proceed] end ``` When to use the orchestrator: - Multiple stakeholders need alignment - Cross-functional dependencies need resolution - Technical implementation must align with UX and business strategy - Conflicts arise between different perspectives ## Professional Skills While agents are personas, **skills are capabilities**. They provide specialized knowledge for specific tasks. I have 19 skills organized into four categories: ### Document Skills | Skill | Purpose | |-------|---------| | `docx` | Word document creation and editing | | `pdf` | PDF manipulation and form filling | | `pptx` | Presentation creation and editing | | `xlsx` | Spreadsheet creation with formulas | ### Design Skills | Skill | Purpose | |-------|---------| | `frontend-design` | Production-grade web interfaces | | `canvas-design` | Visual art in PNG/PDF | | `theme-factory` | 10 pre-set themes for styling | | `algorithmic-art` | Generative art with p5.js | | `brand-guidelines` | Anthropic brand application | The frontend-design skill is particularly powerful: ```markdown --- name: frontend-design description: Create distinctive, production-grade frontend interfaces with high design quality --- ## Design Thinking Before coding, understand the context and commit to a BOLD aesthetic direction: - **Purpose**: What problem does this interface solve? - **Tone**: Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined... - **Constraints**: Technical requirements - **Differentiation**: What makes this UNFORGETTABLE? **CRITICAL**: Choose a clear conceptual direction and execute it with precision. ``` And explicit anti-patterns: ```markdown NEVER use generic AI-generated aesthetics like: - Overused fonts (Inter, Roboto, Arial, system fonts) - Cliched color schemes (purple gradients on white) - Predictable layouts and component patterns - Cookie-cutter design lacking context-specific character ``` ### Content Skills | Skill | Purpose | |-------|---------| | `doc-coauthoring` | Structured documentation workflow | | `prd-generator` | Product Requirements Documents | | `slack-gif-creator` | Animated GIFs for Slack | | `internal-comms` | Status reports, newsletters, FAQs | | `skill-creator` | Create new skills | The PRD generator follows Carlin Yuen's methodology: ```markdown --- name: prd-generator description: Generate PRDs using Carlin Yuen's methodology --- ## Core Principles ### 1. Problem-First Thinking Focus on PROBLEMS, not lack of a solution: - DO: "Users waste 30 minutes daily searching for documents" - DON'T: "Users need a better search feature" ### 2. User-Centric Use Cases Write use cases abstracted from solutions: - "As a [user type], I [need/action] because [reason]" ### 3. Functionality Over Implementation Requirements describe WHAT, not HOW: - DO: "User can identify who has access to their document" - DON'T: "Add dropdown menu in top-right corner" ``` ### Development Skills | Skill | Purpose | |-------|---------| | `mcp-builder` | Create MCP servers | | `webapp-testing` | Test web apps with Playwright | | `web-artifacts-builder` | Complex React/Tailwind artifacts | ## When to Use What: Decision Table | Scenario | Use | |----------|-----| | Need to understand existing code | `/research_codebase` command | | Need architectural guidance | `principal-frontend-architect` agent | | Need to coordinate multiple concerns | `product-team-orchestrator` agent | | Need to create a document | `docx`, `pdf`, or `pptx` skill | | Need to design a UI | `frontend-design` skill | | Need to write requirements | `prd-generator` skill | | Need code explanation with diagrams | `/explain` command | | Need to verify implementation | `/verify-spec` command | ## Agent and Skill Structure ### Agents (`~/.claude/agents/*.md`) ```markdown --- name: agent-name description: When to use this agent with examples model: opus # or sonnet, haiku color: blue # terminal color --- You are [persona description with expertise]... **Core Responsibilities:** ... **Your Approach:** ... ``` ### Skills (`~/.claude/skills/*/SKILL.md`) ```markdown --- name: skill-name description: When to use this skill --- This skill guides [what it does]... ## Guidelines ... ``` Skills can include reference documents in subdirectories: ``` ~/.claude/skills/prd-generator/ ├── SKILL.md └── references/ ├── product-prd-template.md └── feature-prd-template.md ``` ## Conclusion Agents and skills transform Claude Code from a single assistant into a virtual team of specialists. While commands automate workflows, agents bring deep domain expertise and skills provide polished document generation. Key takeaways: - **Agents** are personas with specialized knowledge (architecture, product, data modeling) - **Skills** are tools for generating specific outputs (documents, presentations, designs) - **The orchestrator pattern** coordinates multiple agents for complex cross-functional tasks - **Decision framework**: Use commands for workflows, agents for expertise, skills for artifacts The 10 agents and 19+ skills we covered provide coverage across product development, architecture, data modeling, and content creation. Build on these foundations to create agents tailored to your domain. ## What's Next Agents and skills give you specialized expertise on demand. But true power comes from connecting Claude Code to your entire toolchain. In Part 4, we'll explore: - **MCP connectors** - GitHub, Firebase, Playwright, Slack, and more - **Language server plugins** - TypeScript, Kotlin, Rust analyzers - **Quality plugins** - Security guidance, design patterns - **A complete real-world workflow** - End-to-end feature development Together with the foundation, commands, agents, and skills we've covered, you'll have a complete system for AI-augmented development. --- *Next in series: [MCP Connectors, Plugins, and Complete Integration](/2026/01/mastering-claude-code-mcp-connectors-plugins-and-complete-integration/) - Connecting to your toolchain*

Mastering Claude Code: Custom Commands and Workflow

Sun, 18 Jan 2026 00:00:00 GMT

## Commands as Encoded Best Practices In [Part 1](/2026/01/mastering-claude-code-foundation-and-philosophy/), we established the foundation with CLAUDE.md files and terminal customization. Now we'll build on that with custom slash commands. Custom commands are more than shortcuts. They're **encoded best practices**—complex workflows distilled into simple invocations that ensure consistency across all your projects. I've built 9 commands organized into three categories: | Category | Commands | |----------|----------| | **Research & Documentation** | `/research_codebase`, `/explain`, `/interview` | | **Git Workflow** | `/commit-all`, `/commit-staged`, `/commit-all-main` | | **Quality** | `/fix-test`, `/optimize`, `/verify-spec` | Let's explore each category. ## Research & Documentation Commands Understanding code is the foundation of any change. These commands help you document and comprehend codebases effectively. ### /research_codebase My go-to command for understanding any part of a codebase: ```markdown --- description: Document codebase as-is with thoughts directory for historical context model: opus --- # Research Codebase You are tasked with conducting comprehensive research across the codebase to answer user questions by spawning parallel sub-agents and synthesizing their findings. ## CRITICAL: YOUR ONLY JOB IS TO DOCUMENT AND EXPLAIN ## THE CODEBASE AS IT EXISTS TODAY - DO NOT suggest improvements or changes - DO NOT perform root cause analysis - DO NOT propose future enhancements - DO NOT critique the implementation - ONLY describe what exists, where it exists, how it works ``` Key features: 1. **Parallel research** - Uses specialized agents for efficiency: ```markdown **For codebase research:** - Use the **codebase-locator** agent to find WHERE files and components live - Use the **codebase-analyzer** agent to understand HOW specific code works - Use the **codebase-pattern-finder** agent to find examples of existing patterns ``` 2. **Persistent output** - Research is saved for future reference: ```markdown ## Generate research document: - Filename: `thoughts/shared/research/YYYY-MM-DD-description.md` - Structure the document with YAML frontmatter: - date, researcher, git_commit, branch, repository - topic, tags, status ## Summary [High-level documentation answering the user's question] ## Detailed Findings ### [Component/Area 1] - Description of what exists ([file.ext:line](link)) - How it connects to other components ``` 3. **Historical context** - Integrates with the thoughts directory pattern from Part 1, providing both live codebase findings and historical context from previous research. ### /explain Code explanation with ASCII diagrams for visual understanding: ```markdown Explain code using concise explanations and ASCII diagrams that visualize control flow and data flow. Show how execution proceeds and how data transforms through the system. ``` This command produces explanations with visual diagrams, making it perfect for: - Onboarding onto unfamiliar codebases - Understanding complex control flows - Documenting how data moves through systems - Creating visual references for team discussions ### /interview Deep dive into plans, specifications, or any document: ```markdown --- description: Interview me about the plan argument-hint: [plan] model: opus --- Read this plan file $1 and interview me in detail using the AskUserQuestionTool about literally anything: technical implementation, UI & UX, concerns, tradeoffs, etc. but make sure the questions are not obvious. Be very in-depth and continue interviewing me continually until it's complete, then write the spec to the file. ``` This command forces you to think through edge cases and assumptions you might have missed. It's invaluable for: - Validating implementation plans before starting work - Identifying gaps in specifications - Stress-testing architectural decisions - Uncovering hidden requirements ## Git Workflow Commands These commands enforce consistent git practices across all projects. ### /commit-all My standard commit command with safety checks: ```markdown If you are in Main or main or master branch, you should create a new branch first and then commit your changes. Check all staged and unstaged files and commit everything. You are following git conventional message for commits. ## NEVER DO - do not add claude or CLAUDE text in any case like co-author in the git commit message. ``` This enforces: - **Branch protection** - Never commit directly to main - **Conventional commits** - Consistent message format - **Clean history** - No AI co-author tags ### /commit-staged For when you want more control over what gets committed: ```markdown If you are in Main or main or master branch, you should create a new branch first and then commit your changes. Only commit staged files. You are following git conventional message for commits. ``` Use this when you've carefully staged specific changes and don't want Claude to add anything else. ### /commit-all-main For the rare cases when you explicitly want to commit directly to main: ```markdown You are in Main or main or master branch, do not create a new branch just commit your changes. ``` Use sparingly—typically only for documentation updates or configuration changes that don't need a feature branch. ## Quality Commands These commands help maintain code quality and verify implementations. ### /fix-test Run a specific test and automatically fix failures: ```markdown --- description: Run test and fix the failed test case --- Run test #$ARGUMENTS following our coding standards and fix the failed test case ``` Simple but incredibly useful for TDD workflows. When a test fails, this command: 1. Runs the specified test 2. Analyzes the failure 3. Fixes the code following your project's coding standards 4. Re-runs to verify the fix ### /optimize Performance analysis and suggestions: ```markdown --- description: Analyze code for performance issues --- Analyze this code for performance issues and suggest optimizations and store the suggestions in the plan file. ``` This command examines code for: - Algorithmic inefficiencies - Memory usage patterns - Database query optimization opportunities - Caching opportunities - Bundle size considerations (for frontend code) ### /verify-spec Verify implementation against specifications: ```markdown --- description: Verify the implementation against agreed Spec --- Verify the implementation against the agreed specification. ``` This is invaluable for ensuring your implementation matches what was planned. It: - Compares actual implementation to specification - Identifies missing features or behaviors - Highlights deviations that need discussion - Confirms when work is complete ## The Complete Workflow Here's how these commands work together in practice: ```mermaid flowchart TD A[Feature Request] --> B["/research_codebase"] B --> C[Understand Context] C --> D[Create Plan] D --> E["/interview"] E --> F{Plan Refined?} F -->|No| D F -->|Yes| G[Implement] G --> H["/fix-test"] H --> I["/verify-spec"] I --> J{Spec Met?} J -->|No| G J -->|Yes| K["/commit-all"] K --> L[Done] ``` A typical feature development: 1. **`/research_codebase`** - Understand existing patterns and architecture 2. **Create a plan** - Use Claude's planning capabilities (enforced by CLAUDE.md) 3. **`/interview`** - Deep dive into edge cases and assumptions 4. **Implement** - Write the code 5. **`/fix-test`** - Run tests and fix any failures 6. **`/verify-spec`** - Confirm implementation matches specification 7. **`/commit-all`** - Create branch if needed and commit changes ## Command Structure Commands live in `~/.claude/commands/` as Markdown files: ``` ~/.claude/commands/ ├── research_codebase.md ├── explain.md ├── interview.md ├── commit-all.md ├── commit-staged.md ├── commit-all-main.md ├── fix-test.md ├── optimize.md └── verify-spec.md ``` Each file has: 1. **YAML frontmatter** - metadata like description and model 2. **Markdown body** - the actual command instructions Example structure: ```markdown --- description: Brief description shown in /help argument-hint: [optional argument hint] model: opus # or sonnet, haiku --- # Command Name Instructions for Claude to follow when this command is invoked... ``` The `model` field is powerful—complex commands like `/research_codebase` use Opus for maximum capability, while simpler commands can use Sonnet or Haiku for speed. ## Creating Your Own Commands To create a new command: 1. Create a markdown file in `~/.claude/commands/` 2. Add YAML frontmatter with at least a `description` 3. Write the instructions Claude should follow Example custom command for code review: ```markdown --- description: Review code for security and best practices --- Review the provided code for: 1. Security vulnerabilities (OWASP Top 10) 2. Error handling completeness 3. Input validation 4. Authentication/authorization issues Provide specific, actionable feedback with file:line references. ``` ## Conclusion Custom commands transform repetitive workflows into single invocations. By encoding best practices—research before implementation, interview to validate assumptions, verify against specifications—you create guardrails that improve code quality while reducing cognitive overhead. Key takeaways: - **Research commands** (`/research_codebase`, `/explain`) capture and preserve knowledge - **Git commands** (`/commit-all`, `/commit-staged`) enforce safe branching practices - **Quality commands** (`/fix-test`, `/verify-spec`, `/optimize`) maintain standards - **Command structure** is simple: Markdown file with YAML frontmatter The 9 commands we covered form a complete development lifecycle, but they're just building blocks. Create your own commands for workflows unique to your team or project. ## What's Next Commands are powerful, but they're just one layer. In Part 3, we'll explore: - **10 specialized agents** - Personas with deep domain expertise - **19+ professional skills** - Document generation, design, development workflows - **The orchestrator pattern** - Coordinating multiple agents for complex tasks These agents and skills transform Claude Code from a single assistant into a virtual team of specialists. --- *Next in series: [Specialized Agents and Professional Skills](/2026/01/mastering-claude-code-specialized-agents-and-professional-skills/) - Creating a virtual team*

Mastering Claude Code: Foundation and Philosophy

Sat, 17 Jan 2026 00:00:00 GMT

## Introduction: From Tool to Partner Claude Code out of the box is impressive. But with the right configuration, it transforms from a capable tool into something far more powerful: a personalized engineering partner that understands your workflow, enforces your standards, and adapts to your project's unique requirements. This series documents my complete Claude Code setup after months of refinement. We'll cover: 1. **Foundation & Philosophy** (this post) - Core principles, configuration files, and terminal customization 2. **Custom Commands & Workflow** - 9 slash commands that encode best practices 3. **Specialized Agents & Skills** - 10 agents and 19+ skills for every situation 4. **MCP, Plugins & Integration** - Connecting Claude Code to your entire toolchain 5. **Memory Management** - Hierarchical memory system and context persistence 6. **Chrome Browser Automation** - Live debugging, testing, and browser workflows Let's start with the foundation that makes everything else possible. ## CLAUDE.md: Your Engineering Constitution The `CLAUDE.md` file is where you establish the ground rules. Claude Code supports two levels of configuration: ### Global Configuration (`~/.claude/CLAUDE.md`) This file applies to every Claude Code session, regardless of which project you're in. Here's my global configuration: ```markdown - every time before implementation create a complete plan and checklist and ask me to verify it, if I verified then go ahead and implement it. - every time after delivering the result give a summary of what you have done. - do not commit files and word unless I ask you directly. - always follow SRP (single responsibility) either classes, components or functions. it can help keep function (any units) smaller and more understandable ``` These four principles encode my development philosophy: 1. **Plan before implementation** - Forces structured thinking and prevents Claude from jumping straight into code 2. **Summarize results** - Creates accountability and makes it easy to review what was done 3. **Explicit commit control** - Prevents accidental commits; I control when changes are committed 4. **Single Responsibility Principle** - Keeps generated code clean and maintainable ### Project-Level Configuration (`./CLAUDE.md`) Each project can have its own `CLAUDE.md` that overrides or extends the global settings. Here's an example from my Astro blog project: ```markdown # CLAUDE.md This file provides guidance to Claude Code when working with code in this repository. ## Commands \`\`\`bash # Development npm run dev # Start dev server (localhost:3000) npm run build # Build for production npm run preview # Preview production build # Content tooling npm run mermaid # Generate SVG diagrams from mermaid blocks \`\`\` ## Architecture ### Content System **Two-collection model** using Astro Content Collections: 1. **Posts** (`src/content/posts/*.md`) - Blog articles - Required: `title`, `description`, `pubDate` - Filename convention: `YYYY-MM-DD-slug.md` -> URL: `/YYYY/MM/slug/` 2. **Series** (`src/content/series/*.json`) - Multi-part metadata - Links posts together via `series.name` in frontmatter ``` The project-level file provides Claude with context about the specific codebase, its conventions, and available commands. This dramatically improves the relevance of Claude's suggestions. ### How Configuration Files Merge When both files exist, Claude sees both. The global file establishes your personal standards, while the project file provides context-specific guidance. Think of it as: - **Global**: "How I work" - **Project**: "How this codebase works" ```mermaid flowchart TD subgraph "Configuration Hierarchy" A["~/.claude/CLAUDE.md
(Global Preferences)"] --> C["Claude Code Session"] B["./CLAUDE.md
(Project Context)"] --> C end subgraph "What Claude Sees" C --> D["Personal Standards
+ Project Context"] end style A fill:#e1f5fe style B fill:#fff3e0 style C fill:#f3e5f5 style D fill:#e8f5e9 ``` ## Terminal Customization: Information at a Glance Claude Code supports a custom status line that displays in your terminal. I've configured mine to show everything I need at a glance. ### The Status Line Script Here's my `~/.claude/statusline.sh`: ```bash #!/bin/bash input=$(cat) # Define colors as actual escape sequences GREEN=$'\033[0;32m' YELLOW=$'\033[0;33m' RED=$'\033[0;31m' BLUE=$'\033[0;34m' CYAN=$'\033[1;36m' MAGENTA=$'\033[0;35m' WHITE=$'\033[0;37m' RESET=$'\033[0m' eval "$(echo "$input" | jq -r ' @sh "MODEL=\(.model.display_name)", @sh "DIR=\(.workspace.current_dir)", @sh "COST=\(.cost.total_cost_usd // 0)", @sh "STYLE=\(.output_style.name)", @sh "SESSION=\(.session_id)", @sh "VERSION=\(.version)", @sh "CTX_PCT=\(.context_window.used_percentage)", @sh "CTX_REMAINING_PCT=\(.context_window.remaining_percentage)", @sh "CTX_SIZE=\(.context_window.context_window_size)" ')" # Calculate remaining in k CTX_REMAINING=$(( CTX_SIZE * CTX_REMAINING_PCT / 100 / 1000 ))k # Color based on usage if [ "$CTX_PCT" -ge 80 ]; then CTX_COLOR="$RED" elif [ "$CTX_PCT" -ge 60 ]; then CTX_COLOR="$YELLOW" else CTX_COLOR="$GREEN" fi # Git branch in workspace directory GIT_INFO="" if git -C "$DIR" rev-parse --git-dir > /dev/null 2>&1; then BRANCH=$(git -C "$DIR" branch --show-current 2>/dev/null) [ -n "$BRANCH" ] && GIT_INFO=" ${GREEN} $BRANCH${RESET}" fi SESSION_SHORT="${SESSION:0:8}" printf "%s[%s]%s %s %s%s%s %s %s%% (%s left)%s %s\$%.4f%s %s%s%s %sv%s%s %s#%s%s\n" \ "$CYAN" "$MODEL" "$RESET" \ "$BLUE" "${DIR##*/}" "$RESET" "$GIT_INFO" \ "$CTX_COLOR" "$CTX_PCT" "$CTX_REMAINING" "$RESET" \ "$YELLOW" "$COST" "$RESET" \ "$MAGENTA" "$STYLE" "$RESET" \ "$WHITE" "$VERSION" "$RESET" \ "$WHITE" "$SESSION_SHORT" "$RESET" ``` This displays: - **Model name** (Opus 4.5, Sonnet, etc.) - **Current directory** and git branch - **Context usage** with color-coded warnings (green < 60%, yellow 60-80%, red > 80%) - **Session cost** in USD - **Active output style** - **Claude Code version** - **Session ID** (first 8 characters) ### Enabling the Status Line In `~/.claude/settings.json`: ```json { "statusLine": { "type": "command", "command": "~/.claude/statusline.sh", "padding": 0 } } ``` ### macOS Notifications I also have notifications configured to alert me when Claude needs input: ```json { "hooks": { "Notification": [ { "matcher": "", "hooks": [ { "type": "command", "command": "osascript -e 'display notification \"Claude needs your attention\" with title \"Claude Code\"'" } ] } ] } } ``` This is invaluable when Claude is working on a long task and I'm focused elsewhere. ### Always Thinking Mode One critical setting I always enable: ```json { "alwaysThinkingEnabled": true } ``` This ensures Claude uses extended thinking for all responses, leading to more thorough and accurate results. The quality difference is noticeable. ## The Thoughts Directory Pattern For complex projects, I use a `thoughts/` directory to persist context across sessions: ``` thoughts/ ├── shared/ │ ├── plans/ │ │ └── 2026-01-15-feature-auth.md │ └── research/ │ └── 2026-01-10-auth-patterns.md └── local/ └── scratch.md ``` This pattern provides: 1. **Session persistence** - Plans and research survive beyond a single session 2. **Historical context** - Past decisions inform future work 3. **Separation of concerns** - Shared knowledge vs. personal notes My `/research_codebase` command (covered in Part 2) automatically saves findings to `thoughts/shared/research/`. ## Output Styles: Adapting Claude's Voice Claude Code supports output styles that change how responses are formatted. I've created three styles for different situations: ### Enhanced Readability For quick tasks where I need scannable output: ```markdown --- name: Enhanced Readability description: Improved visual hierarchy for better readability --- ## Response Structure ### Overview Brief summary of what I'm about to do or explain ### Implementation - **Key Actions**: What specific steps I'm taking - **Files Modified**: Clear list of affected files - **Commands**: `formatted as code` for easy copying ### Results Success: What was accomplished Warnings: Any issues to be aware of Errors: Problems that need attention ``` ### Mentor Mode When I'm learning something new or onboarding onto a codebase: ```markdown --- name: Mentor description: Senior engineer mentoring with deep explanations --- When responding: - **Explain the "why"**: Don't just provide solutions; explain the reasoning - **Connect concepts**: Show how different parts relate - **Anticipate questions**: Address common concerns proactively - **Use teaching moments**: Turn every task into learning ``` ### Principal Architect For architectural decisions and system design: ```markdown --- name: Principal Architect description: Comprehensive architectural analysis with trade-offs --- ## Response Structure ### 1. Architectural Context & Problem Analysis - Define the problem space clearly - Identify key constraints (technical, business, organizational) ### 2. Technical Decision Framework - **Primary Recommendation**: Lead with your preferred approach - **Trade-off Analysis**: What you're optimizing vs. sacrificing - **Alternative Approaches**: 2-3 viable alternatives with pros/cons ### 3. System Design Implications - Scalability, Performance, Maintainability, Operational Complexity ``` Switching between styles is as simple as using the `/output-style` command. ## Core Settings Summary Here's the complete `~/.claude/settings.json` with the key configurations: ```json { "statusLine": { "type": "command", "command": "~/.claude/statusline.sh", "padding": 0 }, "hooks": { "Notification": [ { "matcher": "", "hooks": [ { "type": "command", "command": "osascript -e 'display notification \"Claude needs your attention\" with title \"Claude Code\"'" } ] } ] }, "alwaysThinkingEnabled": true, "verbose": true, "includeCoAuthoredBy": false } ``` The `includeCoAuthoredBy: false` setting prevents Claude from adding itself as a co-author on commits, keeping your git history clean. ## Conclusion The foundation we've established here—CLAUDE.md configuration, terminal customization, the thoughts directory pattern, and output styles—creates the framework for everything that follows. These aren't just settings; they're the principles that transform Claude Code from a capable tool into a personalized engineering partner. Key takeaways: - **Global CLAUDE.md** encodes your development philosophy across all projects - **Project CLAUDE.md** provides context-specific guidance for each codebase - **Status line** gives you real-time visibility into cost, context usage, and session state - **Output styles** adapt Claude's communication to your current needs ## What's Next With the foundation in place, Part 2 will dive into the 9 custom commands that make my workflow sing: - `/research_codebase` - Document and understand any codebase - `/interview` - Deep dive into edge cases and assumptions - `/commit-all` - Git workflow enforcement - `/explain` - Code explanation with ASCII diagrams - And 5 more commands that encode best practices The configuration files we've set up here provide the context that makes those commands work effectively. Together, they transform Claude Code from a generic assistant into a specialized engineering partner that understands your standards, respects your workflow, and produces consistently high-quality results. --- *Next in series: [Custom Commands and Workflow](/2026/01/mastering-claude-code-custom-commands-and-workflow/) - Building a powerful command library*

Building Self-Service Analytics Dashboards with Metabase

Fri, 02 Jan 2026 00:00:00 GMT

**The Final Chapter: Turning Your Data Warehouse into Actionable Insights** --- You have spent weeks building a modern analytics stack. Your Python extractors pull data from MongoDB flawlessly. DuckDB loads and stores everything with blazing speed. Your dbt models transform raw chaos into a pristine star schema. But here is the uncomfortable truth: none of that matters if your stakeholders cannot actually *see* the data. This is the final mile problem in analytics. We obsess over extraction patterns and transformation logic, yet the visualization layer often becomes an afterthought. It should not be. This is Article 7 of our series on building a complete analytics platform for Claude AI conversation logs. In previous articles, we covered the Python extraction layer (Article 4), DuckDB loading strategies (Article 5), and dbt transformations using the medallion architecture (Article 6). Now we complete the journey by connecting Metabase to our gold layer models and building self-service dashboards that anyone can explore. Why Metabase? Three reasons. First, it is open-source and self-hostable, which means no vendor lock-in and full control over your data. Second, it has native DuckDB support through a community driver, eliminating the need for intermediate services. Third, its visual query builder empowers business users to explore data without writing SQL, while still offering full SQL access for power users. By the end of this article, you will have three production-ready dashboards: Developer Productivity, AI Interaction Patterns, and Project Insights. More importantly, you will understand how to design self-service analytics that scale with your organization. Let us finish what we started. --- ## Metabase Architecture in Our Stack Before diving into dashboard creation, we need to understand how Metabase fits into our existing Docker Compose infrastructure. The analytics platform runs several interconnected services, and Metabase serves as the presentation layer that queries our DuckDB warehouse. ```mermaid flowchart TB subgraph "Data Sources" JSONL[("~/.claude/projects/*.jsonl")] end subgraph "Sync Service" Watcher["File Watcher"] Buffer["SQLite Buffer"] Sync["MongoDB Sync"] end subgraph "MongoDB" Mongo[("conversations collection")] end subgraph "Analytics Platform" Extractor["Python Extractor"] Loader["DuckDB Loader"] DBT["dbt Transformations"] DuckDB[("DuckDB Warehouse")] end subgraph "Visualization Layer" Metabase["Metabase :3001"] MetabaseDB[("PostgreSQL Metadata")] end JSONL --> Watcher Watcher --> Buffer Buffer --> Sync Sync --> Mongo Mongo --> Extractor Extractor --> Loader Loader --> DuckDB DBT --> DuckDB DuckDB --> Metabase MetabaseDB --> Metabase ``` The key insight here is that Metabase queries DuckDB directly through a shared Docker volume. There is no intermediate API layer or data copying. When a user views a dashboard, Metabase executes SQL against the same `analytics.db` file that dbt populates during transformation runs. ### The Custom Dockerfile Metabase does not ship with DuckDB support out of the box. We need a custom image that includes the community DuckDB driver. Here is our Dockerfile: ```dockerfile # Custom Metabase image with DuckDB support # Based on MotherDuck's recommended approach for glibc compatibility FROM eclipse-temurin:21-jre-jammy # Build arguments for versions ARG METABASE_VERSION=0.56.9 ARG METABASE_DUCKDB_DRIVER_VERSION=1.4.3.0 ENV MB_PLUGINS_DIR=/home/metabase/plugins/ # Create metabase user RUN groupadd -r metabase && useradd -r -g metabase metabase # Install CA certificates for HTTPS downloads and curl for health checks RUN apt-get update && apt-get install -y \ ca-certificates \ curl \ && rm -rf /var/lib/apt/lists/* # Create directories for plugins and data RUN mkdir -p /home/metabase/plugins /home/metabase/data /duckdb && \ chown -R metabase:metabase /home/metabase /duckdb WORKDIR /home/metabase # Download Metabase JAR ADD --chown=metabase:metabase \ https://downloads.metabase.com/v${METABASE_VERSION}/metabase.jar \ /home/metabase/ # Download DuckDB driver ADD --chown=metabase:metabase \ https://github.com/MotherDuck-Open-Source/metabase_duckdb_driver/releases/download/${METABASE_DUCKDB_DRIVER_VERSION}/duckdb.metabase-driver.jar \ /home/metabase/plugins/ # Ensure proper file permissions RUN chmod 755 /home/metabase/metabase.jar && \ chmod 755 /home/metabase/plugins/duckdb.metabase-driver.jar EXPOSE 3000 USER metabase CMD ["java", "-jar", "/home/metabase/metabase.jar"] ``` A few important decisions in this Dockerfile deserve explanation: **Eclipse Temurin base image.** We use `eclipse-temurin:21-jre-jammy` instead of Alpine because the DuckDB driver requires glibc compatibility. Alpine uses musl libc, which causes cryptic runtime errors with DuckDB's native extensions. **Version pinning.** Both `METABASE_VERSION` and `METABASE_DUCKDB_DRIVER_VERSION` are pinned as build arguments. This ensures reproducible builds and allows easy upgrades by changing a single value. **Non-root user.** The container runs as the `metabase` user rather than root, following security best practices for production deployments. ### Docker Compose Integration The Metabase service definition in our Docker Compose file handles the integration: ```yaml metabase: build: context: ./metabase dockerfile: Dockerfile container_name: metabase environment: # PostgreSQL for Metabase metadata - MB_DB_TYPE=postgres - MB_DB_DBNAME=metabase - MB_DB_PORT=5432 - MB_DB_USER=metabase - MB_DB_PASS=metabase - MB_DB_HOST=metabase-db - MB_EMOJI_IN_LOGS=false ports: - "3001:3000" volumes: # DuckDB needs write access for lock files even when reading - duckdb-data:/duckdb depends_on: metabase-db: condition: service_healthy healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:3000/api/health || exit 1"] interval: 30s timeout: 10s retries: 5 start_period: 120s networks: - analytics-network ``` The port mapping `3001:3000` exposes Metabase on port 3001 externally while it runs on 3000 internally. This avoids conflicts with other services that might use port 3000, such as Next.js development servers. The `duckdb-data` volume mount is critical. This shared volume allows both the analytics worker (which runs dbt) and Metabase to access the same DuckDB database file. Note that DuckDB requires write access even for read operations because it creates lock files. --- ## Setting Up the DuckDB Connection With Metabase running, the first step is connecting it to our DuckDB warehouse. Start the analytics stack if you have not already: ```bash cd analytics make up ``` After the containers finish initializing (Metabase takes about 2 minutes on first startup), navigate to `http://localhost:3001` in your browser. Complete the initial setup wizard by creating an admin account. ### Adding DuckDB as a Data Source From the Metabase admin panel, navigate to **Admin Settings > Databases > Add Database**. Select DuckDB as the database type (it appears in the list because we installed the driver in our custom image). Configure the connection with these settings: | Setting | Value | Notes | |---------|-------|-------| | Database type | DuckDB | Community driver | | Display name | Claude Analytics | User-friendly name | | Database file path | `/duckdb/analytics.db` | Path inside container | Click **Save** and Metabase will scan the database schema. Within a few seconds, you should see all the tables from our dbt project appear in the data browser. ### Schema Discovery Metabase automatically discovers three schemas in our DuckDB database: - **raw**: Source tables loaded by the Python loader - **staging**: Bronze layer models (cleaned source data) - **intermediate**: Silver layer models (enriched and joined) - **marts**: Gold layer models (star schema for BI) For dashboards, we focus exclusively on the `marts` schema. These tables are specifically designed for analytics consumption with pre-aggregated metrics and denormalized dimensions. --- ## Understanding the Gold Layer Models Before building dashboards, let us examine what data is available in the gold layer. Our dbt project creates a classic star schema with dimension tables, fact tables, and pre-aggregated summary tables. ```mermaid erDiagram dim_date ||--o{ fct_messages : "date_key" dim_date ||--o{ fct_tool_calls : "date_key" dim_date ||--o{ fct_file_operations : "date_key" dim_date ||--o{ agg_daily_summary : "date_key" dim_sessions ||--o{ fct_messages : "session_id" dim_sessions ||--o{ fct_tool_calls : "session_id" dim_sessions ||--o{ fct_file_operations : "session_id" dim_projects ||--o{ dim_sessions : "project_id" dim_projects ||--o{ fct_messages : "project_id" dim_tools ||--o{ fct_tool_calls : "tool_name" dim_tools ||--o{ agg_tool_efficiency : "tool_name" dim_date { date date_key PK int year int month string day_name bool is_weekend } dim_sessions { string session_id PK string project_id FK timestamp started_at float duration_minutes int message_count } dim_projects { string project_id PK int session_count int total_messages string activity_status } dim_tools { string tool_name PK string tool_category int popularity_rank } fct_messages { string message_id PK string session_id FK string role int content_length bool has_code_block } fct_tool_calls { string tool_call_id PK string session_id FK string tool_name FK float execution_seconds } agg_daily_summary { date date_key PK int session_count int message_count int tool_call_count } agg_tool_efficiency { string tool_name PK int total_calls float avg_execution_seconds int popularity_rank } ``` ### Key Tables for Analytics **Dimension Tables:** - `dim_date`: Calendar dimension with temporal attributes (year, month, day of week, weekend flags) - `dim_sessions`: Session-level attributes including duration, message counts, and project association - `dim_projects`: Project dimension with activity metrics and status classification - `dim_tools`: Tool catalog with categories and popularity rankings **Fact Tables:** - `fct_messages`: Message-level analytics with content analysis (code blocks, questions, length) - `fct_tool_calls`: Individual tool invocations with execution timing - `fct_file_operations`: File read/write/edit operations with path analysis **Aggregate Tables:** - `agg_daily_summary`: Pre-computed daily metrics for time-series dashboards - `agg_hourly_activity`: Activity heatmap data by hour and day of week - `agg_session_metrics`: Session-level statistics and percentiles - `agg_tool_efficiency`: Tool performance and usage rankings The aggregate tables deserve special attention. By pre-computing metrics during dbt runs, we shift computational work from query time to transformation time. This means dashboards load instantly even with millions of underlying records. Consider the `agg_daily_summary` model. Instead of calculating session counts on every dashboard refresh, dbt computes these values once: ```sql -- From agg_daily_summary.sql daily_sessions as ( select date_key, count(*) as session_count, count(distinct project_id) as active_projects, sum(duration_minutes) as total_session_minutes, avg(duration_minutes) as avg_session_minutes, sum(message_count) as total_messages, avg(message_count) as avg_messages_per_session, sum(tool_call_count) as total_tool_calls from sessions group by date_key ) ``` When Metabase queries this table, it simply reads pre-aggregated rows rather than scanning millions of session records. --- ## Building the Sample Dashboards Our analytics platform includes a `sample_questions.json` file that defines 12 pre-configured questions across three dashboards. This file serves as both documentation and a template for recreating these visualizations in Metabase. ### Dashboard Structure ```json { "dashboards": [ { "name": "Developer Productivity", "description": "Session duration, activity patterns, and productivity metrics", "questions": [ "Sessions Over Time", "Average Session Duration", "Active Hours Heatmap", "Task Category Distribution" ] }, { "name": "AI Interaction Patterns", "description": "Tool usage, efficiency, and interaction analysis", "questions": [ "Tool Usage Distribution", "Tool Category Breakdown", "Top 10 Most Used Tools", "Response Time Trends" ] }, { "name": "Project Insights", "description": "Project activity, code changes, and file operations", "questions": [ "Project Activity Ranking", "Project Status Distribution", "Daily Code Changes", "File Types Worked On" ] } ] } ``` Let us build each dashboard step by step. ### Dashboard 1: Developer Productivity This dashboard answers the question: "How am I using Claude AI, and when am I most productive?" **Sessions Over Time (Line Chart)** ```sql SELECT date_key, session_count FROM marts.agg_daily_summary ORDER BY date_key ``` This simple query powers a trend line showing daily session counts. Spikes indicate intense development periods, while dips might correlate with meetings or context switches. **Active Hours Heatmap (Pivot Table)** ```sql SELECT hour_of_day, day_name, total_activity FROM marts.agg_hourly_activity ``` Configure this as a pivot table with `day_name` on rows, `hour_of_day` on columns, and `total_activity` as the cell value. The result is a heatmap showing when you are most actively coding with Claude. Many developers discover surprising patterns, like unexpected productivity during early morning hours. **Average Session Duration (Scalar)** ```sql SELECT AVG(duration_minutes) as avg_duration FROM marts.dim_sessions ``` Display this as a single number card. Track this metric over time to understand whether your Claude sessions are becoming more focused (shorter) or more exploratory (longer). **Task Category Distribution (Pie Chart)** ```sql SELECT task_category, COUNT(*) as count FROM marts.fct_messages GROUP BY task_category ORDER BY count DESC ``` This visualization breaks down your work into categories: debugging, feature development, refactoring, documentation, and others. It provides insight into how you allocate your coding time. ### Dashboard 2: AI Interaction Patterns This dashboard focuses on how you interact with Claude's tools and capabilities. **Tool Usage Distribution (Pie Chart)** ```sql SELECT tool_name, total_calls FROM marts.agg_tool_efficiency ORDER BY total_calls DESC ``` See at a glance which tools dominate your workflow. Is it file reading? Code editing? Shell commands? The distribution often reveals opportunities to expand your usage of underutilized capabilities. **Tool Category Breakdown (Bar Chart)** ```sql SELECT tool_category, SUM(total_calls) as calls FROM marts.agg_tool_efficiency GROUP BY tool_category ORDER BY calls DESC ``` Aggregate tools into categories (file operations, search, shell, web) for a higher-level view of interaction patterns. **Top 10 Most Used Tools (Row Chart)** ```sql SELECT tool_name, total_calls, sessions_used FROM marts.agg_tool_efficiency ORDER BY popularity_rank LIMIT 10 ``` A ranked horizontal bar chart makes it easy to compare your most-used tools. The `sessions_used` column shows breadth of usage across different coding sessions. **Response Time Trends (Line Chart)** ```sql SELECT date_key, AVG(avg_response_time_seconds) as avg_response FROM marts.dim_sessions GROUP BY date_key ORDER BY date_key ``` Track Claude's response latency over time. Sudden increases might indicate complex queries or infrastructure issues. ### Dashboard 3: Project Insights This dashboard provides visibility into project-level activity and code changes. **Project Activity Ranking (Bar Chart)** ```sql SELECT project_id, total_messages, session_count FROM marts.dim_projects ORDER BY total_messages DESC LIMIT 15 ``` See which projects consume the most of your Claude interactions. This helps prioritize documentation and knowledge transfer for heavily-used codebases. **Project Status Distribution (Pie Chart)** ```sql SELECT activity_status, COUNT(*) as project_count FROM marts.dim_projects GROUP BY activity_status ``` Projects are classified as active, dormant, or archived based on recent activity. This visualization shows the health of your project portfolio. **Daily Code Changes (Area Chart)** ```sql SELECT date_key, file_reads, file_writes, file_edits FROM marts.agg_daily_summary WHERE has_activity = true ORDER BY date_key ``` Stacked area chart showing the volume of file operations over time. Large spikes in writes often correlate with feature implementations, while edit-heavy periods suggest refactoring work. **File Types Worked On (Pie Chart)** ```sql SELECT aggregation_key as file_type, total_operations FROM marts.agg_code_changes WHERE aggregation_level = 'by_file_type' ORDER BY total_operations DESC ``` Distribution of file operations by extension (`.ts`, `.py`, `.sql`, etc.). This reveals your technology stack focus over time. --- ## Self-Service Features for End Users The true power of Metabase lies in its self-service capabilities. Once the dashboards are set up, team members can explore data without SQL knowledge or engineering support. ### Visual Query Builder Metabase's visual query builder allows users to create questions through a point-and-click interface. Users select a table, choose columns, add filters, and pick a visualization type. Behind the scenes, Metabase generates optimized SQL. For example, a product manager could: 1. Select the `dim_projects` table 2. Choose `project_id` and `total_messages` columns 3. Add a filter for `activity_status = 'active'` 4. Sort by `total_messages` descending 5. Visualize as a bar chart No SQL required. ### Interactive Filters Add dashboard-level filters to enable dynamic exploration: **Date Range Filter** - Field: `date_key` - Default: Last 30 days - Allows users to analyze specific time periods **Project Filter** - Field: `project_id` - Type: Multi-select dropdown - Filter all dashboard cards simultaneously **Activity Level Filter** - Field: `activity_level` - Values: minimal, light, moderate, heavy - Focus on high-activity or low-activity periods These filters connect to all relevant questions on a dashboard, providing a unified exploration experience. ### Sharing and Collaboration Metabase offers several ways to share insights: - **Public links**: Generate shareable URLs for specific questions or dashboards - **Embedding**: Embed dashboards in internal tools or documentation - **Subscriptions**: Schedule email delivery of dashboard snapshots - **Slack integration**: Post dashboard updates to team channels For our Claude analytics use case, consider setting up a weekly email subscription that summarizes productivity metrics for each team member. ### Permissions Model Metabase supports granular permissions at the database, schema, and table levels. For a self-service analytics deployment, consider this structure: - **Admins**: Full access to all data and Metabase configuration - **Analysts**: Query access to all schemas, can create questions and dashboards - **Viewers**: Can view existing dashboards and filter data, but cannot create new questions This prevents accidental exposure of sensitive data while enabling broad access to insights. --- ## Deployment and Operations Running Metabase in production requires attention to reliability, performance, and data freshness. ### Starting the Stack Launch the complete analytics platform with a single command: ```bash cd analytics make up ``` This starts Prefect (orchestration), the analytics worker, PostgreSQL (Metabase metadata), and Metabase itself. Verify all services are healthy: ```bash docker-compose -f docker-compose.analytics.yml ps ``` You should see all containers in a `healthy` or `running` state. ### Health Monitoring Metabase exposes a health endpoint for monitoring: ```bash curl http://localhost:3001/api/health ``` A healthy response returns `{"status":"ok"}`. Integrate this with your monitoring system (Prometheus, Datadog, etc.) to alert on Metabase availability issues. ### Backup and Restore Two data stores require backup consideration: **PostgreSQL Metadata Database** Metabase stores all questions, dashboards, user accounts, and configuration in PostgreSQL. Back up this database regularly: ```bash docker exec metabase-db pg_dump -U metabase metabase > metabase_backup.sql ``` Restore with: ```bash cat metabase_backup.sql | docker exec -i metabase-db psql -U metabase metabase ``` **DuckDB Analytics Database** The DuckDB file contains your actual analytics data. Since dbt can regenerate this from source data, full backups are less critical. However, for faster recovery, periodically copy the database file: ```bash docker cp analytics-worker:/duckdb/analytics.db ./analytics_backup.db ``` ### Performance Tuning Several strategies improve Metabase performance with DuckDB: **Question Caching** Enable query caching in Metabase settings. For stable historical data, cache results for 1-6 hours. This dramatically reduces DuckDB query load. **Refresh Schedules** Configure appropriate refresh intervals based on data freshness requirements: | Dashboard Type | Refresh Interval | Rationale | |----------------|------------------|-----------| | Real-time activity | 1 hour | Balance freshness with load | | Historical trends | 6 hours | Data changes slowly | | Aggregates | Daily at 3 AM | After nightly dbt runs | **Pre-aggregation** Our dbt models already implement pre-aggregation (the `agg_*` tables). If you add new questions that perform expensive aggregations, consider adding corresponding dbt models rather than computing on the fly. --- ## Series Conclusion: The Complete Data Journey We have reached the end of our seven-article journey. Let us step back and appreciate what we have built together. ```mermaid flowchart LR subgraph "Article 1-3: Data Capture" A[JSONL Files] --> B[File Watcher] B --> C[SQLite Buffer] C --> D[MongoDB] end subgraph "Article 4: Extraction" D --> E[Python Extractor] E --> F[Parquet Files] end subgraph "Article 5: Loading" F --> G[DuckDB Loader] G --> H[(DuckDB)] end subgraph "Article 6: Transformation" H --> I[dbt Bronze] I --> J[dbt Silver] J --> K[dbt Gold] end subgraph "Article 7: Visualization" K --> L[Metabase] L --> M[Dashboards] end ``` **Article 1-3** covered the sync service that watches Claude's JSONL log files, buffers changes in SQLite, and syncs to MongoDB. This gave us durable, queryable storage for conversation history. **Article 4** introduced the Python extraction layer using Prefect for orchestration. We built incremental extraction logic that efficiently pulls new data from MongoDB without full table scans. **Article 5** explored DuckDB as our analytics warehouse. We implemented a loader that converts Parquet files into queryable tables, leveraging DuckDB's columnar storage for analytics performance. **Article 6** was our deepest dive, covering dbt transformations using the medallion architecture. Bronze, silver, and gold layers progressively refined raw data into analytics-ready star schemas. **Article 7** (this article) completed the stack with Metabase dashboards. We connected to DuckDB, built three comprehensive dashboards, and explored self-service features that empower non-technical users. ### What We Built The complete platform provides: - **Real-time data capture** from Claude AI sessions - **Durable storage** in MongoDB for operational queries - **Columnar analytics** in DuckDB for fast aggregations - **Semantic modeling** through dbt transformations - **Self-service visualization** via Metabase dashboards - **Orchestrated pipelines** with Prefect scheduling All of this runs locally in Docker containers, requiring no cloud services or external dependencies. ### Future Enhancements Several directions could extend this platform: 1. **Alerting**: Configure Metabase alerts when metrics exceed thresholds (e.g., session duration drops significantly) 2. **Semantic Layer**: Add a metrics layer using dbt Semantic Layer or Cube.js for consistent metric definitions 3. **Machine Learning**: Export data to train models that predict task complexity or suggest optimal tools 4. **Multi-user Analytics**: Track metrics across a development team rather than individual usage 5. **Cost Attribution**: Correlate Claude usage with API costs for budgeting and optimization ### Final Thoughts Building an analytics platform is an investment. Each layer required careful design decisions, from the choice of SQLite for buffering to DuckDB for warehousing to Metabase for visualization. But the payoff is substantial: complete visibility into how you use AI-assisted development tools. More importantly, this architecture is not specific to Claude. The patterns we explored, which include real-time sync, medallion architecture, and self-service BI, apply to any analytics use case. Swap MongoDB for PostgreSQL, change the dbt models, and you have an analytics platform for a SaaS product, an IoT system, or an e-commerce store. I hope this series has demystified the modern data stack. The tools are mature, the patterns are proven, and the barriers to entry have never been lower. Build something great. --- **Suggested Tags:** Metabase, DuckDB, Data Analytics, Business Intelligence, Data Engineering --- *This is Article 7 of 7 in the "Building a Modern Analytics Platform" series. Read the complete series to learn how to build production-ready data infrastructure from scratch.*

Data Transformation with dbt: From Raw Logs to Business Intelligence

Thu, 01 Jan 2026 00:00:00 GMT

*Transform messy conversation logs into a polished analytics warehouse using dbt's medallion architecture pattern* --- You have data. Lots of it. Raw JSON lines streaming into a database, timestamped entries piling up, and somewhere in that chaos lies the answer to questions you haven't even thought to ask yet. The gap between "we have data" and "we have insights" is where most analytics projects stall. In our previous articles, we built the foundation: Article 4 introduced the analytics architecture that extracts conversation data from MongoDB, and Article 5 covered how the loader gets that data into DuckDB. But raw data in a database is like crude oil in a barrel. It needs refining before it becomes useful. This is where dbt enters the picture. dbt (data build tool) transforms how we think about data transformation. Instead of one-off scripts or brittle stored procedures, we get version-controlled SQL models, automated testing, and documentation that lives alongside our code. In this article, we'll walk through a complete dbt project that transforms raw Claude Code conversation logs into a star schema ready for business intelligence dashboards. ## The dbt Project Structure Every dbt project follows a predictable structure. Understanding this layout is your first step toward building maintainable transformation pipelines. ``` analytics/dbt/ ├── dbt_project.yml # Project configuration ├── profiles.yml # Connection profiles ├── packages.yml # External dependencies ├── models/ │ ├── staging/ # Bronze layer │ ├── intermediate/ # Silver layer │ └── marts/ # Gold layer ├── seeds/ # Static reference data ├── macros/ # Reusable SQL snippets ├── snapshots/ # SCD Type 2 tracking └── tests/ # Custom data tests ``` ### Project Configuration: dbt_project.yml The `dbt_project.yml` file is the heart of your project. It defines how models should be materialized, which schemas they belong to, and project-wide variables. ```yaml name: 'claude_analytics' version: '1.0.0' config-version: 2 profile: 'claude_analytics' model-paths: ["models"] seed-paths: ["seeds"] macro-paths: ["macros"] snapshot-paths: ["snapshots"] test-paths: ["tests"] models: claude_analytics: # Staging layer (Bronze) - Views for quick iteration staging: +materialized: view +schema: staging +tags: ['staging', 'bronze'] # Intermediate layer (Silver) - Enriched data intermediate: +materialized: view +schema: intermediate +tags: ['intermediate', 'silver'] # Marts layer (Gold) - Tables for performance marts: +materialized: table +schema: marts +tags: ['marts', 'gold'] aggregates: +materialized: table +tags: ['aggregates'] ``` Notice the materialization strategy: staging and intermediate layers use views for fast iteration during development, while marts use tables for query performance. This pattern lets you experiment quickly in early layers while ensuring your BI tools hit optimized tables. The configuration also includes task classification patterns as variables, making it easy to adjust business logic without modifying SQL: ```yaml vars: task_patterns: bug_fix: ['bug', 'fix', 'error', 'issue', 'broken'] feature: ['add', 'create', 'implement', 'new', 'build'] refactor: ['refactor', 'restructure', 'clean up', 'simplify'] ``` ### Connection Profiles: profiles.yml The `profiles.yml` file defines how dbt connects to your data warehouse. For our DuckDB-based analytics, we configure three environments: ```yaml claude_analytics: target: dev outputs: dev: type: duckdb path: "{{ env_var('DUCKDB_PATH', '/duckdb/analytics.db') }}" threads: 4 extensions: - parquet prod: type: duckdb path: "{{ env_var('DUCKDB_PATH', '/duckdb/analytics.db') }}" threads: 8 extensions: - parquet test: type: duckdb path: ":memory:" threads: 1 extensions: - parquet ``` The test profile uses an in-memory database for fast CI/CD runs. Environment variables keep sensitive paths out of version control. ### External Packages: packages.yml dbt's package ecosystem provides battle-tested utilities. We use two essential packages: ```yaml packages: - package: dbt-labs/dbt_utils version: ">=1.1.0" - package: calogica/dbt_date version: ">=0.10.0" ``` `dbt_utils` provides surrogate key generation, pivot operations, and schema testing utilities. `dbt_date` simplifies date dimension creation. Install them with `dbt deps`. ## The Medallion Architecture Think of data transformation like refining precious metals. You start with raw ore (Bronze), process it into standardized ingots (Silver), and finally craft finished products (Gold). Each layer adds value and removes impurities. ```mermaid flowchart TB subgraph Bronze["Bronze Layer (Staging)"] raw[(raw.conversations)] stg_conv[stg_conversations] stg_msg[stg_messages] stg_tool[stg_tool_calls] end subgraph Silver["Silver Layer (Intermediate)"] int_msg[int_messages_enriched] int_sess[int_sessions_computed] int_tool[int_tool_usage] end subgraph Gold["Gold Layer (Marts)"] subgraph Dimensions dim_date[dim_date] dim_proj[dim_projects] dim_sess[dim_sessions] dim_tools[dim_tools] end subgraph Facts fct_msg[fct_messages] fct_tool[fct_tool_calls] fct_file[fct_file_operations] end subgraph Aggregates agg_daily[agg_daily_summary] agg_hour[agg_hourly_activity] agg_tool[agg_tool_efficiency] end end raw --> stg_conv stg_conv --> stg_msg stg_conv --> stg_tool stg_msg --> int_msg stg_conv --> int_sess stg_msg --> int_sess stg_tool --> int_sess stg_tool --> int_tool int_msg --> dim_sess int_sess --> dim_sess int_sess --> dim_proj int_tool --> dim_tools int_msg --> fct_msg int_tool --> fct_tool int_tool --> fct_file fct_msg --> agg_daily fct_tool --> agg_daily fct_msg --> agg_hour fct_tool --> agg_hour fct_tool --> agg_tool ``` **Bronze (Staging):** Minimal transformation. Clean column names, handle nulls, cast types. The goal is a reliable foundation, not business logic. **Silver (Intermediate):** Apply business rules. Classify tasks, compute session metrics, enrich with reference data. This layer answers "what happened" with context. **Gold (Marts):** Optimize for consumption. Star schema design enables fast BI queries. Aggregates pre-compute expensive operations. Why this pattern works: each layer has a single responsibility. When business logic changes, you modify the Silver layer. When new sources arrive, you add to Bronze. When dashboards need new metrics, you extend Gold. Changes stay isolated. ## Staging Models: The Bronze Layer The staging layer is your data's first stop. Here we define sources, clean up raw data, and establish the foundation for everything downstream. ### Defining Sources Before transforming data, we declare where it comes from. The `_sources.yml` file documents our raw data and enables freshness checks: ```yaml version: 2 sources: - name: raw description: "Raw data loaded from MongoDB via Parquet extraction" schema: raw freshness: warn_after: {count: 24, period: hour} error_after: {count: 48, period: hour} loaded_at_field: extracted_at tables: - name: conversations description: | Raw conversation entries extracted from MongoDB. Each row represents a single message, tool call, or entry from a Claude Code conversation session. columns: - name: _id description: "Unique identifier from MongoDB" tests: - unique - not_null - name: type description: "Entry type: user, assistant, tool_use, tool_result" tests: - not_null ``` The freshness configuration alerts us when data pipelines stall. If no new data arrives within 24 hours, we get a warning. After 48 hours, builds fail. ### stg_conversations: The Foundation Model This model cleans and standardizes every conversation entry: ```sql -- Staging model for conversations -- Basic cleaning, type casting, and null handling {{ config( materialized='view', tags=['staging', 'bronze'] ) }} with source as ( select * from {{ source('raw', 'conversations') }} ), cleaned as ( select -- Primary key _id as conversation_id, -- Core attributes with null handling coalesce(type, 'unknown') as entry_type, session_id, project_id, -- Timestamps with fallback chain timestamp as original_timestamp, coalesce(timestamp, ingested_at, extracted_at) as effective_timestamp, ingested_at, extracted_at, -- Message content message_role, message_content, message_raw, -- Source tracking source_file, date as partition_date, -- Computed fields case when message_content is not null then length(message_content) else 0 end as content_length, -- Boolean flags for common queries type in ('user', 'assistant') as is_message, type in ('tool_use', 'tool_result') as is_tool_related from source where _id is not null ) select * from cleaned ``` Key patterns here: we rename `_id` to `conversation_id` for clarity, create an `effective_timestamp` with a fallback chain, and add boolean flags that simplify downstream filtering. The `coalesce` function handles missing values gracefully. ### stg_messages: Extracting Conversations This model filters to actual messages and adds sequence information: ```sql {{ config( materialized='view', tags=['staging', 'bronze'] ) }} with conversations as ( select * from {{ ref('stg_conversations') }} ), messages_only as ( select conversation_id, session_id, project_id, effective_timestamp, partition_date, entry_type, coalesce(message_role, entry_type) as role, message_content, content_length, -- Sequence number within session row_number() over ( partition by session_id order by effective_timestamp ) as message_sequence, -- Previous message context lag(message_role) over ( partition by session_id order by effective_timestamp ) as previous_role, -- Response time calculation extract(epoch from ( effective_timestamp - lag(effective_timestamp) over ( partition by session_id order by effective_timestamp ) )) as seconds_since_previous, source_file from conversations where is_message = true and message_content is not null ) select *, -- Classify conversation turn type case when previous_role is null then 'conversation_start' when role = 'user' and previous_role = 'assistant' then 'follow_up' when role = 'assistant' and previous_role = 'user' then 'response' else 'continuation' end as turn_type from messages_only ``` Window functions shine here. `row_number()` gives us message ordering within sessions. `lag()` lets us reference the previous message to calculate response times and classify turn types. ### stg_tool_calls: Parsing AI Tool Usage Tool calls require pattern matching to extract tool names from content: ```sql with tool_entries as ( select * from {{ ref('stg_conversations') }} where is_tool_related = true ), parsed_tools as ( select conversation_id, session_id, project_id, effective_timestamp, partition_date, entry_type, -- Extract tool name via pattern matching case when message_content like '%Read%' then 'Read' when message_content like '%Write%' then 'Write' when message_content like '%Edit%' then 'Edit' when message_content like '%Bash%' then 'Bash' when message_content like '%Glob%' then 'Glob' when message_content like '%Grep%' then 'Grep' -- ... additional tools else 'unknown' end as tool_name, entry_type = 'tool_use' as is_invocation, entry_type = 'tool_result' as is_result, row_number() over ( partition by session_id order by effective_timestamp ) as tool_sequence from tool_entries ) select *, -- Pair invocations with results for timing lead(effective_timestamp) over ( partition by session_id, tool_name order by effective_timestamp ) as next_same_tool_timestamp from parsed_tools ``` The `lead()` function looks ahead to find the next occurrence of the same tool, enabling execution time estimation when we pair `tool_use` with its corresponding `tool_result`. ## Intermediate Models: The Silver Layer The Silver layer is where business logic lives. We enrich data with classifications, compute aggregates, and join with reference data. ### int_messages_enriched: Task Classification This model adds temporal features and classifies tasks based on message content: ```sql {{ config( materialized='view', tags=['intermediate', 'silver'] ) }} with messages as ( select * from {{ ref('stg_messages') }} ), enriched as ( select conversation_id, session_id, project_id, role, message_content, content_length, message_sequence, turn_type, effective_timestamp, partition_date, -- Temporal features extract(hour from effective_timestamp) as hour_of_day, extract(dow from effective_timestamp) as day_of_week, -- Time-of-day classification case when extract(hour from effective_timestamp) between 6 and 11 then 'morning' when extract(hour from effective_timestamp) between 12 and 17 then 'afternoon' when extract(hour from effective_timestamp) between 18 and 21 then 'evening' else 'night' end as time_of_day, case when extract(dow from effective_timestamp) in (0, 6) then 'weekend' else 'weekday' end as day_type, -- Task category classification case when lower(message_content) like '%fix%' or lower(message_content) like '%bug%' or lower(message_content) like '%error%' then 'bug_fix' when lower(message_content) like '%implement%' or lower(message_content) like '%create%' or lower(message_content) like '%add%feature%' then 'feature' when lower(message_content) like '%refactor%' or lower(message_content) like '%clean up%' then 'refactor' when lower(message_content) like '%test%' or lower(message_content) like '%spec%' then 'testing' when lower(message_content) like '%document%' or lower(message_content) like '%readme%' then 'documentation' else 'other' end as task_category, -- Content indicators message_content like '%```%' as has_code_block, message_content like '%?%' as is_question from messages ) select * from enriched ``` The task classification uses pattern matching on message content. This heuristic approach works surprisingly well for understanding what developers are asking an AI assistant to do. ### int_sessions_computed: Session-Level Metrics This model aggregates everything we know about a session: ```sql with conversations as ( select * from {{ ref('stg_conversations') }} ), messages as ( select * from {{ ref('stg_messages') }} ), tool_calls as ( select * from {{ ref('stg_tool_calls') }} ), -- Session boundaries session_boundaries as ( select session_id, project_id, min(effective_timestamp) as session_start, max(effective_timestamp) as session_end, count(*) as total_entries from conversations where session_id is not null group by session_id, project_id ), -- Message statistics per session message_stats as ( select session_id, count(*) as message_count, sum(case when role = 'user' then 1 else 0 end) as user_message_count, sum(case when role = 'assistant' then 1 else 0 end) as assistant_message_count, avg(seconds_since_previous) filter (where seconds_since_previous > 0) as avg_response_time_seconds from messages group by session_id ), -- Tool statistics per session tool_stats as ( select session_id, count(*) as tool_call_count, count(distinct tool_name) as unique_tools_used, mode() within group (order by tool_name) as primary_tool from tool_calls group by session_id ), -- Combine all metrics sessions_computed as ( select sb.session_id, sb.project_id, sb.session_start, sb.session_end, -- Duration extract(epoch from (sb.session_end - sb.session_start)) as duration_seconds, extract(epoch from (sb.session_end - sb.session_start)) / 60.0 as duration_minutes, -- Messages coalesce(ms.message_count, 0) as message_count, coalesce(ms.user_message_count, 0) as user_message_count, coalesce(ms.assistant_message_count, 0) as assistant_message_count, -- Tools coalesce(ts.tool_call_count, 0) as tool_call_count, coalesce(ts.unique_tools_used, 0) as unique_tools_used, ts.primary_tool, -- Classifications case when extract(epoch from (sb.session_end - sb.session_start)) < 60 then 'quick' when extract(epoch from (sb.session_end - sb.session_start)) < 600 then 'short' when extract(epoch from (sb.session_end - sb.session_start)) < 3600 then 'medium' else 'long' end as session_duration_category, case when coalesce(ms.message_count, 0) <= 5 then 'minimal' when coalesce(ms.message_count, 0) <= 20 then 'light' when coalesce(ms.message_count, 0) <= 50 then 'moderate' else 'heavy' end as activity_level from session_boundaries sb left join message_stats ms on sb.session_id = ms.session_id left join tool_stats ts on sb.session_id = ts.session_id ) select * from sessions_computed ``` The `mode()` aggregate function finds the most frequently used tool per session. This tells us what the primary activity was, whether file editing, shell commands, or web searches. ### int_tool_usage: Enriched Tool Data This model joins tool calls with our seed data for category classification: ```sql with tool_calls as ( select * from {{ ref('stg_tool_calls') }} ), tool_categories as ( select * from {{ ref('tool_categories') }} ), enriched_tools as ( select tc.conversation_id, tc.session_id, tc.tool_name, tc.is_invocation, tc.is_result, tc.effective_timestamp, -- Join with seed data coalesce(cat.tool_category, 'unknown') as tool_category, cat.description as tool_description, -- Extract file paths from content case when tc.tool_name in ('Read', 'Write', 'Edit', 'Glob') and tc.tool_content like '%/%' then regexp_extract(tc.tool_content, '([/][a-zA-Z0-9_./-]+)', 1) else null end as file_path, -- Classify operation types tc.tool_name in ('Read', 'Write', 'Edit', 'NotebookEdit', 'MultiEdit') as is_file_operation, tc.tool_name in ('Glob', 'Grep', 'WebSearch') as is_search_operation, tc.tool_name = 'Bash' as is_shell_command from tool_calls tc left join tool_categories cat on tc.tool_name = cat.tool_name ) select * from enriched_tools ``` The join with `tool_categories` seed data adds human-readable descriptions and standardized categories without hardcoding them in SQL. ## Mart Models: The Gold Layer The Gold layer delivers a star schema optimized for BI tools. Dimension tables describe entities; fact tables record events. ```mermaid erDiagram dim_date ||--o{ fct_messages : "date_key" dim_date ||--o{ fct_tool_calls : "date_key" dim_date ||--o{ fct_file_operations : "date_key" dim_sessions ||--o{ fct_messages : "session_key" dim_sessions ||--o{ fct_tool_calls : "session_key" dim_sessions ||--o{ fct_file_operations : "session_key" dim_tools ||--o{ fct_tool_calls : "tool_key" dim_tools ||--o{ fct_file_operations : "tool_key" dim_projects ||--o{ dim_sessions : "project_id" dim_date { date date_key PK int year int quarter int month int day_of_week boolean is_weekend } dim_sessions { string session_key PK string session_id string project_id FK date date_key FK timestamp session_start int duration_minutes string activity_level } dim_projects { string project_key PK string project_id timestamp first_seen timestamp last_active int session_count string activity_status } dim_tools { string tool_key PK string tool_name string tool_category int total_calls int popularity_rank } fct_messages { string message_key PK string session_key FK date date_key FK string role int content_length string task_category } fct_tool_calls { string tool_call_key PK string session_key FK string tool_key FK date date_key FK string tool_name boolean is_file_operation } fct_file_operations { string file_operation_key PK string session_key FK string tool_key FK date date_key FK string operation_type string file_type_category } ``` ### dim_date: The Date Dimension Every analytics warehouse needs a date dimension. DuckDB's `generate_series` makes this straightforward: ```sql {{ config( materialized='table', tags=['marts', 'gold', 'dimension'] ) }} with date_boundaries as ( select min(partition_date) as min_date, max(partition_date) as max_date from {{ ref('stg_conversations') }} ), date_spine as ( select generate_series::date as date_day from date_boundaries, generate_series( date_boundaries.min_date, date_boundaries.max_date, interval '1 day' ) ), dim_date as ( select date_day as date_key, -- ISO components extract(year from date_day) as year, extract(quarter from date_day) as quarter, extract(month from date_day) as month, extract(dow from date_day) as day_of_week, -- Readable labels strftime(date_day, '%Y-%m') as year_month, strftime(date_day, '%B') as month_name, strftime(date_day, '%A') as day_name, -- Week boundaries date_trunc('week', date_day)::date as week_start_date, -- Boolean flags extract(dow from date_day) in (0, 6) as is_weekend, date_day = current_date as is_today, -- Relative indicators current_date - date_day as days_ago from date_spine ) select * from dim_date order by date_key ``` The dimension generates one row per day between the first and last data points. Boolean flags like `is_weekend` and `is_today` simplify common filter patterns in dashboards. ### Fact Tables with Incremental Materialization Fact tables use incremental materialization to avoid reprocessing all history: ```sql {{ config( materialized='incremental', unique_key='message_key', tags=['marts', 'gold', 'fact'] ) }} with messages as ( select * from {{ ref('int_messages_enriched') }} {% if is_incremental() %} where partition_date >= (select max(date_key) - interval '1 day' from {{ this }}) {% endif %} ), dim_sessions as ( select session_key, session_id from {{ ref('dim_sessions') }} ), fct_messages as ( select md5(m.conversation_id) as message_key, -- Dimension foreign keys ds.session_key, m.partition_date as date_key, m.project_id, -- Message attributes m.role, m.turn_type, m.message_sequence, m.content_length, m.task_category, m.time_of_day, -- Flags m.has_code_block, m.is_question, m.effective_timestamp from messages m left join dim_sessions ds on m.session_id = ds.session_id ) select * from fct_messages ``` The `{% if is_incremental() %}` block only processes recent data after the initial load. The one-day overlap (`max(date_key) - interval '1 day'`) handles late-arriving data gracefully. ## Aggregate Tables: Pre-Computed Metrics Aggregate tables trade storage for query speed. They pre-compute metrics that dashboards request repeatedly. ### agg_daily_summary: Time-Series Metrics ```sql {{ config( materialized='table', tags=['marts', 'gold', 'aggregate'] ) }} with sessions as ( select * from {{ ref('dim_sessions') }} ), messages as ( select * from {{ ref('fct_messages') }} ), tool_calls as ( select * from {{ ref('fct_tool_calls') }} ), dim_date as ( select * from {{ ref('dim_date') }} ), daily_sessions as ( select date_key, count(*) as session_count, count(distinct project_id) as active_projects, sum(duration_minutes) as total_session_minutes, avg(duration_minutes) as avg_session_minutes from sessions group by date_key ), daily_messages as ( select date_key, count(*) as message_count, sum(case when role = 'user' then 1 else 0 end) as user_messages, sum(case when has_code_block then 1 else 0 end) as messages_with_code from messages group by date_key ), daily_tools as ( select date_key, count(*) as tool_call_count, count(distinct tool_name) as unique_tools_used from tool_calls group by date_key ) select d.date_key, d.year, d.month, d.day_name, d.is_weekend, coalesce(ds.session_count, 0) as session_count, coalesce(ds.active_projects, 0) as active_projects, coalesce(dm.message_count, 0) as message_count, coalesce(dt.tool_call_count, 0) as tool_call_count, coalesce(ds.session_count, 0) > 0 as has_activity, current_timestamp as computed_at from dim_date d left join daily_sessions ds on d.date_key = ds.date_key left join daily_messages dm on d.date_key = dm.date_key left join daily_tools dt on d.date_key = dt.date_key order by d.date_key desc ``` This aggregate powers daily trend charts. The `computed_at` timestamp helps debug stale data issues. ### agg_hourly_activity: Heatmap Data For work pattern analysis, we need a 24x7 activity matrix: ```sql -- Create complete hour x day matrix hour_day_matrix as ( select h.hour_of_day, d.day_of_week from (select unnest(generate_series(0, 23)) as hour_of_day) h cross join (select unnest(generate_series(0, 6)) as day_of_week) d ), hourly_activity as ( select hdm.hour_of_day, hdm.day_of_week, case hdm.day_of_week when 0 then 'Sunday' when 1 then 'Monday' -- ... etc end as day_name, lpad(hdm.hour_of_day::text, 2, '0') || ':00' as hour_label, coalesce(hm.message_count, 0) + coalesce(ht.tool_call_count, 0) as total_activity from hour_day_matrix hdm left join hourly_messages hm on hdm.hour_of_day = hm.hour_of_day and hdm.day_of_week = hm.day_of_week left join hourly_tools ht on hdm.hour_of_day = ht.hour_of_day and hdm.day_of_week = ht.day_of_week ) ``` The cross join ensures every hour-day combination exists, even with zero activity. This prevents gaps in heatmap visualizations. ## Seeds: Static Reference Data Seeds are CSV files that dbt loads as tables. They're perfect for reference data that changes infrequently. ### tool_categories.csv ```csv tool_name,tool_category,description Read,file_operations,Read file contents Write,file_operations,Write file contents Edit,file_operations,Edit file contents NotebookEdit,file_operations,Edit Jupyter notebook cells MultiEdit,file_operations,Multiple file edits in one operation Bash,shell,Execute shell commands Glob,search,Search files by glob pattern Grep,search,Search file contents with regex Task,agent,Spawn sub-agent for complex tasks TodoRead,planning,Read todo list TodoWrite,planning,Write/update todo list WebFetch,network,Fetch web page content WebSearch,network,Search the web AskFollowupQuestion,interaction,Ask user for clarification AttemptCompletion,interaction,Signal task completion ``` The `_seeds.yml` file documents and tests seed data: ```yaml version: 2 seeds: - name: tool_categories description: "Reference table mapping Claude Code tool names to categories" columns: - name: tool_name tests: - unique - not_null - name: tool_category tests: - not_null - accepted_values: values: - file_operations - shell - search - agent - planning - network - interaction ``` Load seeds with `dbt seed`. They reload on every run, making updates simple: edit the CSV and run `dbt seed` again. ## Testing Strategy dbt tests validate data quality at build time. Tests are defined in schema YAML files alongside model documentation. ### Schema Tests Each model has tests defined in its `_schema.yml`: ```yaml version: 2 models: - name: stg_conversations columns: - name: conversation_id tests: - unique - not_null - name: entry_type tests: - not_null - accepted_values: values: - user - assistant - tool_use - tool_result - system - summary - unknown ``` ### Relationship Tests Referential integrity between dimensions and facts: ```yaml - name: fct_messages columns: - name: session_key tests: - relationships: to: ref('dim_sessions') field: session_key - name: date_key tests: - relationships: to: ref('dim_date') field: date_key ``` ### Running Tests ```bash # Test everything dbt test # Test specific model dbt test --select stg_conversations # Test only marts layer dbt test --select marts ``` Failed tests produce clear error messages pointing to the exact rows that violate constraints. ## DuckDB Integration DuckDB provides an excellent development experience for analytics workloads. Its columnar storage and vectorized execution make it fast, while file-based databases eliminate infrastructure overhead. ### DuckDB-Specific Functions Our models use several DuckDB functions: - `generate_series()` - Creates date spines without helper tables - `strftime()` - Formats dates as strings - `regexp_extract()` - Extracts patterns from text - `mode() within group` - Finds most frequent values - `percentile_cont()` - Calculates percentiles ### Profile Configuration The development profile enables the Parquet extension for reading source data: ```yaml dev: type: duckdb path: "{{ env_var('DUCKDB_PATH', '/duckdb/analytics.db') }}" threads: 4 extensions: - parquet ``` Multiple threads parallelize query execution. For production, increase this based on available CPU cores. ## Putting It All Together With all models in place, building the entire warehouse is a single command: ```bash # Install packages dbt deps # Load seed data dbt seed # Build all models dbt run # Run all tests dbt test # Generate documentation dbt docs generate dbt docs serve ``` The dependency graph ensures models build in the correct order. Staging first, then intermediate, then marts. dbt figures this out from the `{{ ref() }}` calls in your SQL. --- The medallion architecture provides a clean mental model for organizing transformation logic. Bronze cleans, Silver enriches, Gold optimizes. Each layer has one job. dbt makes this pattern practical by handling dependencies, testing, and documentation. Your transformation logic lives in version control, reviewed like application code, tested before deployment. The star schema at the end powers dashboards that answer real questions: When do developers use AI assistants most? Which tools drive productivity? How do session patterns change over time? Start with your own data. Identify the Bronze-Silver-Gold layers. Define your dimensions and facts. Let dbt handle the rest. --- **Suggested Tags:** dbt, Data Engineering, Analytics, SQL, DuckDB

Building an ETL Pipeline with Python, Prefect, and DuckDB

Wed, 31 Dec 2025 00:00:00 GMT

*A deep dive into production-ready data extraction, loading, and orchestration* --- This is **Article 5** in a 7-part series on building a complete analytics platform for Claude Code conversation logs. In [Article 4](/article-4-analytics-architecture), we explored the high-level architecture of our analytics module, including the medallion pattern (Bronze, Silver, Gold layers) and how dbt transformations create business-ready data models. Now we are going to get our hands dirty with the Python code that powers the ELT pipeline. We will walk through every module in the `analytics/analytics/` package, examining how each component fulfills a single responsibility while working together as a cohesive system. By the end of this article, you will understand how to: - Build type-safe configuration with Pydantic Settings - Extract data from MongoDB with incremental high-water-mark tracking - Load Parquet files into DuckDB with upsert semantics - Orchestrate everything with Prefect flows and tasks - Validate data quality with Great Expectations Let us dive in. --- ## Module Structure Overview Before examining individual files, let us understand how the package is organized. Each module has a single, clear responsibility: ``` analytics/ ├── analytics/ │ ├── __init__.py # Package initialization, exports │ ├── config.py # Configuration management (Pydantic) │ ├── extractor.py # MongoDB → Parquet extraction │ ├── loader.py # Parquet → DuckDB loading │ ├── quality.py # Great Expectations integration │ ├── cli.py # Command-line interface (Typer) │ └── flows/ │ ├── __init__.py # Flow exports │ ├── main_pipeline.py # Prefect flow definitions │ └── deployment.py # Deployment helpers ├── prefect.yaml # Prefect deployment config └── pyproject.toml # Package metadata ``` The data flows through these modules in a clear sequence: ```mermaid graph LR A[config.py] --> B[extractor.py] A --> C[loader.py] B --> D[Parquet Files] D --> C C --> E[DuckDB] F[flows/main_pipeline.py] --> B F --> C F --> G[dbt] H[cli.py] --> F H --> B H --> C I[quality.py] --> E ``` The `config.py` module provides settings to all other modules. The `extractor.py` pulls from MongoDB and writes Parquet. The `loader.py` reads Parquet and loads into DuckDB. The `flows/` package orchestrates these steps with Prefect. And `cli.py` provides human-friendly commands for all operations. --- ## Configuration Management with Pydantic Settings Good configuration management is the foundation of any production system. The `config.py` module uses Pydantic Settings to provide type-safe, validated configuration with automatic environment variable loading. ### Nested Settings Classes Rather than one monolithic settings class, the configuration is organized into focused, nested classes: ```python from pydantic import Field, field_validator from pydantic_settings import BaseSettings, SettingsConfigDict class MongoSettings(BaseSettings): """MongoDB source configuration.""" model_config = SettingsConfigDict( env_prefix="MONGO_", env_file=".env.analytics", env_file_encoding="utf-8", extra="ignore", ) uri: str = Field( default="mongodb://localhost:27017", description="MongoDB connection URI", ) db: str = Field( default="claude_logs", description="Database name containing conversation logs", ) collection: str = Field( default="conversations", description="Collection name with conversation entries", ) ``` Each settings class has its own environment variable prefix. The `MongoSettings` class reads `MONGO_URI`, `MONGO_DB`, and `MONGO_COLLECTION` from the environment. This prevents naming collisions and makes configuration self-documenting. The pattern continues for other concerns: ```python class DuckDBSettings(BaseSettings): """DuckDB target configuration.""" model_config = SettingsConfigDict(env_prefix="DUCKDB_") path: Path = Field( default=Path("/duckdb/analytics.db"), description="Path to DuckDB database file", ) threads: int = Field( default=4, ge=1, le=32, description="Number of threads for DuckDB queries", ) ``` Notice the `ge=1, le=32` constraints on `threads`. Pydantic validates these at load time, failing fast if someone configures an invalid value. ### The Main Settings Aggregator All nested settings come together in the main `Settings` class: ```python class Settings(BaseSettings): """Main settings class that aggregates all configuration sections.""" model_config = SettingsConfigDict( env_file=".env.analytics", env_file_encoding="utf-8", extra="ignore", ) # Nested settings mongo: MongoSettings = Field(default_factory=MongoSettings) duckdb: DuckDBSettings = Field(default_factory=DuckDBSettings) data: DataSettings = Field(default_factory=DataSettings) pipeline: PipelineSettings = Field(default_factory=PipelineSettings) prefect: PrefectSettings = Field(default_factory=PrefectSettings) dbt: DbtSettings = Field(default_factory=DbtSettings) great_expectations: GreatExpectationsSettings = Field( default_factory=GreatExpectationsSettings ) logging: LoggingSettings = Field(default_factory=LoggingSettings) alerting: AlertingSettings = Field(default_factory=AlertingSettings) def setup(self) -> None: """Initialize settings: create directories, configure logging.""" self.data.ensure_directories() ``` ### Singleton Pattern with lru_cache Settings should be loaded once and reused. The `get_settings()` function uses `lru_cache` to ensure this: ```python from functools import lru_cache @lru_cache def get_settings() -> Settings: """Get cached settings instance.""" settings = Settings() settings.setup() return settings ``` Any module can call `get_settings()` and receive the same cached instance. This avoids repeated file I/O and ensures consistent configuration across the application. --- ## Extractor Deep Dive: MongoDB to Parquet The `extractor.py` module is responsible for pulling data from MongoDB and writing it to Parquet files. This is the "E" in our ELT pipeline. Let us examine its key components. ### PyArrow Schema Definition The extractor defines a strict schema for the output Parquet files: ```python import pyarrow as pa CONVERSATION_SCHEMA = pa.schema([ # Primary identifiers ("_id", pa.string()), ("type", pa.string()), ("session_id", pa.string()), ("project_id", pa.string()), # Timestamps ("timestamp", pa.timestamp("us", tz="UTC")), ("ingested_at", pa.timestamp("us", tz="UTC")), ("extracted_at", pa.timestamp("us", tz="UTC")), # Message content (flattened) ("message_role", pa.string()), ("message_content", pa.string()), ("message_raw", pa.string()), # Source tracking ("source_file", pa.string()), # Partitioning ("date", pa.date32()), ]) ``` Defining the schema upfront has several benefits: 1. **Type safety**: PyArrow validates data against the schema at write time 2. **Documentation**: The schema serves as a contract between extraction and loading 3. **Performance**: DuckDB can read typed Parquet more efficiently than schemaless JSON ### Document Transformation MongoDB documents are flexible, but Parquet needs flat, typed columns. The `DocumentTransformer` class handles this translation: ```python class DocumentTransformer: """Transforms MongoDB documents into flat structure for Parquet.""" @staticmethod def flatten_message(message: Any) -> tuple[str | None, str | None, str | None]: """Flatten the message field into role, content, and raw JSON.""" if message is None: return None, None, None if isinstance(message, str): return None, message, None if isinstance(message, dict): role = message.get("role") content = message.get("content") # Handle content that might be a list of blocks if isinstance(content, list): text_parts = [] for block in content: if isinstance(block, dict): if block.get("type") == "text": text_parts.append(block.get("text", "")) elif block.get("type") == "tool_use": text_parts.append(f"[tool_use: {block.get('name', 'unknown')}]") content = "\n".join(text_parts) if text_parts else None raw_json = json.dumps(message, default=str) if message else None return role, content, raw_json return None, None, json.dumps(message, default=str) ``` The transformer handles the three common message formats: 1. **Simple strings**: Stored directly as content 2. **Objects with role/content**: Extracted to separate columns 3. **Content block arrays**: Text blocks joined, tool usage summarized The original message is also preserved in `message_raw` for cases where downstream analysis needs the full structure. ### High Water Mark Tracking For incremental extraction, we need to track the last successfully extracted timestamp. The `HighWaterMark` class manages this: ```python class HighWaterMark: """Tracks last extraction timestamp for incremental extraction.""" def __init__(self, file_path: Path): self.file_path = file_path def get(self) -> datetime | None: """Get the last extraction timestamp.""" if not self.file_path.exists(): return None try: data = json.loads(self.file_path.read_text()) ts = data.get("last_extracted_at") if ts: return datetime.fromisoformat(ts) except (json.JSONDecodeError, ValueError) as e: logger.warning(f"Failed to read high water mark: {e}") return None def set(self, timestamp: datetime) -> None: """Set the last extraction timestamp.""" self.file_path.parent.mkdir(parents=True, exist_ok=True) data = { "last_extracted_at": timestamp.isoformat(), "updated_at": datetime.now(timezone.utc).isoformat(), } self.file_path.write_text(json.dumps(data, indent=2)) ``` This simple file-based approach provides durability without requiring an external state store. If the pipeline crashes, the next run picks up from where it left off. ### The MongoExtractor Class The main extractor class ties everything together: ```python class MongoExtractor: """Extracts conversation data from MongoDB to Parquet files.""" def __init__(self, settings: Settings | None = None): self.settings = settings or get_settings() self.transformer = DocumentTransformer() self.high_water_mark = HighWaterMark( self.settings.pipeline.high_water_mark_file ) self._client: MongoClient | None = None def extract( self, full_backfill: bool = False, output_dir: Path | None = None, ) -> list[Path]: """Extract data from MongoDB and write to Parquet files.""" output_dir = output_dir or self.settings.data.raw_dir # Determine start time since = None if not full_backfill: since = self.high_water_mark.get() if since: logger.info(f"Incremental extraction since {since.isoformat()}") extracted_at = datetime.now(timezone.utc) records_by_date: dict[str, list[dict]] = {} try: self.connect() for doc in self._fetch_documents(since=since): record = self.transformer.transform(doc, extracted_at) # Group by partition date date_key = record["date"].isoformat() if date_key not in records_by_date: records_by_date[date_key] = [] records_by_date[date_key].append(record) # Write partitions for date_key, records in records_by_date.items(): partition_date = datetime.fromisoformat(date_key) self._write_partition(records, partition_date, output_dir) finally: self.disconnect() return written_files ``` Key design decisions: - **Lazy connection**: MongoDB client is created on first use - **Batched writes**: Records are grouped by date before writing - **Partition directories**: Output follows Hive-style `date=YYYY-MM-DD` partitioning - **Guaranteed cleanup**: `finally` block ensures connection closes even on error ### Date-Partitioned Output The `_write_partition` method writes records with Snappy compression: ```python def _write_partition( self, records: list[dict], partition_date: datetime, output_dir: Path, ) -> Path: """Write records to a Parquet file in date-partitioned directory.""" date_str = partition_date.strftime("%Y-%m-%d") partition_dir = output_dir / f"date={date_str}" partition_dir.mkdir(parents=True, exist_ok=True) timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S_%f") file_path = partition_dir / f"conversations_{timestamp}.parquet" table = pa.Table.from_pylist(records, schema=CONVERSATION_SCHEMA) pq.write_table( table, file_path, compression="snappy", write_statistics=True, ) return file_path ``` Hive-style partitioning (`date=2024-01-15/`) enables DuckDB to read only relevant partitions when filtering by date, dramatically improving query performance. --- ## Loader Deep Dive: Parquet to DuckDB The `loader.py` module handles the "L" in ELT, taking extracted Parquet files and loading them into DuckDB. DuckDB is an excellent choice here because it can read Parquet files natively without intermediate steps. ### Schema Creation The loader creates the target schema and table if they do not exist: ```python CREATE_RAW_SCHEMA = "CREATE SCHEMA IF NOT EXISTS raw;" CREATE_CONVERSATIONS_TABLE = """ CREATE TABLE IF NOT EXISTS raw.conversations ( _id VARCHAR PRIMARY KEY, type VARCHAR, session_id VARCHAR, project_id VARCHAR, timestamp TIMESTAMP WITH TIME ZONE, ingested_at TIMESTAMP WITH TIME ZONE, extracted_at TIMESTAMP WITH TIME ZONE, message_role VARCHAR, message_content VARCHAR, message_raw VARCHAR, source_file VARCHAR, date DATE ); """ CREATE_INDEXES = [ "CREATE INDEX IF NOT EXISTS idx_conversations_project_id ON raw.conversations(project_id);", "CREATE INDEX IF NOT EXISTS idx_conversations_session_id ON raw.conversations(session_id);", "CREATE INDEX IF NOT EXISTS idx_conversations_date ON raw.conversations(date);", "CREATE INDEX IF NOT EXISTS idx_conversations_type ON raw.conversations(type);", "CREATE INDEX IF NOT EXISTS idx_conversations_timestamp ON raw.conversations(timestamp);", ] ``` Indexes are created on columns commonly used in WHERE clauses and GROUP BY operations. The `_id` column serves as the primary key for upsert operations. ### Upsert with ON CONFLICT The magic happens in the load SQL. DuckDB supports PostgreSQL-style upsert syntax: ```python LOAD_FROM_PARQUET = """ INSERT INTO raw.conversations SELECT * FROM read_parquet('{parquet_path}', hive_partitioning=true) ON CONFLICT (_id) DO UPDATE SET type = EXCLUDED.type, session_id = EXCLUDED.session_id, project_id = EXCLUDED.project_id, timestamp = EXCLUDED.timestamp, ingested_at = EXCLUDED.ingested_at, extracted_at = EXCLUDED.extracted_at, message_role = EXCLUDED.message_role, message_content = EXCLUDED.message_content, message_raw = EXCLUDED.message_raw, source_file = EXCLUDED.source_file, date = EXCLUDED.date; """ ``` This pattern is incredibly powerful: 1. **New records**: Inserted normally 2. **Existing records**: Updated with new values (idempotent reprocessing) 3. **Native Parquet reading**: `read_parquet()` with `hive_partitioning=true` understands our directory structure ### The DuckDBLoader Class ```python class DuckDBLoader: """Loads Parquet files into DuckDB database.""" def __init__(self, settings: Settings | None = None): self.settings = settings or get_settings() self._conn: duckdb.DuckDBPyConnection | None = None def connect(self) -> duckdb.DuckDBPyConnection: """Establish connection to DuckDB database.""" if self._conn is not None: return self._conn db_path = self.settings.duckdb.path db_path.parent.mkdir(parents=True, exist_ok=True) self._conn = duckdb.connect(str(db_path)) self._conn.execute(f"SET threads TO {self.settings.duckdb.threads};") return self._conn def load_from_parquet( self, parquet_path: Path | str, full_refresh: bool = False, ) -> int: """Load Parquet files into DuckDB.""" parquet_path = Path(parquet_path) self.create_database() if parquet_path.is_dir(): glob_pattern = str(parquet_path / "**" / "*.parquet") else: glob_pattern = str(parquet_path) count_before = self._get_row_count() if full_refresh: self.conn.execute("DELETE FROM raw.conversations;") load_sql = LOAD_FROM_PARQUET.format(parquet_path=glob_pattern) self.conn.execute(load_sql) count_after = self._get_row_count() return count_after - count_before if not full_refresh else count_after ``` The loader is intentionally simple. DuckDB handles the heavy lifting of reading Parquet, and the upsert pattern handles both initial loads and incremental updates. ### Statistics Gathering The loader also provides diagnostic information: ```python def get_table_stats(self) -> dict[str, Any]: """Get statistics about the loaded data.""" stats = {"row_count": self._get_row_count()} # Date range date_range = self.conn.execute(""" SELECT MIN(date), MAX(date), COUNT(DISTINCT date) FROM raw.conversations """).fetchone() if date_range and date_range[0]: stats["date_range"] = { "min": str(date_range[0]), "max": str(date_range[1]), "count": date_range[2], } # Type distribution type_counts = self.conn.execute(""" SELECT type, COUNT(*) as count FROM raw.conversations GROUP BY type ORDER BY count DESC """).fetchall() stats["type_distribution"] = [ {"type": row[0], "count": row[1]} for row in type_counts ] return stats ``` These statistics are invaluable for monitoring pipeline health and understanding data characteristics. --- ## Prefect Flows: Orchestrating the Pipeline With extraction and loading in place, we need orchestration. Prefect provides a Python-native way to define workflows with built-in retry logic, observability, and scheduling. ### Task Definitions Each pipeline step is a Prefect task with retry configuration: ```python from prefect import flow, task, get_run_logger RETRY_DELAYS = [30, 60, 120] # Exponential backoff @task( name="extract-mongodb", description="Extract data from MongoDB to Parquet files", retries=3, retry_delay_seconds=RETRY_DELAYS, ) def extract_task( full_backfill: bool = False, since: Optional[datetime] = None, ) -> dict: """Extract data from MongoDB and write to Parquet files.""" logger = get_run_logger() settings = get_settings() logger.info("Starting MongoDB extraction") extractor = MongoExtractor(settings=settings) try: if full_backfill: stats = extractor.full_extract() else: stats = extractor.incremental_extract(since=since) return stats finally: extractor.disconnect() ``` The retry configuration uses exponential backoff (30s, 60s, 120s) to handle transient failures gracefully. ### Load and Transform Tasks ```python @task( name="load-duckdb", description="Load Parquet files into DuckDB", retries=3, retry_delay_seconds=RETRY_DELAYS, ) def load_task( extraction_stats: dict, full_refresh: bool = False, ) -> dict: """Load Parquet files into DuckDB.""" logger = get_run_logger() settings = get_settings() loader = DuckDBLoader(settings=settings) try: loader.create_database() parquet_path = Path(settings.data.raw_dir) if full_refresh: stats = loader.load_from_parquet(str(parquet_path)) else: stats = loader.upsert_incremental(str(parquet_path)) return {"load_stats": stats, "table_stats": loader.get_table_stats()} finally: loader.disconnect() @task( name="transform-dbt", description="Run dbt transformations", retries=2, retry_delay_seconds=RETRY_DELAYS, ) def transform_task( load_stats: dict, full_refresh: bool = False, select: Optional[str] = None, ) -> dict: """Run dbt transformations.""" logger = get_run_logger() settings = get_settings() cmd = [ "dbt", "build", "--project-dir", str(settings.dbt.project_dir), "--profiles-dir", str(settings.dbt.profiles_dir), "--target", settings.dbt.target, ] if full_refresh: cmd.append("--full-refresh") result = subprocess.run(cmd, capture_output=True, text=True) if result.returncode != 0: raise RuntimeError(f"dbt build failed: {result.stdout}") return {"build_success": True, "output": result.stdout} ``` Notice how `load_task` takes `extraction_stats` as a parameter. This creates an explicit dependency in Prefect, ensuring extraction completes before loading begins. ### The Main Flow The flow orchestrates all tasks: ```python @flow( name="claude-analytics-pipeline", description="Main ELT pipeline for Claude conversation analytics", version="1.0.0", ) def analytics_pipeline( full_backfill: bool = False, full_refresh: bool = False, skip_extract: bool = False, skip_load: bool = False, skip_transform: bool = False, dbt_select: Optional[str] = None, ) -> dict: """Main analytics pipeline orchestrating extract, load, and transform.""" logger = get_run_logger() results = {} logger.info("Starting Claude Analytics Pipeline") # Step 1: Extract if not skip_extract: extraction_stats = extract_task(full_backfill=full_backfill) results["extraction"] = extraction_stats else: results["extraction"] = {"skipped": True} # Step 2: Load if not skip_load: load_stats = load_task( extraction_stats=results["extraction"], full_refresh=full_refresh, ) results["load"] = load_stats else: results["load"] = {"skipped": True} # Step 3: Transform if not skip_transform: transform_stats = transform_task( load_stats=results["load"], full_refresh=full_refresh, select=dbt_select, ) results["transform"] = transform_stats else: results["transform"] = {"skipped": True} return results ``` The flow supports skip flags for partial runs, useful during development or when reprocessing specific stages. ### Pipeline Flow Diagram ```mermaid flowchart TD subgraph Flow["analytics_pipeline"] A[Start] --> B{skip_extract?} B -->|No| C[extract_task] B -->|Yes| D[Skip extraction] C --> E{skip_load?} D --> E E -->|No| F[load_task] E -->|Yes| G[Skip loading] F --> H{skip_transform?} G --> H H -->|No| I[transform_task] H -->|Yes| J[Skip transform] I --> K[Return results] J --> K end C -.->|retries: 3| C F -.->|retries: 3| F I -.->|retries: 2| I ``` ### Deployment Configuration The `prefect.yaml` file defines multiple deployment variants: ```yaml name: claude-analytics deployments: # Hourly incremental pipeline - name: hourly-analytics version: "1.0.0" tags: [analytics, scheduled, hourly] description: "Hourly incremental analytics pipeline" entrypoint: analytics/flows/main_pipeline.py:scheduled_pipeline work_pool: name: analytics-pool schedules: - interval: 3600 # 1 hour # Daily full refresh at 2 AM - name: daily-full-refresh version: "1.0.0" tags: [analytics, scheduled, daily] description: "Daily full refresh of analytics (2 AM)" entrypoint: analytics/flows/main_pipeline.py:analytics_pipeline work_pool: name: analytics-pool schedules: - cron: "0 2 * * *" parameters: full_backfill: false full_refresh: true # Manual full backfill - name: full-backfill version: "1.0.0" tags: [analytics, manual, backfill] description: "Full historical backfill" entrypoint: analytics/flows/main_pipeline.py:analytics_pipeline work_pool: name: analytics-pool parameters: full_backfill: true full_refresh: true ``` This gives us: - **Hourly incremental**: Keeps data fresh throughout the day - **Daily full refresh**: Rebuilds all dbt models nightly - **Manual backfill**: For initial setup or recovery scenarios --- ## CLI Interface: Developer Experience Matters The `cli.py` module provides a polished command-line interface using Typer and Rich. A good CLI makes the difference between a tool developers avoid and one they reach for daily. ### Command Structure ```python import typer from rich.console import Console from rich.table import Table app = typer.Typer( name="claude-analytics", help="Claude Analytics Platform - ELT pipeline for conversation logs", add_completion=False, ) console = Console() ``` ### Available Commands The CLI exposes six primary commands: **1. config** - Display current configuration: ```bash $ claude-analytics config Current Configuration MongoDB URI: mongodb://localhost:27017 MongoDB DB: claude_logs DuckDB Path: /duckdb/analytics.db Batch Size: 10000 ``` **2. extract** - Run MongoDB extraction: ```bash $ claude-analytics extract --full-backfill --verbose MongoDB Extraction Source: mongodb://localhost:27017/claude_logs.conversations Mode: Full Backfill Output: /data/raw Extraction complete! Written 5 Parquet file(s): ┌─────────────────────────────────────┬────────────────┐ │ File │ Partition │ ├─────────────────────────────────────┼────────────────┤ │ conversations_20240115_143022.parq │ date=2024-01-15│ │ conversations_20240115_143022.parq │ date=2024-01-14│ └─────────────────────────────────────┴────────────────┘ ``` **3. load** - Load into DuckDB: ```bash $ claude-analytics load --stats DuckDB Loading Database: /duckdb/analytics.db Source: /data/raw Mode: Upsert Loading complete! Rows loaded/updated: 15234 Table Statistics: Total rows: 45892 Date range: 2024-01-01 to 2024-01-15 (15 days) ``` **4. transform** - Run dbt models: ```bash $ claude-analytics transform --models "+fct_messages" ``` **5. pipeline** - Run the complete ELT pipeline: ```bash $ claude-analytics pipeline --full-backfill --prefect Claude Analytics Pipeline Mode: Full Backfill Refresh: Incremental Steps: Prefect Running via Prefect... Pipeline complete! ``` **6. validate** - Run data quality checks: ```bash $ claude-analytics validate --build-docs ``` ### Implementation Example Here is the `pipeline` command implementation showing how options are handled: ```python @app.command() def pipeline( full_backfill: bool = typer.Option( False, "--full-backfill", help="Run full historical backfill", ), full_refresh: bool = typer.Option( False, "--full-refresh", help="Rebuild all data and dbt models", ), skip_extract: bool = typer.Option( False, "--skip-extract", help="Skip extraction step", ), use_prefect: bool = typer.Option( False, "--prefect", help="Run via Prefect orchestration", ), verbose: bool = typer.Option( False, "--verbose", "-v", help="Enable verbose logging", ), ) -> None: """Run the complete analytics pipeline.""" setup_logging("DEBUG" if verbose else "INFO") console.print("[bold blue]Claude Analytics Pipeline[/bold blue]\n") console.print(f" Mode: {'Full Backfill' if full_backfill else 'Incremental'}") if use_prefect: from analytics.flows import analytics_pipeline as prefect_pipeline result = prefect_pipeline( full_backfill=full_backfill, full_refresh=full_refresh, ) console.print(f"[bold green]Pipeline complete![/bold green]") else: # Direct execution without Prefect # ... step-by-step execution with progress output ``` The CLI supports both Prefect-orchestrated runs (for production) and direct execution (for development and debugging). --- ## Data Quality with Great Expectations Data pipelines are only as good as the data they produce. The `quality.py` module integrates Great Expectations for automated data validation. ### The DataQualityValidator Class ```python class DataQualityValidator: """Data quality validator using Great Expectations.""" def __init__(self, ge_project_dir: Path | None = None): self.settings = get_settings() self.ge_project_dir = ge_project_dir or Path( self.settings.great_expectations.project_dir ) self._context = None @property def context(self) -> Any: """Get or create Great Expectations data context.""" if self._context is None: try: import great_expectations as gx self._context = gx.get_context( context_root_dir=str(self.ge_project_dir) ) except ImportError: logger.warning("Great Expectations not installed") return None return self._context ``` The validator lazily loads the Great Expectations context, gracefully handling cases where the library is not installed. ### Bronze Layer Expectations The bronze (raw) layer validates fundamental data integrity: ```json { "expectation_suite_name": "bronze_expectations", "expectations": [ { "expectation_type": "expect_column_values_to_not_be_null", "kwargs": { "column": "_id" }, "meta": { "notes": "Primary key must not be null" } }, { "expectation_type": "expect_column_values_to_be_unique", "kwargs": { "column": "_id" }, "meta": { "notes": "Primary key must be unique" } }, { "expectation_type": "expect_column_values_to_be_in_set", "kwargs": { "column": "type", "value_set": ["user", "assistant", "tool_use", "tool_result"] }, "meta": { "notes": "Entry types must be valid" } } ] } ``` These checks catch data corruption early, before it propagates through the pipeline. ### Silver Layer Expectations The silver (intermediate) layer validates business logic: ```json { "expectation_suite_name": "silver_expectations", "expectations": [ { "expectation_type": "expect_column_values_to_be_in_set", "kwargs": { "column": "task_category", "value_set": ["bug_fix", "feature", "refactor", "testing", "documentation", "review", "other"] } }, { "expectation_type": "expect_column_values_to_be_in_set", "kwargs": { "column": "time_of_day", "value_set": ["morning", "afternoon", "evening", "night"] } }, { "expectation_type": "expect_column_values_to_be_between", "kwargs": { "column": "hour_of_day", "min_value": 0, "max_value": 23 } } ] } ``` These expectations verify that dbt transformations produce valid categorizations and derived fields. ### Running Validations ```python def validate_pipeline_data( validate_bronze: bool = True, validate_silver: bool = True, ) -> dict[str, Any]: """Validate pipeline data.""" validator = DataQualityValidator() results = {} if validate_bronze: results["bronze"] = validator.validate_bronze() if validate_silver: results["silver"] = validator.validate_silver() results["success"] = all( r.get("success", False) for r in results.values() if isinstance(r, dict) ) return results ``` This function provides a simple interface for validating data at any point in the pipeline. --- ## Putting It All Together We have covered a lot of ground. Let us see how to use this pipeline in practice. ### Initial Setup ```bash # Install the package cd analytics pip install -e ".[all]" # Configure environment cp .env.analytics.example .env.analytics # Edit .env.analytics with your MongoDB URI # Initialize DuckDB schema claude-analytics load --init-only # Run initial backfill claude-analytics pipeline --full-backfill --full-refresh ``` ### Daily Operations ```bash # Run incremental update claude-analytics pipeline # Check data quality claude-analytics validate # View statistics claude-analytics load --stats ``` ### Production Deployment ```bash # Start Prefect and deploy flows make up make deploy # Flows will run on schedule: # - hourly-analytics: Every hour # - daily-full-refresh: 2 AM daily # Trigger manual run prefect deployment run 'claude-analytics-pipeline/adhoc-analytics' ``` --- ## What We Built In this article, we explored the Python modules that power our analytics ELT pipeline: 1. **config.py**: Type-safe configuration with Pydantic Settings 2. **extractor.py**: MongoDB extraction with incremental high-water-mark tracking 3. **loader.py**: DuckDB loading with native Parquet support and upsert semantics 4. **flows/**: Prefect orchestration with retries and scheduling 5. **cli.py**: Developer-friendly command-line interface 6. **quality.py**: Great Expectations integration for data validation Each module follows the single responsibility principle, making the system easy to understand, test, and maintain. In [Article 6](/article-6-dbt-transformations), we will dive deep into the dbt transformations that turn this raw data into business-ready analytics models. We will explore the medallion architecture implementation, incremental models, and the fact/dimension tables that power our Metabase dashboards. --- *Have questions or suggestions? Share your thoughts in the comments below. If you found this useful, follow for more articles in this series.* **Suggested Tags:** Python, ETL, Data Engineering, Prefect, DuckDB

Designing a Modern Analytics Platform with Python, dbt, and Metabase

Tue, 30 Dec 2025 00:00:00 GMT

*Building an enterprise-grade ELT pipeline for AI conversation analytics* --- **This is Article 4 of a 7-part series on building a real-time data pipeline for Claude Code conversation logs.** - Article 1: System Overview and Architecture - Article 2: Building the Real-Time Sync Service - Article 3: Creating the Conversation Browser UI - **Article 4: Designing the Analytics Platform (You are here)** - Article 5: Python ETL with Prefect Orchestration - Article 6: Data Modeling with dbt - Article 7: Building Dashboards with Metabase --- ## Introduction: Why Analyze Conversation Logs? In Article 1, we built a sync service that captures Claude Code conversations in real-time, storing them in MongoDB. But raw data sitting in a database is like an unread book on a shelf. The real value emerges when we can answer questions: How are developers using the AI assistant? Which tools get invoked most frequently? What patterns indicate productive coding sessions? This is where analytics transforms curiosity into insight. Building an analytics platform on conversation logs presents unique challenges. The data is semi-structured (nested JSON with variable schemas), arrives continuously, and needs to support both operational dashboards and ad-hoc exploration. We need a system that handles incremental updates efficiently while maintaining data quality. In this article, I will walk you through the high-level architecture of our analytics platform. We will explore the medallion pattern for data organization, understand why we chose each technology in our stack, and see how the pieces fit together. Think of this as the architectural blueprint. The subsequent articles will dive deep into each component: Python ETL in Article 5, dbt transformations in Article 6, and Metabase dashboards in Article 7. Let's start by understanding how we organize data as it flows through the system. --- ## The Medallion Architecture: Bronze, Silver, and Gold The medallion architecture is a data design pattern that organizes data into layers of increasing refinement. Picture a refinery: crude oil enters, and through successive processing stages, it becomes gasoline, plastics, and other refined products. Our data goes through a similar journey. ```mermaid flowchart LR subgraph Bronze["BRONZE LAYER
(Raw Zone)"] direction TB RAW[(raw.conversations)] STG1[stg_conversations] STG2[stg_messages] STG3[stg_tool_calls] end subgraph Silver["SILVER LAYER
(Cleaned Zone)"] direction TB INT1[int_messages_enriched] INT2[int_tool_usage] INT3[int_sessions_computed] end subgraph Gold["GOLD LAYER
(Business Zone)"] direction TB DIM[Dimensions] FCT[Facts] AGG[Aggregates] end Bronze --> Silver --> Gold ``` ### Bronze Layer: The Foundation The bronze layer is where raw data lands after extraction from MongoDB. We perform minimal transformations here: type casting, null handling, and basic cleaning. The goal is to create a reliable, queryable version of the source data without losing any information. Our bronze layer contains three staging models: - **stg_conversations**: Core conversation metadata (session ID, project ID, timestamps) - **stg_messages**: Individual messages with role and content - **stg_tool_calls**: Tool invocations extracted from assistant responses These models are materialized as views because they simply reshape the source data. No heavy computation happens here. ### Silver Layer: Where Business Logic Lives The silver layer is where we apply business rules and enrich the data. We join related entities, compute derived fields, and handle edge cases that the bronze layer exposes. Three intermediate models form our silver layer: - **int_messages_enriched**: Messages joined with session context, content length calculated, code blocks detected - **int_tool_usage**: Tool calls with execution metadata and categorization - **int_sessions_computed**: Sessions with calculated duration, message counts, and activity levels The silver layer answers the question: "What does this data actually mean in our business context?" ### Gold Layer: Ready for Consumption The gold layer follows dimensional modeling principles, producing tables optimized for analytics queries. This is where we build the star schema that powers our dashboards. **Dimensions** provide context: - `dim_date`: Calendar attributes for time-based analysis - `dim_projects`: Project-level aggregations and metadata - `dim_sessions`: Session statistics and classifications - `dim_tools`: Tool catalog with categories **Facts** capture events: - `fct_messages`: One row per message with foreign keys to dimensions - `fct_tool_calls`: Tool usage events with timing and status - `fct_file_operations`: Code modification tracking **Aggregates** pre-compute common queries: - `agg_daily_summary`: Daily metrics for trend analysis - `agg_hourly_activity`: Hour-of-day patterns for heatmaps - `agg_session_metrics`: Per-session productivity indicators The gold layer tables are materialized as physical tables (not views) because they serve high-frequency dashboard queries. Pre-computing these aggregations dramatically improves dashboard performance. --- ## Technology Stack Overview Choosing the right tools for an analytics platform requires balancing several concerns: development velocity, operational simplicity, cost, and scalability. Here is why we selected each component of our stack. ``` +-----------------------------------------------------------+ | VISUALIZATION LAYER | | Metabase (Custom DuckDB) | +-----------------------------------------------------------+ | TRANSFORMATION LAYER | | dbt-core + dbt-duckdb (Medallion Architecture) | +-----------------------------------------------------------+ | STORAGE LAYER | | DuckDB (OLAP) + Parquet Files | +-----------------------------------------------------------+ | ORCHESTRATION LAYER | | Prefect 2.x + PostgreSQL Backend | +-----------------------------------------------------------+ | EXTRACTION LAYER | | Python 3.11 + PyMongo + PyArrow | +-----------------------------------------------------------+ | SOURCE LAYER | | MongoDB (Conversation Logs) | +-----------------------------------------------------------+ ``` ### Python for Extraction Python is the natural choice for data extraction. PyMongo provides a mature MongoDB client, and PyArrow enables efficient Parquet file generation. We use Pydantic for configuration management, giving us type-safe settings with environment variable support. The extraction layer handles: - Incremental extraction using high water marks - Batched processing (10,000 documents per batch) - Date-partitioned Parquet output with Snappy compression - Dead letter handling for malformed documents ### Prefect for Orchestration Prefect 2.x orchestrates our pipeline with a clean Python-native API. Unlike Airflow's DAG-centric model, Prefect treats flows as regular Python functions decorated with `@flow` and `@task`. This makes testing and local development straightforward. Key Prefect features we leverage: - **Retry with exponential backoff**: Extraction retries 3 times with 30-second delays - **Work pools**: Distribute execution across workers - **Scheduled deployments**: Hourly incremental, daily full refresh - **Web UI**: Monitor runs at `localhost:4200` ### DuckDB as the Analytical Database DuckDB is an embedded OLAP database that runs in-process. Think of it as SQLite for analytics. This choice might seem unconventional compared to cloud data warehouses, but it offers compelling advantages for our use case: - **Zero infrastructure**: No server to manage, just a file - **Columnar storage**: Optimized for analytical queries - **Excellent Parquet support**: Native reading with predicate pushdown - **SQL compatibility**: Standard SQL with analytical functions For a single-user or small-team analytics platform processing millions of rows, DuckDB delivers impressive performance without the complexity of distributed systems. ### dbt for Transformations dbt (data build tool) has become the industry standard for SQL-based transformations. It brings software engineering practices to analytics: - **Version-controlled SQL**: Models live in Git alongside application code - **Dependency management**: dbt builds models in the correct order - **Testing framework**: Schema tests, custom tests, data freshness checks - **Documentation**: Auto-generated data lineage and model docs Our dbt project uses the `dbt-duckdb` adapter, which integrates seamlessly with our embedded database. ### Metabase for Visualization Metabase provides self-service BI capabilities without requiring SQL knowledge. Users can explore data through a point-and-click interface, while power users can write custom queries. We run a custom Metabase image with the DuckDB driver pre-installed, enabling direct queries against our analytical database. The dashboard configuration lives in the repository, making it reproducible across environments. ### Great Expectations for Data Quality Data quality validation happens at multiple pipeline stages using Great Expectations. We define expectations (rules) for each layer: - **Bronze**: Required fields present, valid data types - **Silver**: Referential integrity, business rule compliance - **Gold**: Aggregate consistency, freshness thresholds When expectations fail, the pipeline alerts but continues processing. This prevents bad data from silently corrupting downstream models while avoiding complete pipeline failures for minor issues. --- ## Module Organization The analytics platform follows a clean separation of concerns. Each subdirectory has a single responsibility, making the codebase navigable and maintainable. ``` analytics/ ├── analytics/ # Python ETL package │ ├── __init__.py │ ├── cli.py # Typer CLI interface │ ├── config.py # Pydantic settings │ ├── extractor.py # MongoDB extraction │ ├── loader.py # DuckDB loading │ ├── quality.py # Great Expectations │ └── flows/ # Prefect orchestration │ ├── main_pipeline.py │ └── deployment.py ├── dbt/ # Transformation models │ ├── dbt_project.yml │ ├── profiles.yml │ └── models/ │ ├── staging/ # Bronze layer (3 models) │ ├── intermediate/ # Silver layer (3 models) │ └── marts/ # Gold layer (12 models) ├── great_expectations/ # Data quality config ├── metabase/ # Custom Metabase image ├── Dockerfile # Multi-stage build ├── docker-compose.analytics.yml ├── Makefile # Operational commands └── prefect.yaml # Deployment definitions ``` ### The Python Package (Article 5 Preview) The `analytics/` Python package contains our extraction and loading logic. Key classes include: - **MongoExtractor**: Connects to MongoDB, queries with timestamp filters, writes Parquet - **DuckDBLoader**: Creates schemas, loads Parquet files, manages indexes - **DataQualityValidator**: Runs Great Expectations checkpoints The CLI (`cli.py`) exposes these capabilities through a Typer interface, enabling both interactive use and scripted automation. ### The dbt Project (Article 6 Preview) The `dbt/` directory contains 18 SQL models organized by layer: | Layer | Models | Materialization | |-------|--------|-----------------| | Staging (Bronze) | 3 | View | | Intermediate (Silver) | 3 | View | | Marts (Gold) | 12 | Table | Each model includes documentation and tests defined in accompanying `_schema.yml` files. ### Metabase Configuration (Article 7 Preview) The `metabase/` directory contains a custom Dockerfile that builds Metabase with the DuckDB driver. Dashboard configurations can be exported and version-controlled, enabling reproducible analytics environments. --- ## Data Flow: From MongoDB to Dashboard Understanding the complete data journey helps diagnose issues and plan optimizations. Here is how a conversation entry flows through our system: ```mermaid flowchart TB subgraph Source["Source System"] MONGO[(MongoDB
conversations)] end subgraph Extract["Extraction"] EXT[MongoExtractor] HWM[High Water Mark] end subgraph Land["Landing Zone"] PQ[("Parquet Files
/data/raw/date=*/")] end subgraph Load["Loading"] LOADER[DuckDBLoader] end subgraph Store["Data Warehouse"] DUCK[(DuckDB
analytics.db)] end subgraph Transform["Transformation"] DBT[dbt Build] end subgraph Serve["Presentation"] META[Metabase] end MONGO -->|1. Query| EXT EXT <-->|Track Progress| HWM EXT -->|2. Write| PQ PQ -->|3. Read| LOADER LOADER -->|4. Upsert| DUCK DUCK -->|5. Transform| DBT DBT -->|6. Materialize| DUCK DUCK -->|7. Query| META ``` ### Step 1: Extraction The MongoExtractor queries MongoDB for documents newer than the last high water mark. Documents arrive in batches of 10,000, transformed into a flat structure suitable for columnar storage. ```python # Simplified extraction flow def extract(self, full_backfill: bool = False) -> List[Path]: query = {} if full_backfill else {"ingestedAt": {"$gt": self.hwm.get()}} for batch in self.client.find(query).batch_size(10000): records = [self.transform(doc) for doc in batch] parquet_path = self.write_parquet(records) self.hwm.set(max_timestamp) return written_files ``` ### Step 2: Parquet Landing Zone Extracted data lands in date-partitioned Parquet files: ``` /data/raw/ ├── date=2026-01-01/ │ └── conversations_001.parquet ├── date=2026-01-02/ │ └── conversations_001.parquet └── ... ``` Parquet provides excellent compression (typically 10x versus JSON) and enables predicate pushdown during queries. ### Step 3-4: Loading into DuckDB The DuckDBLoader reads Parquet files using glob patterns and upserts into the `raw.conversations` table: ```sql INSERT OR REPLACE INTO raw.conversations SELECT * FROM read_parquet('/data/raw/date=*/conversations_*.parquet', hive_partitioning=true); ``` Five indexes support common query patterns: project_id, session_id, date, type, and timestamp. ### Step 5-6: dbt Transformation dbt builds models in dependency order. A single `dbt run` command: 1. Reads from `raw.conversations` 2. Builds staging views (bronze) 3. Builds intermediate views (silver) 4. Materializes mart tables (gold) The entire transformation completes in seconds for typical daily volumes. ### Step 7: Metabase Queries Metabase connects directly to DuckDB and queries the gold layer tables. Pre-aggregated tables like `agg_daily_summary` power dashboard widgets without expensive real-time computation. --- ## Infrastructure with Docker Compose The analytics platform runs as six Docker services, orchestrated via `docker-compose.analytics.yml`. This setup provides isolation, reproducibility, and simple deployment. ```mermaid flowchart TB subgraph DockerHost["Docker Host"] subgraph Network["analytics-network"] subgraph Prefect["Prefect Stack"] PS[prefect-server
:4200] PDB[(prefect-db
PostgreSQL)] PW[prefect-worker] end subgraph Analytics["Analytics Stack"] AW[analytics-worker] end subgraph BI["BI Stack"] MB[metabase
:3001] MDB[(metabase-db
PostgreSQL)] end end subgraph Volumes["Persistent Volumes"] V1[(prefect-db-data)] V2[(metabase-db-data)] V3[(analytics-data)] V4[(duckdb-data)] end end subgraph External["External"] MONGO[(MongoDB)] end PS --> PDB PW --> PS MB --> MDB PW --> V3 PW --> V4 AW --> V3 AW --> V4 MB --> V4 PW -.-> MONGO AW -.-> MONGO ``` ### Service Overview | Service | Port | Purpose | |---------|------|---------| | prefect-server | 4200 | Workflow orchestration UI and API | | prefect-db | - | PostgreSQL for Prefect metadata | | prefect-worker | - | Executes scheduled flows | | analytics-worker | - | CLI access and ad-hoc commands | | metabase | 3001 | BI dashboards | | metabase-db | - | PostgreSQL for Metabase metadata | ### Key Configuration Patterns **Shared Volumes**: Both workers and Metabase mount `duckdb-data`, enabling shared access to the analytical database. DuckDB handles concurrent reads gracefully. **Host Network Access**: The `extra_hosts` directive maps `host.docker.internal` to the host gateway, allowing containers to reach MongoDB running on the host machine. **Health Checks**: Prefect server and Metabase include HTTP health checks, enabling dependent services to wait for readiness. ```yaml # Example health check configuration healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:4200/api/health || exit 1"] interval: 30s timeout: 10s retries: 5 start_period: 30s ``` --- ## Running the Stack Getting the analytics platform running requires just a few commands. The Makefile abstracts Docker Compose complexity into memorable targets. ### Quick Start ```bash cd analytics # 1. Start all services make up # 2. Deploy flows to Prefect (registers schedules) make deploy # 3. Run initial backfill make run-backfill ``` After these commands complete, you have: - Prefect UI at `http://localhost:4200` - Metabase at `http://localhost:3001` - dbt docs at `http://localhost:8080` ### Essential Makefile Commands ```bash # Infrastructure make up # Start all services make up-prefect # Start only Prefect (no Metabase) make down # Stop all services make logs # Follow worker logs make shell # Interactive shell in worker # Deployments make deploy # Deploy flows to Prefect make status # List deployments # Pipeline Execution make run-adhoc # Incremental run make run-backfill # Full historical backfill make run-daily # Daily full refresh make pipeline # Run directly (bypass Prefect) ``` ### Scheduled Deployments Once deployed, four pipeline variants run on schedule: | Deployment | Schedule | Use Case | |------------|----------|----------| | hourly-analytics | Every hour | Incremental sync | | daily-full-refresh | 2:00 AM | Rebuild all tables | | adhoc-analytics | Manual | On-demand runs | | full-backfill | Manual | Initial load or recovery | The hourly schedule ensures dashboards reflect recent activity, while the daily full refresh rebuilds aggregates and catches any missed incremental updates. ### Development vs Production For local development, the default configuration works well. Production deployments should consider: 1. **External MongoDB**: Update `MONGO_URI` to point to your production database 2. **Persistent storage**: Map volumes to durable storage, not ephemeral Docker volumes 3. **Resource limits**: Add memory and CPU constraints to prevent runaway queries 4. **Monitoring**: Export Prefect and Metabase metrics to your observability stack --- ## Data Model Preview The gold layer implements a star schema optimized for analytics queries. Here is a simplified view of the core entities: ```mermaid erDiagram DIM_DATE { date date_key PK int year int month int day string day_name boolean is_weekend } DIM_SESSIONS { string session_key PK string project_id FK timestamp session_start int duration_seconds int message_count string activity_level } DIM_PROJECTS { string project_key PK string project_name int total_sessions timestamp first_activity } FCT_MESSAGES { string message_key PK string session_key FK date date_key FK string role int content_length boolean has_code_block } FCT_TOOL_CALLS { string tool_call_key PK string session_key FK string tool_name string status } DIM_DATE ||--o{ FCT_MESSAGES : date_key DIM_SESSIONS ||--o{ FCT_MESSAGES : session_key DIM_SESSIONS ||--o{ FCT_TOOL_CALLS : session_key DIM_PROJECTS ||--o{ DIM_SESSIONS : project_id ``` This schema supports questions like: - "How many messages were exchanged last week?" (join FCT_MESSAGES with DIM_DATE) - "Which projects have the longest average session duration?" (aggregate DIM_SESSIONS by project) - "What is the tool usage breakdown by day of week?" (join FCT_TOOL_CALLS with DIM_DATE) The aggregate tables pre-compute common queries. For example, `agg_daily_summary` provides: | Metric | Description | |--------|-------------| | total_sessions | Sessions started that day | | total_messages | Messages exchanged | | total_tool_calls | Tools invoked | | avg_session_duration | Mean session length in seconds | | unique_projects | Distinct projects active | Dashboard widgets query these aggregates directly, returning results in milliseconds rather than scanning fact tables. --- ## Series Navigation This article provided the architectural overview of our analytics platform. We covered the medallion pattern, technology choices, and infrastructure setup. Now you understand how the pieces fit together. The next three articles dive deep into each major component: **Article 5: Python ETL with Prefect Orchestration** - MongoExtractor implementation details - High water mark state management - Prefect flow and task definitions - Error handling and retry strategies **Article 6: Data Modeling with dbt** - Staging model patterns - Intermediate enrichment logic - Dimensional modeling decisions - Testing and documentation **Article 7: Building Dashboards with Metabase** - DuckDB driver setup - Dashboard design principles - Key visualizations for conversation analytics - Sharing and embedding options Each article includes complete code examples and practical guidance. Whether you are building a similar platform or adapting these patterns for different data sources, the series provides a comprehensive reference. --- ## Wrapping Up Building an analytics platform is a journey from raw data to actionable insight. The medallion architecture gives us a clear mental model: bronze for raw data, silver for business logic, gold for consumption. Our technology stack (Python, Prefect, DuckDB, dbt, Metabase) balances power with operational simplicity. The entire platform runs on a single machine using Docker Compose. No cloud data warehouse fees, no complex distributed systems. For many use cases, this simplicity is not a limitation but a feature. You can always scale up later, but starting simple lets you iterate quickly and understand your data deeply. If you are building analytics on top of application data, I encourage you to try this stack. Clone the repository, run `make up`, and explore the conversation logs. The patterns demonstrated here apply far beyond AI assistant analytics. Any application generating semi-structured event data can benefit from similar treatment. In Article 5, we will open the hood on the Python ETL code. See you there. --- **Tags**: Data Engineering, Python, dbt, Analytics, Docker --- *Have questions or feedback? Share your thoughts in the comments below. If you found this article helpful, follow for the next installment in the series.*

Building a Conversation Log Browser with Next.js and shadcn/ui

Mon, 29 Dec 2025 00:00:00 GMT

**How to create a modern, responsive UI for browsing AI conversation logs stored in MongoDB** --- In the first two articles of this series, we built a robust pipeline for capturing Claude Code conversation logs: Article 1 covered the overall architecture, and Article 2 detailed the sync service that watches JSONL files and persists them to MongoDB. Now we have thousands of conversation entries sitting in a database. But data without visibility is just noise. In this article, we'll build the user interface layer that transforms raw conversation data into an explorable, searchable, and visually informative experience. We're creating a conversation log browser using Next.js, TypeScript, and shadcn/ui that connects directly to our MongoDB backend. By the end, you'll understand how to structure a Next.js App Router application for data-heavy dashboards, implement efficient cursor-based pagination, and leverage shadcn/ui for rapid, beautiful component development. --- ## The Tech Stack Before diving into code, let's understand what we're working with: | Technology | Version | Purpose | |------------|---------|---------| | Next.js | 16.1 | App Router, API routes, React Server Components | | React | 19.2 | UI framework with latest features | | TypeScript | 5.x | Type safety across the stack | | Tailwind CSS | 4.x | Utility-first styling | | shadcn/ui | 3.6 | Pre-built, customizable components | | TanStack Query | 5.x | Server state management | | Recharts | 2.x | Data visualization | | MongoDB | 7.x | Database driver | | Zod | 4.x | Runtime validation | This stack gives us the best of both worlds: the developer experience of modern React with the performance of server-side rendering and direct database access. --- ## Project Architecture Understanding the project structure is crucial before we explore individual components. Here's how the UI layer is organized: ``` ui/ ├── src/ │ ├── app/ # Next.js App Router │ │ ├── api/ # API route handlers │ │ │ ├── conversations/ │ │ │ │ ├── route.ts # Paginated list endpoint │ │ │ │ └── export/ │ │ │ │ └── route.ts # JSON export endpoint │ │ │ ├── projects/ │ │ │ │ └── route.ts # Distinct projects endpoint │ │ │ ├── sessions/ │ │ │ │ └── route.ts # Sessions per project │ │ │ └── stats/ │ │ │ └── route.ts # Aggregated chart data │ │ ├── globals.css # Tailwind + theme variables │ │ ├── layout.tsx # Root layout with providers │ │ ├── page.tsx # Main dashboard page │ │ └── providers.tsx # React Query provider │ ├── components/ │ │ ├── ui/ # shadcn/ui components │ │ │ ├── button.tsx │ │ │ ├── card.tsx │ │ │ ├── chart.tsx │ │ │ ├── table.tsx │ │ │ └── ... │ │ ├── ConversationDetail.tsx │ │ ├── ConversationList.tsx │ │ ├── FilterPanel.tsx │ │ └── SessionChart.tsx │ ├── hooks/ │ │ ├── useConversations.ts # Infinite query hook │ │ └── useStats.ts # Stats query hook │ └── lib/ │ ├── mongodb.ts # Database connection │ ├── types.ts # TypeScript interfaces │ ├── utils.ts # Utility functions │ └── dateUtils.ts # Date range helpers ├── components.json # shadcn/ui configuration ├── next.config.ts # Next.js configuration ├── package.json └── tsconfig.json ``` The architecture follows a clear separation of concerns: - **API Routes** handle all database interactions, keeping MongoDB logic server-side - **Custom Hooks** abstract TanStack Query complexity from components - **Components** focus purely on presentation and user interaction - **Lib** contains shared utilities and type definitions Here's how data flows through the application: ``` ┌─────────────────────────────────────────────────────────────┐ │ Browser │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ page.tsx │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌───────────────┐ │ │ │ │ │ FilterPanel │ │SessionChart │ │ConversationList│ │ │ │ │ └──────┬──────┘ └──────┬──────┘ └───────┬───────┘ │ │ │ │ │ │ │ │ │ │ │ └───────────────┼────────────────┘ │ │ │ │ │ │ │ │ │ ┌──────────▼──────────┐ │ │ │ │ │ Custom Hooks │ │ │ │ │ │ useConversations │ │ │ │ │ │ useStats │ │ │ │ │ └──────────┬──────────┘ │ │ │ └─────────────────────────┼───────────────────────────┘ │ │ │ fetch() │ └────────────────────────────┼────────────────────────────────┘ │ ┌────────────────────────────▼────────────────────────────────┐ │ Next.js Server │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ API Routes │ │ │ │ /api/projects /api/sessions /api/conversations │ │ │ │ /api/stats /api/conversations/export │ │ │ └──────────────────────────┬──────────────────────────┘ │ │ │ │ │ ┌──────────────────────────▼──────────────────────────┐ │ │ │ lib/mongodb.ts │ │ │ │ (Connection Pool Singleton) │ │ │ └──────────────────────────┬──────────────────────────┘ │ └─────────────────────────────┼───────────────────────────────┘ │ ┌─────────────────────────────▼───────────────────────────────┐ │ MongoDB │ │ conversations collection │ └─────────────────────────────────────────────────────────────┘ ``` --- ## shadcn/ui Integration One of my favorite aspects of this project is how shadcn/ui accelerates development. Unlike traditional component libraries where you install a package, shadcn/ui copies component source code directly into your project. You own the code, which means complete customization freedom. ### Configuration The `components.json` file defines how shadcn/ui integrates with your project: ```json { "$schema": "https://ui.shadcn.com/schema.json", "style": "new-york", "rsc": true, "tsx": true, "tailwind": { "config": "", "css": "src/app/globals.css", "baseColor": "neutral", "cssVariables": true, "prefix": "" }, "iconLibrary": "lucide", "aliases": { "components": "@/components", "utils": "@/lib/utils", "ui": "@/components/ui", "lib": "@/lib", "hooks": "@/hooks" } } ``` We're using the "new-york" style variant, which provides a slightly more refined aesthetic compared to the default style. The `rsc: true` setting indicates React Server Components compatibility, and the aliases map to our project's directory structure. ### Components in Use The UI leverages these shadcn/ui components: | Component | Usage | |-----------|-------| | Card | Container for filter panel, conversation list, charts | | Button | Actions, sort toggles, load more | | Table | Conversation list display | | Select | Project/session dropdowns | | Calendar | Date range pickers | | Popover | Calendar trigger containers | | ScrollArea | Scrollable conversation list | | Badge | Message type indicators | | Separator | Visual content dividers | | Input | Search field | ### The cn() Utility Pattern Every shadcn/ui component uses the `cn()` utility for class name merging: ```typescript import { clsx, type ClassValue } from "clsx" import { twMerge } from "tailwind-merge" export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)) } ``` This small utility is surprisingly powerful. It combines `clsx` for conditional class handling with `tailwind-merge` for intelligent Tailwind class deduplication. When you pass `className="p-4"` to a component that already has `className="p-2"`, `twMerge` ensures only `p-4` applies. ### Theming with CSS Variables The theme is defined in `globals.css` using the modern oklch color space: ```css :root { --background: oklch(0.9789 0.0082 121.6272); --foreground: oklch(0 0 0); --primary: oklch(0.5106 0.2301 276.9656); --primary-foreground: oklch(1.0000 0 0); --muted: oklch(0.9551 0 0); --muted-foreground: oklch(0.3211 0 0); /* ... more variables */ --chart-1: oklch(0.5106 0.2301 276.9656); --chart-2: oklch(0.7038 0.1230 182.5025); --radius: 1rem; } .dark { --background: oklch(0 0 0); --foreground: oklch(1.0000 0 0); /* ... dark mode overrides */ } ``` Components reference these variables, making theme changes trivial. Want a different primary color? Change one line. Need dark mode? The `.dark` class handles it automatically. --- ## Data Layer Architecture The UI communicates with MongoDB through Next.js API routes. This keeps database credentials server-side and provides a clean separation between frontend and backend concerns. ### MongoDB Connection Singleton Database connections are expensive to create. We use a singleton pattern to reuse connections across requests: ```typescript // src/lib/mongodb.ts import { MongoClient, Db } from 'mongodb'; const uri = process.env.MONGO_URI || 'mongodb://localhost:27017'; const dbName = process.env.MONGO_DB || 'claude_logs'; let cachedClient: MongoClient | null = null; let cachedDb: Db | null = null; export async function getDatabase(): Promise { if (cachedDb) { return cachedDb; } if (!cachedClient) { cachedClient = await MongoClient.connect(uri, { maxPoolSize: 10, }); } cachedDb = cachedClient.db(dbName); return cachedDb; } export async function getConversationsCollection() { const db = await getDatabase(); return db.collection('conversations'); } ``` The `maxPoolSize: 10` setting limits concurrent connections, preventing connection exhaustion under load. In serverless environments, this caching pattern is essential for performance. ### API Routes Structure Each API route follows a consistent pattern: validate input with Zod, query MongoDB, and return JSON: ```typescript // src/app/api/conversations/route.ts import { NextRequest, NextResponse } from 'next/server'; import { ObjectId, Filter, Document } from 'mongodb'; import { z } from 'zod'; import { getConversationsCollection } from '@/lib/mongodb'; const querySchema = z.object({ projectId: z.string().min(1), sessionId: z.string().optional(), search: z.string().optional(), startDate: z.string().optional(), endDate: z.string().optional(), cursor: z.string().optional(), limit: z.coerce.number().min(1).max(100).default(50), sortOrder: z.enum(['asc', 'desc']).default('desc'), }); export async function GET(request: NextRequest) { const searchParams = Object.fromEntries(request.nextUrl.searchParams); const parsed = querySchema.safeParse(searchParams); if (!parsed.success) { return NextResponse.json( { error: 'Invalid parameters', details: parsed.error.issues }, { status: 400 } ); } const { projectId, sessionId, search, startDate, endDate, cursor, limit, sortOrder } = parsed.data; try { const collection = await getConversationsCollection(); const query: Filter = { projectId }; // Build query filters... if (sessionId) query.sessionId = sessionId; if (search) query.$text = { $search: search }; // Date range filtering if (startDate || endDate) { query.timestamp = { $ne: null, $exists: true }; if (startDate) query.timestamp.$gte = startDate; if (endDate) query.timestamp.$lte = endDate + 'T23:59:59.999Z'; } // Cursor-based pagination (details below) // ... const docs = await collection .find(query) .sort({ timestamp: sortOrder === 'desc' ? -1 : 1, _id: sortOrder === 'desc' ? -1 : 1 }) .limit(limit + 1) .toArray(); const hasMore = docs.length > limit; const data = hasMore ? docs.slice(0, -1) : docs; return NextResponse.json({ data, pagination: { nextCursor, hasMore, total }, }); } catch (error) { console.error('Failed to fetch conversations:', error); return NextResponse.json({ error: 'Failed to fetch conversations' }, { status: 500 }); } } ``` ### Cursor-Based Pagination Instead of offset pagination (skip N records), we use cursor-based pagination for better performance with large datasets. The cursor encodes the last seen record's position: ```typescript // Cursor format: base64(timestamp|_id) if (cursor) { const decoded = Buffer.from(cursor, 'base64').toString(); const [cursorTimestamp, cursorId] = decoded.split('|'); const cursorOid = new ObjectId(cursorId); if (sortOrder === 'desc') { query.$or = [ { timestamp: { $lt: cursorTimestamp } }, { timestamp: cursorTimestamp, _id: { $lt: cursorOid } }, ]; } else { query.$or = [ { timestamp: { $gt: cursorTimestamp } }, { timestamp: cursorTimestamp, _id: { $gt: cursorOid } }, ]; } } // Generate next cursor from last document const nextCursor = hasMore && data.length > 0 ? Buffer.from(`${data[data.length - 1].timestamp}|${data[data.length - 1]._id.toString()}`).toString('base64') : null; ``` This approach is more efficient because MongoDB can use indexes to jump directly to the cursor position rather than counting through skipped records. ### Stats Aggregation Pipeline The chart data comes from a MongoDB aggregation pipeline that groups messages by time period: ```typescript // src/app/api/stats/route.ts const pipeline = [ { $match: matchStage }, { $group: { _id: { period: { $dateToString: { format: dateFormat, date: { $toDate: '$timestamp' } } }, type: '$type', }, count: { $sum: 1 }, }, }, { $sort: { '_id.period': -1 } }, { $limit: periodCount * 10 }, ]; const results = await collection.aggregate(pipeline).toArray(); ``` The `dateFormat` changes based on granularity (hourly, daily, weekly), enabling flexible time-series visualization. --- ## Component Patterns Let's examine the key UI components and the patterns they employ. ### FilterPanel: URL-Based State Management The FilterPanel demonstrates using URL search parameters as state. This provides shareable, bookmarkable filter states: ```typescript // src/components/FilterPanel.tsx 'use client'; import { useRouter, useSearchParams } from 'next/navigation'; import { useEffect, useState } from 'react'; export function FilterPanel({ onExport, chartFilterActive, onResetChartFilter }) { const router = useRouter(); const searchParams = useSearchParams(); const [projects, setProjects] = useState([]); const projectId = searchParams.get('projectId') || ''; const sessionId = searchParams.get('sessionId') || ''; // Update URL when filter changes const updateFilter = (key: string, value: string) => { const params = new URLSearchParams(searchParams.toString()); if (value) { params.set(key, value); } else { params.delete(key); } // Clear dependent filters if (key === 'projectId') { params.delete('sessionId'); } router.push(`?${params.toString()}`); }; return (

{/* More filters... */}

); } ``` ### ConversationList: Infinite Scroll The conversation list uses TanStack Query's `useInfiniteQuery` for seamless infinite scrolling: ```typescript // src/hooks/useConversations.ts 'use client'; import { useInfiniteQuery } from '@tanstack/react-query'; export function useConversations(filters: Omit) { return useInfiniteQuery({ queryKey: ['conversations', filters], queryFn: ({ pageParam }) => fetchConversations({ ...filters, cursor: pageParam, limit: 50 }), getNextPageParam: (lastPage) => lastPage.pagination.nextCursor, initialPageParam: undefined, enabled: !!filters.projectId, }); } ``` The component consumes this hook and renders with proper loading states: ```typescript // src/components/ConversationList.tsx export function ConversationList({ conversations, hasMore, total, onLoadMore, isLoading, sortOrder, onSortChange, }) { const [selectedId, setSelectedId] = useState(null); return (

Showing {conversations.length} of {total} conversations Type Preview {conversations.map((conv) => ( setSelectedId(conv._id.toString())} className={cn('cursor-pointer', selectedId === conv._id.toString() && 'bg-muted')} > {conv.type} {formatTimestamp(conv.timestamp)} {conv.message?.slice(0, 60)}... ))}

{hasMore && (

)}

{selectedConversation && ( setSelectedId(null)} /> )}

); } ``` ### SessionChart: Interactive Visualization The chart component uses Recharts wrapped with shadcn's ChartContainer for consistent styling: ```typescript // src/components/SessionChart.tsx 'use client'; import { Bar, BarChart, CartesianGrid, XAxis, YAxis } from 'recharts'; import { ChartContainer, ChartConfig, ChartTooltip, ChartLegend } from '@/components/ui/chart'; const chartConfig: ChartConfig = { user: { label: 'User', color: 'hsl(221, 83%, 53%)', }, assistant: { label: 'Assistant', color: 'hsl(142, 71%, 45%)', }, }; export function SessionChart({ projectId, sessionId, onBarClick }) { const [granularity, setGranularity] = useState('day'); const { data: statsData, isLoading } = useStats({ projectId, sessionId, granularity, periodCount: 14, }); const handleBarClick = (data) => { if (onBarClick && data.payload?.periodStart) { onBarClick(data.payload.periodStart, granularity); } }; return ( Message Activity

); } ``` --- ## State Management Strategy This application uses a deliberately simple state management approach that avoids the complexity of global state libraries. ### URL as Source of Truth All filter state lives in the URL. This provides several benefits: - **Shareable**: Copy the URL to share exact filter state - **Bookmarkable**: Save specific views for later - **Back button works**: Browser history navigation just works - **No hydration issues**: State is consistent between server and client ```typescript // Reading state from URL const projectId = searchParams.get('projectId') || ''; const sortOrder = searchParams.get('sortOrder') as SortOrder || 'desc'; // Writing state to URL const handleSortChange = (newSortOrder: SortOrder) => { const params = new URLSearchParams(searchParams.toString()); params.set('sortOrder', newSortOrder); router.push(`?${params.toString()}`); }; ``` ### TanStack Query for Server State Server data is managed entirely by TanStack Query. The query key includes all filter parameters, so changing any filter automatically triggers a refetch: ```typescript return useInfiniteQuery({ queryKey: ['conversations', filters], // filters object becomes part of cache key queryFn: ({ pageParam }) => fetchConversations({ ...filters, cursor: pageParam }), staleTime: 60 * 1000, // Cache for 1 minute refetchOnWindowFocus: false, }); ``` ### Local State for UI Component-specific UI state (like which row is selected) stays local: ```typescript const [selectedId, setSelectedId] = useState(null); ``` This three-tier approach keeps the codebase simple and predictable. --- ## Interactive Features ### Chart-to-Filter Interaction Clicking a bar in the chart filters the conversation list to that time period. This creates a natural drill-down workflow: ```typescript // page.tsx const handleChartBarClick = (periodStart: string, granularity: TimeGranularity) => { const { startDate, endDate } = periodToDateRange(periodStart, granularity); const params = new URLSearchParams(searchParams.toString()); params.set('startDate', startDate); params.set('endDate', endDate); params.set('chartFilter', 'true'); // Visual indicator router.push(`?${params.toString()}`); }; ``` The `periodToDateRange` helper handles the conversion from chart periods to filter dates: ```typescript // lib/dateUtils.ts export function periodToDateRange(periodStart: string, granularity: TimeGranularity): DateRange { switch (granularity) { case 'hour': const date = parseISO(periodStart); const dayStr = format(date, 'yyyy-MM-dd'); return { startDate: dayStr, endDate: dayStr }; case 'day': return { startDate: periodStart, endDate: periodStart }; case 'week': return parseISOWeekToDateRange(periodStart); } } ``` ### Debounced Search The search input uses a debounce pattern to avoid excessive API calls: ```typescript const [search, setSearch] = useState(searchParams.get('search') || ''); useEffect(() => { const timer = setTimeout(() => { updateFilter('search', search); }, 300); return () => clearTimeout(timer); }, [search]); ``` ### Export Functionality Users can export filtered conversations as JSON. The export endpoint respects all active filters: ```typescript const handleExport = () => { const params = new URLSearchParams(); params.set('projectId', projectId); if (sessionId) params.set('sessionId', sessionId); if (search) params.set('search', search); if (startDate) params.set('startDate', startDate); if (endDate) params.set('endDate', endDate); window.open(`/api/conversations/export?${params.toString()}`, '_blank'); }; ``` The server sets proper headers for file download: ```typescript return new NextResponse(JSON.stringify(docs, null, 2), { headers: { 'Content-Type': 'application/json', 'Content-Disposition': `attachment; filename="${filename}"`, }, }); ``` --- ## Development and Deployment ### Local Development Getting the UI running locally is straightforward: ```bash cd ui npm install # First time only npm run dev # Starts on http://localhost:3000 ``` The UI expects MongoDB to be running and the sync service to have populated the `conversations` collection. Environment variables are loaded from the parent directory's `.env` file through `next.config.ts`: ```typescript // next.config.ts import { config } from "dotenv"; import path from "path"; config({ path: path.resolve(__dirname, "../.env") }); const nextConfig: NextConfig = {}; export default nextConfig; ``` ### Production Build For production deployment: ```bash npm run build # Creates optimized production build npm start # Runs production server ``` The production build benefits from: - Static page generation where possible - Optimized JavaScript bundles - Automatic code splitting - Server components for reduced client JavaScript --- ## Conclusion We've built a feature-rich conversation log browser that transforms raw MongoDB data into an explorable interface. The key architectural decisions that make this work: 1. **URL-based state** keeps the application predictable and shareable 2. **API routes** isolate database logic from the frontend 3. **Cursor pagination** scales efficiently with large datasets 4. **shadcn/ui** accelerates development with customizable, accessible components 5. **TanStack Query** handles caching, refetching, and infinite scroll seamlessly The UI now provides real visibility into Claude Code interactions: browse by project and session, search message content, visualize activity over time, and export data for further analysis. In the next article, we'll build the analytics layer that takes this data even further with dbt transformations and Metabase dashboards for deeper insights. --- **Try it yourself**: Clone the repository, start MongoDB and the sync service (covered in Article 2), then run `npm run dev` in the UI directory. Your Claude Code conversations become instantly explorable. **Questions or improvements?** Open an issue or PR on the repository. I'd love to hear how you're using this for your own observability needs. --- **Suggested Tags**: nextjs, typescript, mongodb, shadcn-ui, react

Building a Robust File-to-MongoDB Sync Service in TypeScript

Sun, 28 Dec 2025 00:00:00 GMT

**How to build a crash-resistant data pipeline that never loses a line** --- *This is Article 2 of a 7-part series on building a complete log analytics platform. In [Article 1], we covered the high-level architecture of our system. Today, we are rolling up our sleeves and diving deep into the sync service—the component that watches files, buffers changes durably, and syncs them to MongoDB.* --- ## The Problem: Files That Never Stop Growing Imagine you have an application that continuously appends JSON lines to log files. Maybe it is an AI coding assistant logging every conversation, a web server writing access logs, or an IoT device streaming sensor readings. These files grow constantly, and you need to get that data into MongoDB for querying and analysis. Sounds simple, right? Just tail the file and insert each line into the database. But here is where it gets interesting. What happens when: - **Your service crashes** mid-sync? How do you know where you left off? - **MongoDB goes down** for maintenance? Do you lose all the data generated during that window? - **You restart the service**? Does it re-process the entire file, creating duplicates? The naive approach—keeping file positions in memory—fails spectacularly on any of these scenarios. We need something more robust. ## The Solution: A Three-Layer Architecture Our sync service solves these problems with three distinct components, each with a single responsibility: ```mermaid flowchart LR subgraph "File System" F["*.jsonl files"] end subgraph "Sync Service" W["Watcher"] B["SQLite Buffer"] S["MongoSync"] end subgraph "Database" M["MongoDB"] end F -->|"new lines"| W W -->|"entries + positions"| B B -->|"pending batches"| S S -->|"insertMany"| M S -.->|"mark synced"| B ``` Here is the key insight: **SQLite sits in the middle as a durable buffer**. If MongoDB is down, entries queue up in SQLite. If the service crashes, SQLite remembers exactly which byte position we reached in each file. This design makes the entire pipeline crash-resistant. Let me walk you through each component, showing you the actual TypeScript code that makes this work. ## The Type Foundation Before diving into the components, let us establish our type contracts. These interfaces define the shape of data as it flows through our system: ```typescript // types.ts export interface ClaudeEntry { type: string; sessionId?: string; timestamp?: string; message?: unknown; [key: string]: unknown; } export interface BufferedEntry { id: number; project_id: string; session_id: string | null; source_file: string; entry_json: string; created_at: string; synced: number; } export interface MongoDocument extends ClaudeEntry { projectId: string; sourceFile: string; ingestedAt: Date; } export interface SyncStats { pending: number; synced: number; lastSyncAt: Date | null; mongoConnected: boolean; } ``` Notice how `BufferedEntry` represents the SQLite row structure (snake_case, as is conventional for SQL), while `MongoDocument` extends our base entry with metadata (camelCase for JavaScript/MongoDB). The `SyncStats` interface powers our health endpoint, giving us visibility into the pipeline state. ## Deep Dive: File Watching with Chokidar The `Watcher` class is our eyes on the file system. It uses [chokidar](https://github.com/paulmillr/chokidar), a battle-tested file watching library, to detect when JSONL files change. ### The Configuration That Matters ```typescript // watcher.ts import chokidar, { FSWatcher } from 'chokidar'; import fs from 'fs'; import readline from 'readline'; import { Buffer } from './buffer'; import { ClaudeEntry } from './types'; export class Watcher { private watcher: FSWatcher | null = null; private buffer: Buffer; private watchDir: string; private processing = new Set(); constructor(watchDir: string, buffer: Buffer) { this.watchDir = watchDir; this.buffer = buffer; } start(): void { this.watcher = chokidar.watch(`${this.watchDir}/**/*.jsonl`, { persistent: true, ignoreInitial: false, awaitWriteFinish: { stabilityThreshold: 300, pollInterval: 100, }, usePolling: false, alwaysStat: true, }); this.watcher.on('add', (filePath) => this.processFile(filePath)); this.watcher.on('change', (filePath) => this.processFile(filePath)); this.watcher.on('error', (error) => console.error('Watcher error:', error)); } // ... more methods } ``` Let me explain the key configuration options: - **`ignoreInitial: false`**: When the service starts, it processes all existing files. Combined with our position tracking, this means we pick up exactly where we left off. - **`awaitWriteFinish`**: This is crucial for log files. Applications often write in bursts, and we do not want to process a half-written line. The `stabilityThreshold` of 300ms ensures the file has stopped changing before we read it. - **`alwaysStat: true`**: We always get file stats with each event, which we need to compare against our stored position. ### The Processing Lock Pattern Here is a subtle but important detail—the `processing` Set: ```typescript private processing = new Set(); private async processFile(filePath: string): Promise { // Prevent concurrent processing of same file if (this.processing.has(filePath)) return; this.processing.add(filePath); try { // ... process the file } finally { this.processing.delete(filePath); } } ``` Why do we need this? File system events can fire rapidly. If a file receives multiple writes in quick succession, we might get overlapping `change` events. Without this lock, we could end up with two concurrent reads of the same file, leading to duplicate entries or corrupted state. The `Set` gives us O(1) lookups, and the `try/finally` ensures we always release the lock, even if processing throws an error. ### Stream-Based Line Parsing Here is where the magic of crash recovery happens: ```typescript private async processFile(filePath: string): Promise { if (this.processing.has(filePath)) return; this.processing.add(filePath); try { const projectId = this.extractProjectId(filePath); const startPos = this.buffer.getFilePosition(filePath); let stats: fs.Stats; try { stats = fs.statSync(filePath); } catch { return; // File deleted } // Skip if file hasn't grown if (stats.size <= startPos) return; // Start reading from where we left off const stream = fs.createReadStream(filePath, { start: startPos, encoding: 'utf8', }); const rl = readline.createInterface({ input: stream, crlfDelay: Infinity, }); const entries: Array<{ projectId: string; sessionId?: string; sourceFile: string; data: ClaudeEntry; }> = []; for await (const line of rl) { if (!line.trim()) continue; try { const data = JSON.parse(line) as ClaudeEntry; entries.push({ projectId, sessionId: data.sessionId, sourceFile: filePath, data, }); } catch (e) { console.error(`Parse error in ${path.basename(filePath)}:`, (e as Error).message); } } if (entries.length > 0) { this.buffer.insertEntries(entries); console.log(`Buffered ${entries.length} entries from ${path.basename(filePath)}`); } // Update position AFTER successful processing this.buffer.updateFilePosition(filePath, stats.size); } finally { this.processing.delete(filePath); } } ``` The critical insight here is the `start` parameter in `createReadStream`. We ask SQLite for the last known position, then tell Node.js to start reading from that byte offset. This means after a crash, we resume exactly where we stopped—no duplicates, no missed lines. Notice also that we update the file position *after* successfully buffering entries. This ordering is intentional: if we crash between buffering and updating the position, we will re-read those lines on restart. But that is fine—duplicates are handled downstream. The alternative (updating position first) could lose data. ## Deep Dive: SQLite as a Durable Buffer The `Buffer` class is the heart of our crash-resistance strategy. It uses SQLite with carefully chosen configuration to provide durability without sacrificing performance. ### Why SQLite and Not In-Memory? You might wonder: why not just use an in-memory queue? Three reasons: 1. **Crash survival**: In-memory state vanishes on restart. SQLite persists to disk. 2. **MongoDB outages**: If MongoDB is down for hours, you need somewhere to store the backlog. Memory would eventually overflow. 3. **Simplicity**: SQLite is a single file, requires no separate server, and has excellent Node.js bindings via `better-sqlite3`. ### WAL Mode and Performance Pragmas ```typescript // buffer.ts import Database, { Database as DatabaseType, Statement } from 'better-sqlite3'; import path from 'path'; import os from 'os'; import fs from 'fs'; export class Buffer { private db: DatabaseType; constructor(dbPath?: string) { const defaultPath = path.join(os.homedir(), '.claude-sync', 'buffer.db'); const finalPath = dbPath || defaultPath; fs.mkdirSync(path.dirname(finalPath), { recursive: true }); this.db = new Database(finalPath); this.db.pragma('journal_mode = WAL'); this.db.pragma('synchronous = NORMAL'); this.initSchema(); this.statements = this.prepareStatements(); } // ... more methods } ``` Two pragmas make a huge difference here: - **`journal_mode = WAL`** (Write-Ahead Logging): Instead of overwriting pages in place, SQLite appends changes to a separate log file. This allows concurrent reads while writing and provides better crash recovery. For a sync service that reads pending entries while writing new ones, WAL is essential. - **`synchronous = NORMAL`**: This is a tradeoff between safety and speed. `FULL` would sync after every transaction (slowest, safest). `OFF` would never sync (fastest, dangerous). `NORMAL` syncs at critical moments but not after every transaction—a good balance for our use case where occasional data loss is acceptable if the machine loses power. ### The Schema Design ```typescript private initSchema(): void { this.db.run(` CREATE TABLE IF NOT EXISTS file_positions ( file_path TEXT PRIMARY KEY, position INTEGER NOT NULL, updated_at TEXT NOT NULL ); CREATE TABLE IF NOT EXISTS pending_entries ( id INTEGER PRIMARY KEY AUTOINCREMENT, project_id TEXT NOT NULL, session_id TEXT, source_file TEXT NOT NULL, entry_json TEXT NOT NULL, created_at TEXT NOT NULL, synced INTEGER DEFAULT 0 ); CREATE INDEX IF NOT EXISTS idx_pending_synced ON pending_entries(synced); CREATE INDEX IF NOT EXISTS idx_pending_project ON pending_entries(project_id); `); } ``` ```mermaid erDiagram file_positions { TEXT file_path PK INTEGER position TEXT updated_at } pending_entries { INTEGER id PK TEXT project_id TEXT session_id TEXT source_file TEXT entry_json TEXT created_at INTEGER synced } ``` We have two tables: 1. **`file_positions`**: Tracks the byte offset for each watched file. The `file_path` is the primary key since each file has exactly one position. 2. **`pending_entries`**: The queue of entries waiting to be synced. The `synced` flag (0 or 1) marks whether an entry has been successfully written to MongoDB. We keep synced entries around for 7 days for debugging purposes. The indexes are carefully chosen: `idx_pending_synced` speeds up our main query (fetching unsynced entries), and `idx_pending_project` helps if you ever need to query by project. ### Prepared Statements for Performance Here is a pattern that makes a significant performance difference: ```typescript private statements: { getPosition: Statement<[string]>; upsertPosition: Statement<[string, number]>; insertEntry: Statement<[string, string | null, string, string]>; getPending: Statement<[number]>; markSynced: Statement<[number]>; cleanupSynced: Statement<[]>; countPending: Statement<[]>; countSynced: Statement<[]>; }; private prepareStatements() { return { getPosition: this.db.prepare( 'SELECT position FROM file_positions WHERE file_path = ?' ), upsertPosition: this.db.prepare(` INSERT INTO file_positions (file_path, position, updated_at) VALUES (?, ?, datetime('now')) ON CONFLICT(file_path) DO UPDATE SET position = excluded.position, updated_at = datetime('now') `), insertEntry: this.db.prepare(` INSERT INTO pending_entries (project_id, session_id, source_file, entry_json, created_at) VALUES (?, ?, ?, ?, datetime('now')) `), getPending: this.db.prepare(` SELECT id, entry_json, project_id, source_file, session_id FROM pending_entries WHERE synced = 0 ORDER BY id LIMIT ? `), markSynced: this.db.prepare( 'UPDATE pending_entries SET synced = 1 WHERE id = ?' ), cleanupSynced: this.db.prepare(` DELETE FROM pending_entries WHERE synced = 1 AND created_at < datetime('now', '-7 days') `), countPending: this.db.prepare( 'SELECT COUNT(*) as count FROM pending_entries WHERE synced = 0' ), countSynced: this.db.prepare( 'SELECT COUNT(*) as count FROM pending_entries WHERE synced = 1' ), }; } ``` Why prepare statements upfront? Two reasons: 1. **Parse once, run many**: SQLite parses and compiles the SQL into bytecode once during `prepare()`. Each subsequent `run()` or `get()` skips parsing entirely. 2. **Type safety**: The `Statement<[string, number]>` generic tells TypeScript exactly what parameters this statement expects. Try to pass the wrong types, and you get a compile error. The `upsertPosition` statement uses SQLite's `ON CONFLICT` clause for atomic upserts—no need for separate "check if exists, then insert or update" logic. ### Transaction Wrapping for Batch Inserts When inserting multiple entries, we wrap them in a transaction: ```typescript insertEntries(entries: Array<{ projectId: string; sessionId?: string; sourceFile: string; data: ClaudeEntry; }>): void { const insertMany = this.db.transaction((items: typeof entries) => { for (const entry of items) { this.statements.insertEntry.run( entry.projectId, entry.sessionId || null, entry.sourceFile, JSON.stringify(entry.data) ); } }); insertMany(entries); } ``` The `better-sqlite3` transaction wrapper ensures all inserts succeed or all fail together. It also provides a massive performance boost—instead of committing after each insert, we commit once at the end. ## Deep Dive: The MongoDB Sync Engine The `MongoSync` class handles the final hop from SQLite to MongoDB. Its design prioritizes resilience over strict consistency. ### Connection Handling ```typescript // sync.ts import { MongoClient, Collection, Db } from 'mongodb'; import { Buffer } from './buffer'; import { MongoDocument, SyncStats } from './types'; export class MongoSync { private client: MongoClient | null = null; private collection: Collection | null = null; private buffer: Buffer; private uri: string; private dbName: string; private batchSize: number; private syncInterval: number; private intervalId: NodeJS.Timeout | null = null; private lastSyncAt: Date | null = null; async connect(): Promise { try { if (this.client) return true; this.client = await MongoClient.connect(this.uri, { serverSelectionTimeoutMS: 5000, connectTimeoutMS: 10000, }); const db: Db = this.client.db(this.dbName); this.collection = db.collection('conversations'); // Create indexes for common query patterns await this.collection.createIndex({ projectId: 1, timestamp: -1 }); await this.collection.createIndex({ sessionId: 1 }); await this.collection.createIndex({ 'ingestedAt': -1 }); await this.collection.createIndex({ message: 'text' }); console.log('MongoDB connected'); return true; } catch (err) { console.error('MongoDB connection failed:', (err as Error).message); this.client = null; this.collection = null; return false; } } // ... more methods } ``` The connection logic is defensive: - **Short timeouts** (5s for server selection, 10s for connection) prevent the service from hanging indefinitely if MongoDB is unreachable. - **Index creation** happens on every connect. MongoDB's `createIndex` is idempotent—if the index exists, it returns immediately. - **Graceful failure**: If connection fails, we log and return `false`. The caller can decide what to do (in our case, buffer the entries and retry later). ### The Sync Cycle Here is where things get interesting: ```typescript async sync(): Promise { const connected = await this.connect(); if (!connected || !this.collection) { const stats = this.buffer.getStats(); if (stats.pending > 0) { console.log(`MongoDB unavailable, ${stats.pending} entries buffered`); } return 0; } const pending = this.buffer.getPendingEntries(this.batchSize); if (pending.length === 0) return 0; const docs: MongoDocument[] = pending.map((row) => { const entry = JSON.parse(row.entry_json); return { ...entry, projectId: row.project_id, sourceFile: row.source_file, ingestedAt: new Date(), }; }); try { await this.collection.insertMany(docs, { ordered: false }); this.buffer.markAsSynced(pending.map((r) => r.id)); this.lastSyncAt = new Date(); console.log(`Synced ${docs.length} entries to MongoDB`); return docs.length; } catch (err: unknown) { const mongoErr = err as { code?: number; writeErrors?: Array<{ code: number }> }; // Handle duplicate key errors (partial success) if (mongoErr.code === 11000 || mongoErr.writeErrors?.some(e => e.code === 11000)) { this.buffer.markAsSynced(pending.map((r) => r.id)); console.log('Handled duplicates, marked as synced'); return docs.length; } console.error('Sync error:', (err as Error).message); await this.disconnect(); return 0; } } ``` ### The `ordered: false` Strategy This single option—`{ ordered: false }`—is perhaps the most important design decision in the entire sync engine: ```typescript await this.collection.insertMany(docs, { ordered: false }); ``` By default, `insertMany` is *ordered*: if document 5 out of 100 fails, MongoDB stops and documents 6-100 are never inserted. With `ordered: false`, MongoDB attempts to insert *all* documents, collecting errors along the way. Why does this matter? Consider this scenario: 1. We sync 100 entries to MongoDB 2. Entry 23 already exists (duplicate key) 3. With `ordered: true`: entries 24-100 are lost until next sync 4. With `ordered: false`: entries 24-100 are inserted, only 23 fails Combined with our duplicate handling code, this means we can safely retry syncs without worrying about partial failures: ```typescript if (mongoErr.code === 11000 || mongoErr.writeErrors?.some(e => e.code === 11000)) { this.buffer.markAsSynced(pending.map((r) => r.id)); console.log('Handled duplicates, marked as synced'); return docs.length; } ``` Error code 11000 means "duplicate key." When we see it, we know the data is already in MongoDB, so we mark it as synced and move on. No data loss, no infinite retry loops. ### The Sync State Machine ```mermaid stateDiagram-v2 [*] --> Idle Idle --> Connecting: sync() called Connecting --> Syncing: connected Connecting --> Idle: connection failed Syncing --> MarkingSynced: insertMany success Syncing --> HandlingDuplicates: duplicate key error Syncing --> Disconnecting: other error HandlingDuplicates --> MarkingSynced: mark as synced MarkingSynced --> Idle: complete Disconnecting --> Idle: cleanup complete ``` The periodic sync runs on a configurable interval: ```typescript startPeriodicSync(): void { // Initial sync this.sync(); // Periodic sync this.intervalId = setInterval(() => this.sync(), this.syncInterval); console.log(`Sync interval: ${this.syncInterval}ms`); } stopPeriodicSync(): void { if (this.intervalId) { clearInterval(this.intervalId); this.intervalId = null; } } ``` ## Deep Dive: Bootstrap and Graceful Shutdown The `index.ts` file ties everything together with proper lifecycle management. ### Configuration Loading ```typescript // index.ts import { config as loadEnv } from 'dotenv'; import path from 'path'; import os from 'os'; import http from 'http'; import { Buffer } from './buffer'; import { Watcher } from './watcher'; import { MongoSync } from './sync'; function expandTilde(filePath: string): string { if (filePath.startsWith('~/')) { return path.join(os.homedir(), filePath.slice(2)); } return filePath; } const config = { claudeDir: expandTilde(process.env.CLAUDE_DIR || path.join(os.homedir(), '.claude', 'projects')), mongoUri: process.env.MONGO_URI || 'mongodb://localhost:27017', dbName: process.env.MONGO_DB || 'claude_logs', sqlitePath: expandTilde(process.env.SQLITE_PATH || path.join(os.homedir(), '.claude-sync', 'buffer.db')), syncIntervalMs: parseInt(process.env.SYNC_INTERVAL_MS || '5000', 10), batchSize: parseInt(process.env.BATCH_SIZE || '100', 10), cleanupIntervalMs: parseInt(process.env.CLEANUP_INTERVAL_MS || '3600000', 10), healthPort: parseInt(process.env.HEALTH_PORT || '9090', 10), }; ``` The `expandTilde` helper handles `~/` paths, making configuration more user-friendly. All settings come from environment variables with sensible defaults. ### Component Initialization ```typescript async function main(): Promise { console.log('Starting Claude MongoDB Sync'); console.log(`Watch dir: ${config.claudeDir}`); console.log(`Buffer: ${config.sqlitePath}`); console.log(`MongoDB: ${config.mongoUri}/${config.dbName}`); // Initialize components in dependency order buffer = new Buffer(config.sqlitePath); watcher = new Watcher(config.claudeDir, buffer); watcher.start(); mongoSync = new MongoSync(buffer, { uri: config.mongoUri, dbName: config.dbName, batchSize: config.batchSize, syncIntervalMs: config.syncIntervalMs, }); mongoSync.startPeriodicSync(); // Periodic cleanup of old synced entries cleanupIntervalId = setInterval(() => { buffer.cleanup(); console.log('Cleanup completed'); }, config.cleanupIntervalMs); // ... health endpoint setup } ``` The initialization order matters: Buffer first (it is a dependency of both Watcher and MongoSync), then Watcher, then MongoSync. The cleanup interval removes synced entries older than 7 days to prevent unbounded SQLite growth. ### Health Endpoint ```typescript healthServer = http.createServer((req, res) => { if (req.url === '/health' || req.url === '/') { const stats = mongoSync.getStats(); const response = { status: 'ok', ...stats, uptime: process.uptime(), }; res.writeHead(200, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(response, null, 2)); } else if (req.url === '/stats') { const stats = mongoSync.getStats(); res.writeHead(200, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(stats, null, 2)); } else { res.writeHead(404); res.end('Not found'); } }); healthServer.listen(config.healthPort, () => { console.log(`Health endpoint: http://localhost:${config.healthPort}/health`); }); ``` A simple health endpoint that returns: ```json { "status": "ok", "pending": 42, "synced": 1337, "lastSyncAt": "2024-01-15T10:30:00.000Z", "mongoConnected": true, "uptime": 3600.5 } ``` This is invaluable for monitoring. You can set up alerts when `pending` grows too large or `mongoConnected` goes false. ### Graceful Shutdown ```typescript async function shutdown(signal: string): Promise { console.log(`\n${signal} received, shutting down gracefully...`); // Stop accepting new work watcher?.stop(); mongoSync?.stopPeriodicSync(); clearInterval(cleanupIntervalId); // Final sync attempt try { console.log('Final sync...'); await mongoSync?.sync(); } catch (e) { console.error('Final sync failed:', (e as Error).message); } // Cleanup await mongoSync?.disconnect(); buffer?.close(); healthServer?.close(); console.log('Shutdown complete'); process.exit(0); } // Graceful shutdown handlers process.on('SIGINT', () => shutdown('SIGINT')); process.on('SIGTERM', () => shutdown('SIGTERM')); process.on('uncaughtException', (err) => { console.error('Uncaught exception:', err); shutdown('uncaughtException'); }); ``` The shutdown sequence is carefully ordered: 1. **Stop accepting new work**: The watcher stops detecting file changes, and the sync interval is cleared. 2. **Final sync**: We attempt one last sync to minimize data left in the buffer. 3. **Close connections**: MongoDB connection and SQLite database are properly closed. 4. **Exit**: Clean exit with code 0. This ensures that `Ctrl+C` or a `kill` command does not result in data loss. ## Key Design Patterns Summary Let us recap the patterns that make this service robust: | Pattern | Problem Solved | Implementation | |---------|---------------|----------------| | SQLite as durable buffer | Crash recovery, MongoDB outages | `Buffer` class with WAL mode | | Prepared statements | SQL parsing overhead | Pre-compiled at startup | | Processing lock (Set) | Concurrent file access | `processing` Set in Watcher | | Unordered inserts | Partial failures | `{ ordered: false }` in insertMany | | Position tracking | Resume after restart | `file_positions` table | | Graceful shutdown | Data loss on termination | Signal handlers with final sync | ## Configuration Reference Create a `.env` file with these options: | Variable | Default | Description | |----------|---------|-------------| | `MONGO_URI` | `mongodb://localhost:27017` | MongoDB connection string | | `MONGO_DB` | `claude_logs` | Database name | | `CLAUDE_DIR` | `~/.claude/projects` | Directory to watch for JSONL files | | `SQLITE_PATH` | `~/.claude-sync/buffer.db` | SQLite buffer file location | | `SYNC_INTERVAL_MS` | `5000` | How often to sync (milliseconds) | | `BATCH_SIZE` | `100` | Entries per sync batch | | `HEALTH_PORT` | `9090` | Port for health endpoint | ## Production Deployment with PM2 For production, we use PM2 to manage the Node.js process. Here is our `ecosystem.config.js`: ```javascript module.exports = { apps: [ { name: 'claude-mongo-sync', script: './dist/index.js', instances: 1, autorestart: true, watch: false, max_memory_restart: '200M', env: { NODE_ENV: 'development', }, env_production: { NODE_ENV: 'production', }, // Restart behavior exp_backoff_restart_delay: 100, max_restarts: 10, min_uptime: '10s', // Logging log_date_format: 'YYYY-MM-DD HH:mm:ss Z', error_file: '~/.pm2/logs/claude-mongo-sync-error.log', out_file: '~/.pm2/logs/claude-mongo-sync-out.log', merge_logs: true, // Graceful shutdown kill_timeout: 10000, listen_timeout: 5000, shutdown_with_message: true, }, ], }; ``` Key PM2 settings explained: - **`max_memory_restart: '200M'`**: Automatic restart if memory exceeds 200MB, preventing memory leaks from becoming outages. - **`exp_backoff_restart_delay`**: Exponential backoff on crashes prevents rapid restart loops. - **`kill_timeout: 10000`**: Gives our graceful shutdown handler 10 seconds to complete the final sync. To deploy: ```bash # Build TypeScript npm run build # Start with PM2 pm2 start ecosystem.config.js --env production # Check status pm2 status # View logs pm2 logs claude-mongo-sync # Monitor health endpoint curl localhost:9090/health ``` ## Wrapping Up We have built a sync service that solves the three core problems of file-to-database synchronization: 1. **Crash recovery**: SQLite stores file positions, so we always know where we left off. 2. **Batching for efficiency**: Configurable batch sizes balance throughput and latency. 3. **Duplicate handling**: Unordered inserts with error code 11000 handling make retries safe. The code is straightforward—no complex frameworks, no magic. Just TypeScript, SQLite, and MongoDB, composed with careful attention to failure modes. In the next article, we will build a Next.js UI to browse and search these synced conversations. But that is a story for another day. --- **Want to try it yourself?** The complete source code is available on GitHub. Clone it, configure your `.env`, and run `npm run dev`. Watch as your JSONL files flow seamlessly into MongoDB. **Have questions or improvements?** Drop a comment below. I read every one. --- **Suggested Tags**: TypeScript, MongoDB, Node.js, Data Engineering, Backend Development

Building a Real-Time Claude AI Conversation Analytics Platform: Architecture Overview

Sat, 27 Dec 2025 00:00:00 GMT

**How to capture, sync, and analyze AI coding assistant conversations with a modern TypeScript + Python data stack** --- Every day, I have dozens of conversations with Claude Code. Some are brilliant problem-solving sessions where we refactor complex systems together. Others are debugging marathons that finally end with that satisfying "aha!" moment. And a few are... well, let's just say they teach me more about prompt engineering than I planned. But here's the thing: all those conversations disappear into the void. Or rather, they hide away in scattered JSONL files across my filesystem, never to be seen again. I wanted to change that. What if I could search through past conversations? What if I could see which tools Claude uses most often in my projects? What if I could understand my own AI-assisted development patterns? This article introduces a platform I built to do exactly that. We'll explore the architecture of a real-time sync system that captures Claude Code conversations and transforms them into searchable, analyzable data. By the end, you'll understand how the pieces fit together and be ready to dive deeper into each component in the articles that follow. --- ## The Problem: Invisible Conversations Claude Code stores your conversation history as JSONL files in `~/.claude/projects/`. Each project gets its own directory, and each session creates entries with timestamps, messages, and tool calls. Here's what a typical entry looks like: ```json { "type": "message", "sessionId": "abc123", "timestamp": "2024-12-15T10:30:00.000Z", "message": { "role": "assistant", "content": [ {"type": "text", "text": "I'll help you refactor that function..."}, {"type": "tool_use", "name": "Edit", "input": {...}} ] } } ``` This format works great for Claude Code's internal use, but it creates several challenges for developers who want to learn from their AI interactions: **No Search Capability**: Want to find that conversation where Claude helped you set up a complex Docker configuration? Good luck grepping through nested JSON across hundreds of files. **No Analytics**: How often does Claude use the `Edit` tool versus `Write`? Which projects generate the most conversation volume? These patterns remain hidden in raw log files. **No Historical Review**: Debugging a production issue and need to remember how you solved something similar three months ago? You'll be manually scanning through files trying to find it. **No Aggregation**: Each project lives in isolation. You can't query across your entire Claude Code usage history. These limitations matter because understanding your AI-assisted workflows can make you more productive. Knowing which tools work best for certain tasks, seeing patterns in successful debugging sessions, or identifying where you spend the most time with AI assistance are all valuable insights for improving how you work. --- ## The Solution: A Three-Component Architecture To solve these problems, I designed a platform with three distinct components, each handling a specific responsibility: ``` +------------------+ +------------------+ +------------------+ | Sync Service | | UI | | Analytics | | (TypeScript) | | (Next.js) | | (Python) | +------------------+ +------------------+ +------------------+ | | | | | | | - File watching | | - Browsing | | - ELT pipeline | | - SQLite buffer | | - Search | | - dbt models | | - MongoDB sync | | - Filtering | | - Dashboards | | | | | | | +--------+---------+ +--------+---------+ +--------+---------+ | | | v v v +--------------------------------------------+ | | MongoDB |<--------+ +--------------------------------------------+ ``` **Sync Service** handles the real-time ingestion. It watches JSONL files for changes, buffers entries in SQLite for resilience, and batch-syncs to MongoDB. Written in TypeScript for strong typing and excellent async file handling. **UI** provides a Next.js application for browsing and searching conversations. Built with shadcn/ui components, it offers filtering by project, session, and date range, plus visualization of conversation patterns over time. **Analytics** runs a Python-based ELT pipeline. It extracts data from MongoDB to Parquet files, loads them into DuckDB, and uses dbt to transform raw data through Bronze, Silver, and Gold layers. Metabase provides self-service dashboards. Why three separate components instead of one monolith? Each serves a different access pattern: - **Sync** needs to run continuously, handling file events as they happen - **UI** needs to serve interactive queries with low latency - **Analytics** needs to run batch transformations on large datasets Separating them means each can scale, deploy, and fail independently. --- ## Complete Data Flow Here's the complete architecture, from JSONL files to analytics dashboards: ``` ~/.claude/projects/**/*.jsonl | v +---------+ | Watcher | (chokidar) +----+----+ | v +---------+ | SQLite | (buffer.db) | Buffer | +----+----+ | v +---------+ | MongoDB | +----+----+ | +----+----+ | | v v +------+ +----------+ | UI | | Analytics| |(3000)| | Extractor| +------+ +----+-----+ | v +---------+ | DuckDB | <- dbt (Bronze->Silver->Gold) +----+----+ | v +----------+ | Metabase | | (3001) | +----------+ ``` Let me walk you through what happens when you have a conversation with Claude Code: **1. Claude Code writes entries** to a JSONL file in `~/.claude/projects/[project-hash]/conversations.jsonl`. **2. Chokidar detects the file change** within milliseconds. The Watcher component tracks file positions using SQLite, so it only reads new lines, not the entire file. **3. New entries go into the SQLite buffer**. This is the critical resilience layer. If MongoDB is down, entries wait safely in SQLite. If the process restarts, file positions are preserved. **4. Every 5 seconds, the sync worker** pulls pending entries from SQLite and batch-inserts them into MongoDB. Failed syncs leave entries in pending state for retry. **5. MongoDB stores the canonical data**, indexed for efficient queries by project, session, and timestamp. Both the UI and Analytics pipeline read from here. **6. The UI queries MongoDB directly** for interactive browsing. Users can search, filter by date range, and drill into specific sessions. **7. The Analytics Extractor** periodically pulls data from MongoDB and writes Parquet files with date-based partitioning. A high-water mark ensures incremental extraction. **8. DuckDB loads the Parquet files**, providing a fast analytical query engine. dbt models transform raw data through the medallion architecture. **9. Metabase connects to DuckDB** for self-service analytics and dashboards. This flow provides real-time sync (seconds of latency) for the UI while enabling batch analytics workloads that can run on larger time windows. --- ## Component Deep Dive Let's explore each component at a higher level. Subsequent articles in this series will provide implementation details. ### Sync Service: The Real-Time Backbone The sync service consists of three TypeScript classes with clear responsibilities: ```typescript // watcher.ts - Monitors JSONL files export class Watcher { private watcher: FSWatcher | null = null; private buffer: Buffer; private processing = new Set(); // Prevents concurrent file processing start(): void { this.watcher = chokidar.watch(`${this.watchDir}/**/*.jsonl`, { persistent: true, ignoreInitial: false, awaitWriteFinish: { stabilityThreshold: 300, pollInterval: 100 }, }); this.watcher.on('add', (filePath) => this.processFile(filePath)); this.watcher.on('change', (filePath) => this.processFile(filePath)); } } ``` The `Watcher` uses chokidar to monitor all JSONL files recursively. The `awaitWriteFinish` option ensures we don't read partial lines while Claude Code is still writing. ```typescript // buffer.ts - SQLite persistence layer export class Buffer { private db: DatabaseType; private statements: { getPosition: Statement<[string]>; insertEntry: Statement<[string, string | null, string, string]>; getPending: Statement<[number]>; markSynced: Statement<[number]>; // ... more prepared statements }; constructor(dbPath?: string) { this.db = new Database(dbPath); this.db.pragma('journal_mode = WAL'); // Better concurrent performance this.statements = this.prepareStatements(); } } ``` The `Buffer` uses SQLite with WAL mode for concurrent reads and writes. All queries are prepared at startup for performance. The buffer tracks two key things: file positions (where we left off reading each file) and pending entries (waiting to sync). ```typescript // sync.ts - MongoDB batch sync export class MongoSync { async sync(): Promise { const pending = this.buffer.getPendingEntries(this.batchSize); if (pending.length === 0) return 0; const docs = pending.map((row) => ({ ...JSON.parse(row.entry_json), projectId: row.project_id, ingestedAt: new Date(), })); // Unordered insert allows partial success on duplicates await this.collection.insertMany(docs, { ordered: false }); this.buffer.markAsSynced(pending.map((r) => r.id)); return docs.length; } } ``` The `MongoSync` class pulls pending entries in batches and inserts them to MongoDB. Using `ordered: false` means a duplicate key error won't stop the entire batch, only the specific duplicate. This is essential for idempotent retries. ### UI: Conversation Browser The UI is a Next.js 14 application using shadcn/ui components. It connects directly to MongoDB for low-latency queries. ```tsx // page.tsx - Main conversation viewer function ConversationViewer() { const { data, fetchNextPage, hasNextPage, isLoading } = useConversations({ projectId, sessionId, search, startDate, endDate, sortOrder, }); return ( <> fetchNextPage()} /> ); } ``` Key features include: - **Infinite scroll** with React Query for efficient pagination - **Project and session filtering** to narrow down results - **Date range selection** with chart-based filtering - **Full-text search** across conversation content - **Export functionality** for offline analysis The session chart provides a visual timeline of conversation activity, and clicking a bar filters the list to that time period. ### Analytics Pipeline: From Raw to Insights The analytics component uses a modern Python data stack: ``` MongoDB -> Extractor -> Parquet Files -> DuckDB -> dbt -> Metabase ``` **Extractor** (Python with PyMongo): ```python class MongoExtractor: def extract(self, full_backfill: bool = False) -> list[Path]: # Track where we left off since = None if full_backfill else self.high_water_mark.get() for doc in self._fetch_documents(since=since): record = self.transformer.transform(doc, extracted_at) records_by_date[date_key].append(record) # Write date-partitioned Parquet files for date_key, records in records_by_date.items(): self._write_partition(records, partition_date, output_dir) ``` The extractor pulls from MongoDB, transforms nested message structures into flat records, and writes Parquet files partitioned by date. A high-water mark file tracks the last extraction timestamp for incremental runs. **Loader** (DuckDB): ```python class DuckDBLoader: def load_from_parquet(self, parquet_path: Path) -> int: # Upsert with conflict handling self.conn.execute(""" INSERT INTO raw.conversations SELECT * FROM read_parquet('{path}', hive_partitioning=true) ON CONFLICT (_id) DO UPDATE SET ... """) ``` DuckDB's native Parquet reader with Hive partitioning support makes loading efficient. The upsert pattern handles re-extraction gracefully. **dbt Models** (Medallion Architecture): ``` staging/ <- Bronze: Clean source data stg_conversations.sql stg_messages.sql stg_tool_calls.sql intermediate/ <- Silver: Enriched entities int_messages_enriched.sql int_sessions_computed.sql int_tool_usage.sql marts/ <- Gold: Analytics-ready dim_projects.sql dim_sessions.sql fct_messages.sql fct_tool_calls.sql agg_daily_summary.sql ``` The staging layer cleans and types the raw data. Intermediate models enrich entities with computed fields. Marts provide fact and dimension tables optimized for BI tools. **Prefect Orchestration**: ```python @flow(name="claude-analytics-pipeline") def analytics_pipeline( full_backfill: bool = False, full_refresh: bool = False, ) -> dict: # Step 1: Extract from MongoDB extraction_stats = extract_task(full_backfill=full_backfill) # Step 2: Load into DuckDB load_stats = load_task(extraction_stats, full_refresh=full_refresh) # Step 3: Run dbt transformations transform_stats = transform_task(load_stats, full_refresh=full_refresh) return {"extraction": extraction_stats, "load": load_stats, "transform": transform_stats} ``` Prefect orchestrates the ELT pipeline with retries, logging, and scheduling. You can run ad-hoc backfills or schedule hourly incremental runs. --- ## Key Design Decisions Several design choices make this architecture robust: ### SQLite Buffer with WAL Mode Why buffer through SQLite instead of writing directly to MongoDB? Resilience. If MongoDB goes down, you don't lose entries. If the sync process crashes, file positions are preserved. When everything comes back up, pending entries sync automatically. WAL (Write-Ahead Logging) mode enables concurrent reads and writes without blocking. The watcher can insert new entries while the sync worker reads pending ones. ### Processing Lock for File Events ```typescript private processing = new Set(); private async processFile(filePath: string): Promise { if (this.processing.has(filePath)) return; this.processing.add(filePath); try { // ... process file } finally { this.processing.delete(filePath); } } ``` File watchers can fire multiple events rapidly. This Set prevents concurrent processing of the same file, avoiding race conditions where two handlers might read overlapping content. ### Unordered MongoDB Inserts ```typescript await this.collection.insertMany(docs, { ordered: false }); ``` With `ordered: false`, a duplicate key error on one document doesn't fail the entire batch. This is crucial for idempotent retries. If a batch partially succeeds and then fails, re-running it will skip the duplicates and insert the rest. ### High Water Mark for Incremental Extraction ```python class HighWaterMark: def get(self) -> datetime | None: # Read last extraction timestamp from file def set(self, timestamp: datetime) -> None: # Update after successful extraction ``` Instead of re-extracting everything, the analytics pipeline tracks the last successfully extracted timestamp. Each run only pulls documents newer than the high water mark, making hourly incremental runs efficient. ### Medallion Architecture for dbt Organizing dbt models into Bronze (staging), Silver (intermediate), and Gold (marts) layers provides several benefits: - **Testability**: Each layer has specific data quality expectations - **Debuggability**: You can query intermediate results when something breaks - **Reusability**: Silver layer entities feed multiple Gold layer aggregations - **Performance**: Materializing intermediate results speeds up downstream models --- ## Deployment Strategy The platform uses different deployment approaches for each component: ### Sync Service: PM2 ```bash # Install PM2 globally npm install -g pm2 # Start the sync service pm2 start ecosystem.config.js # Useful commands pm2 status pm2 logs claude-mongo-sync pm2 restart claude-mongo-sync # Auto-start on boot pm2 startup pm2 save ``` PM2 provides process management, automatic restarts on failure, and log aggregation. For production Linux servers, a systemd service file is also available. ### Analytics: Docker Compose ```bash cd analytics make up # Start Prefect + Metabase make deploy # Deploy pipeline flows make run-backfill # Initial full extraction ``` Docker Compose orchestrates the analytics services, including the Prefect server, worker, and Metabase. Volumes persist DuckDB data and Metabase configuration. ### Port Assignments | Service | Port | Purpose | |---------|------|---------| | Sync Health | 9090 | Health check endpoint | | UI | 3000 | Next.js application | | Metabase | 3001 | Analytics dashboards | | Prefect UI | 4200 | Pipeline orchestration | | dbt Docs | 8080 | Data model documentation | The health endpoint at `localhost:9090/health` returns sync status: ```json { "status": "ok", "pending": 0, "synced": 1523, "lastSyncAt": "2024-12-15T10:30:00.000Z", "mongoConnected": true, "uptime": 3600 } ``` --- ## What's Next: The Series Roadmap This article provided the architecture overview. The next six articles will dive deep into each component: **Article 2: Real-Time File Watching with Chokidar and SQLite Buffering** We'll implement the Watcher and Buffer classes from scratch, exploring file system events, SQLite prepared statements, and graceful error handling. **Article 3: Resilient MongoDB Sync with TypeScript** Building the MongoSync class with connection management, batch processing, duplicate handling, and the health endpoint. **Article 4: Building a Conversation Browser with Next.js and shadcn/ui** Creating the UI with infinite scroll, filtering, search, and interactive charts using React Query and Recharts. **Article 5: ELT Pipeline Design: MongoDB to DuckDB with Python** Implementing the Extractor and Loader with PyMongo, PyArrow, and DuckDB. Covering high water mark tracking and Parquet partitioning. **Article 6: Medallion Architecture with dbt: From Raw to Analytics** Designing dbt models across Bronze, Silver, and Gold layers. Writing data quality tests and generating documentation. **Article 7: Self-Service Analytics with Metabase Dashboards** Connecting Metabase to DuckDB, building dashboards for conversation analytics, and deploying with Docker Compose. --- ## Conclusion Building an analytics platform for Claude AI conversations transforms scattered log files into searchable, analyzable data. The three-component architecture separates concerns: real-time sync handles ingestion, a Next.js UI enables browsing, and a Python pipeline powers analytics. Key takeaways: - **Resilience through buffering**: SQLite bridges the gap between file events and MongoDB availability - **Separation of concerns**: Each component scales and fails independently - **Modern data stack**: dbt's medallion architecture brings software engineering practices to data transformation If you're using Claude Code regularly, understanding your conversation patterns can make you more productive. This platform gives you the tools to do exactly that. The complete source code is available on [GitHub](https://github.com/farshad-akbari-hb/claude-code-conversation-analytics). In the next article, we'll start building: implementing real-time file watching with chokidar and SQLite buffering.