SecondBrain - Collaborative AI Assistant with RAG

🚀 Current Status

📁 Project Structure

secondbrain/
├── frontend/                    # SvelteKit Frontend
│   ├── src/
│   │   ├── lib/
│   │   │   ├── components/     # Svelte components (ChatMessage, Sidebar, etc.)
│   │   │   └── api.ts          # API client with TypeScript types
│   │   ├── routes/
│   │   │   ├── +layout.svelte  # Root layout with CSS imports
│   │   │   └── +page.svelte    # Main chat interface
│   │   └── app.pcss           # TailwindCSS imports
│   ├── package.json
│   ├── tailwind.config.js     # TailwindCSS v3 configuration
│   └── vite.config.ts
│
├── backend/                     # FastAPI Backend
│   ├── app/
│   │   ├── api/
│   │   │   ├── chat.py         # Chat API endpoints
│   │   │   ├── tags.py         # Tag management API endpoints
│   │   │   └── document_tags.py # Document-tag association endpoints
│   │   ├── models/
│   │   │   ├── database.py     # SQLAlchemy async setup
│   │   │   └── chat.py         # Database models (Chat, Message, Document, Tag)
│   │   ├── schemas/
│   │   │   ├── chat.py         # Pydantic models for API
│   │   │   └── tag.py          # Tag-related Pydantic models
│   │   ├── services/
│   │   │   ├── azure_openai.py # Azure OpenAI integration
│   │   │   ├── chat_service.py # Business logic with tag filtering
│   │   │   └── embedding_service.py # Vector search with tag filtering
│   │   ├── config.py           # Settings and environment variables
│   │   └── main.py             # FastAPI application
│   ├── alembic/                # Database migrations
│   ├── requirements.txt        # Python dependencies
│   ├── run.py                  # Development server
│   └── .env.example            # Environment template
│
├── database/                    # Database Setup (Future)
│   └── docker-compose.yml      # PostgreSQL + pgvector container
│
└── README.md                   # This documentation

🚀 Quick Start

Prerequisites

Option 1: Docker Compose (Recommended)

Development Mode

# Clone the repository
git clone <repository-url>
cd secondbrain

# Start in development mode with hot-reloading
./dev.sh

# Or manually:
docker-compose -f docker-compose.dev.yml up

Production Mode

# Copy and configure environment variables
cp .env.example .env
# Edit .env with your LLM provider settings (see configuration section below)

# Start in production mode
./prod.sh --build

# Or manually:
docker-compose -f docker-compose.prod.yml up --build

Option 2: Manual Setup

1. Backend Setup

cd backend

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Copy environment template
cp .env.example .env
# Edit .env with your LLM provider settings (see configuration section)

# Start the development server
python run.py

2. Frontend Setup

cd frontend

# Install dependencies
npm install

# Start development server
npm run dev

3. Database Setup

# Start PostgreSQL with pgvector
docker run -d \
  --name secondbrain-postgres \
  -e POSTGRES_DB=secondbrain \
  -e POSTGRES_USER=secondbrain \
  -e POSTGRES_PASSWORD=your_password \
  -p 5432:5432 \
  pgvector/pgvector:pg16

3. Test the System

4. Test Your LLM Provider

# Test if Ollama is running
curl http://localhost:11434/api/tags

# Test chat completion
curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "smollm2:135m",
    "messages": [{"role": "user", "content": "Hello, are you working?"}],
    "stream": false
  }'

# Test embeddings
curl -X POST http://localhost:11434/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mxbai-embed-large",
    "input": "Hello world"
  }'

5. Try the Tag System

🐳 Docker Deployment

Development vs Production

Development Mode

# Quick start
./dev.sh

# With specific options
docker-compose -f docker-compose.dev.yml up -d
docker-compose -f docker-compose.dev.yml logs -f

Production Mode

# Configure environment
cp .env.example .env
# Edit .env with your production settings

# Deploy
./prod.sh --build

Environment Configuration

# PostgreSQL Configuration
POSTGRES_USER=secondbrain
POSTGRES_PASSWORD=your_secure_password_here

# Azure OpenAI Configuration
AZURE_OPENAI_API_KEY=your_azure_openai_api_key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_DEPLOYMENT_NAME=your-deployment-name
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME=your-embedding-deployment-name
AZURE_OPENAI_API_VERSION=2023-05-15

# Port Configuration (optional)
BACKEND_PORT=8000
FRONTEND_PORT=3000

Azure Storage Integration

# In docker-compose.prod.yml
volumes:
  uploads:
    driver: azure_file
    driver_opts:
      share_name: secondbrain-uploads
      storage_account_name: ${AZURE_STORAGE_ACCOUNT}

Helper Scripts

⚙️ Configuration

LLM Provider Setup

Option A: Ollama Configuration (Recommended for Local Development)

# Download and install from https://ollama.com/
# Or use package manager:
curl -fsSL https://ollama.com/install.sh | sh

# Chat model (choose one based on your hardware):
ollama pull smollm2:135m        # Lightweight (135M parameters)
ollama pull llama3.2:3b         # Medium (3B parameters)
ollama pull llama3.2:8b         # Larger (8B parameters)

# Embedding model:
ollama pull mxbai-embed-large   # High-quality embeddings
# OR
ollama pull nomic-embed-text    # Alternative embedding model

# LLM Provider Selection
LLM_PROVIDER=ollama

# Ollama Configuration
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=smollm2:135m
OLLAMA_EMBEDDING_MODEL=mxbai-embed-large

# Database Configuration
DATABASE_URL=postgresql+asyncpg://secondbrain:password@postgres:5432/secondbrain
POSTGRES_USER=secondbrain
POSTGRES_PASSWORD=password
POSTGRES_DB=secondbrain

# CORS Configuration
CORS_ORIGINS=http://localhost:3000,http://localhost:5173,http://localhost:4173

# Optional: Secret key for sessions
SECRET_KEY=your_secret_key_here

Option B: Azure OpenAI Configuration

# LLM Provider Selection
LLM_PROVIDER=azure_openai

# Azure OpenAI Configuration
AZURE_OPENAI_API_KEY=your_azure_openai_api_key
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2023-12-01-preview
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME=text-embedding-ada-002

# Database Configuration
DATABASE_URL=postgresql+asyncpg://secondbrain:password@postgres:5432/secondbrain
POSTGRES_USER=secondbrain
POSTGRES_PASSWORD=password
POSTGRES_DB=secondbrain

# CORS Configuration
CORS_ORIGINS=http://localhost:3000,http://localhost:5173,http://localhost:4173

# Secret Key
SECRET_KEY=your_secret_key_here

Recommended Model Combinations

Hardware Requirements

Frontend Configuration

const API_BASE_URL = 'http://localhost:8000/api/v1';  // Change this for production

🔌 API Endpoints

Chat Management

Chat Interaction

Knowledge Base

Tag Management

Document-Tag Associations

System

API Usage Examples

Send Chat Message (Streaming)

const response = await fetch('http://localhost:8000/api/v1/chat', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    message: "Explain quantum computing in simple terms",
    chat_id: null // Creates new chat, or use existing chat ID
  })
});

// Handle Server-Sent Events streaming response
const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  
  const chunk = decoder.decode(value);
  const lines = chunk.split('\n');
  
  for (const line of lines) {
    if (line.startsWith('data: ')) {
      const data = JSON.parse(line.slice(6));
      
      if (data.type === 'content') {
        console.log('Streaming content:', data.content);
      } else if (data.type === 'done') {
        console.log('Message complete:', data.message_id);
      }
    }
  }
}

Get Chat History

// Get all chats
const chats = await fetch('http://localhost:8000/api/v1/chats').then(r => r.json());

// Get specific chat with messages
const chat = await fetch(`http://localhost:8000/api/v1/chats/${chatId}`).then(r => r.json());

🏗️ Architecture Overview

Frontend Stack (SvelteKit + TypeScript)

Backend Stack (FastAPI + Python)

Database Design (PostgreSQL + pgvector)

-- Core chat functionality
CREATE TABLE chats (
    id UUID PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE TABLE messages (
    id UUID PRIMARY KEY,
    chat_id UUID REFERENCES chats(id) ON DELETE CASCADE,
    content TEXT NOT NULL,
    role VARCHAR(20) CHECK (role IN ('user', 'assistant')),
    token_count INTEGER,
    embedding VECTOR(1536),  -- OpenAI ada-002 dimensions
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- RAG document support (Phase 2)
CREATE TABLE documents (
    id UUID PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    source_type VARCHAR(50) NOT NULL,  -- 'chat', 'file', 'url', etc.
    source_id VARCHAR(255),  -- chat_id, file_path, url, etc.
    filename VARCHAR(255),  -- Only for file source_type
    file_type VARCHAR(50),  -- Only for file source_type
    file_size INTEGER,  -- Only for file source_type
    document_metadata TEXT,  -- JSON for source-specific metadata
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE TABLE document_chunks (
    id UUID PRIMARY KEY,
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    content TEXT NOT NULL,
    chunk_index INTEGER NOT NULL,
    token_count INTEGER NOT NULL,
    embedding VECTOR(1536) NOT NULL,
    document_metadata TEXT,  -- JSON
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Tagging system (Phase 2.5)
CREATE TABLE tags (
    id UUID PRIMARY KEY,
    name VARCHAR(50) UNIQUE NOT NULL,
    description TEXT,
    color VARCHAR(7) DEFAULT '#808080',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE TABLE document_tags (
    document_id UUID REFERENCES documents(id) ON DELETE CASCADE,
    tag_id UUID REFERENCES tags(id) ON DELETE CASCADE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    PRIMARY KEY (document_id, tag_id)
);

-- Indexes for performance
CREATE INDEX idx_tags_name ON tags(name);
CREATE INDEX idx_document_tags_document ON document_tags(document_id);
CREATE INDEX idx_document_tags_tag ON document_tags(tag_id);

Data Flow Architecture

┌─────────────────┐    HTTP/SSE     ┌──────────────────┐    OpenAI API    ┌─────────────────┐
│   SvelteKit     │◄────────────────┤    FastAPI       │◄─────────────────┤  Azure OpenAI   │
│   Frontend      │                 │    Backend       │                  │   (GPT-4)       │
└─────────────────┘                 └──────────────────┘                  └─────────────────┘
         │                                     │
         │                                     │ SQLAlchemy
         │                                     ▼
         │                          ┌──────────────────┐
         │                          │   PostgreSQL     │
         │                          │   + pgvector     │
         └──────────────────────────┤   Database       │
              WebSocket (Future)    └──────────────────┘

🗺️ Development Roadmap

✅ Phase 1: Core Chat System (COMPLETE)

✅ Phase 2: Database & RAG Integration (COMPLETE)

✅ Phase 2.5: Document Tagging System (COMPLETE)

🚧 Phase 3: Production & Advanced Features (IN PROGRESS)

💭 Future Enhancement: Advanced RAG for Large Documents

Current RAG Limitations

Proposed Solution: Automated Document Summaries + Smart Context Expansion

Document Upload Pipeline:
1. Original Document → Chunked for detailed RAG
2. Generate Executive Summary (2-3 sentences)
3. Generate Section Summaries (paragraph per major section)
4. Generate Detailed Summary (comprehensive overview)
5. Store document outline/structure metadata

// Query types would automatically select appropriate summary level:
"What is this document about?" → Executive Summary
"Summarize the methodology section" → Section Summary 
"Give me a comprehensive overview" → Detailed Summary + key chunks
"What does it say about performance?" → Standard RAG chunks

Alternative Approaches Considered

Implementation Roadmap

Technical Considerations

def select_rag_strategy(query, referenced_docs):
    summary_triggers = ["summarize", "overview", "explain the whole", "main points"]
    
    if any(trigger in query.lower() for trigger in summary_triggers):
        return "summary_mode"
    elif referenced_docs and len(query.split()) < 8:
        return "detailed_summary_mode" 
    else:
        return "standard_rag_mode"

Next Steps

🧠 Future Enhancement: Short-Term Memory System

Current Conversation Limitations

Proposed Solution: Sliding Window Short-Term Memory

Recent Memory Window (7-14 days):
1. Key Topics & Entities → Extracted and weighted by recency/frequency
2. User Preferences → Communication style, preferred explanations depth
3. Ongoing Projects → Multi-session work contexts
4. Recent Insights → Important conclusions and decisions
5. Conversation Patterns → Common question types and domains

Technical Architecture

-- Short-term memory storage
CREATE TABLE memory_contexts (
    id UUID PRIMARY KEY,
    user_id UUID, -- Future: for multi-user support
    context_type VARCHAR(50), -- 'topic', 'preference', 'project', 'insight'
    content TEXT NOT NULL,
    entities JSONB, -- Extracted entities (people, places, concepts)
    relevance_score FLOAT DEFAULT 1.0,
    decay_factor FLOAT DEFAULT 0.95, -- Daily relevance decay
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    last_accessed TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE -- Automatic cleanup
);

-- Conversation-to-memory links
CREATE TABLE conversation_memory_links (
    conversation_id UUID REFERENCES chats(id) ON DELETE CASCADE,
    memory_context_id UUID REFERENCES memory_contexts(id) ON DELETE CASCADE,
    relevance_score FLOAT DEFAULT 1.0,
    PRIMARY KEY (conversation_id, memory_context_id)
);

-- Memory decay and access tracking
CREATE INDEX idx_memory_contexts_decay ON memory_contexts(decay_factor, last_accessed);
CREATE INDEX idx_memory_contexts_expiry ON memory_contexts(expires_at);

class ShortTermMemoryService:
    
    async def extract_conversation_context(self, chat: Chat) -> List[MemoryContext]:
        """Extract key information from completed conversation"""
        # 1. Entity extraction (people, projects, concepts)
        # 2. Topic identification and weighting
        # 3. User preference detection (explanation style, depth)
        # 4. Decision points and conclusions
        # 5. Ongoing project identification
        
    async def update_memory_relevance(self):
        """Daily task: apply decay factors and cleanup expired memories"""
        # Apply time-based decay (0.95 daily factor)
        # Remove memories below threshold (0.1 relevance)
        # Consolidate similar/duplicate memories
        
    async def retrieve_relevant_memories(self, query: str, limit: int = 5) -> List[MemoryContext]:
        """Get relevant short-term memories for current query"""
        # Vector similarity search on memory content
        # Weight by recency and relevance score
        # Filter by user context

Memory Integration with Chat

async def generate_response_with_memory(
    db: AsyncSession,
    chat_id: UUID,
    user_message: str,
    use_rag: bool = True,
    use_memory: bool = True
) -> AsyncGenerator:
    
    # 1. Standard RAG retrieval (current implementation)
    rag_context = await embedding_service.similarity_search(...)
    
    # 2. Short-term memory retrieval (NEW)
    if use_memory:
        memory_contexts = await memory_service.retrieve_relevant_memories(
            query=user_message,
            user_id=current_user_id, 
            limit=3
        )
    
    # 3. Enhanced system prompt construction
    system_prompt = build_enhanced_system_prompt(
        base_prompt=base_system_prompt,
        rag_context=rag_context,
        memory_context=memory_contexts,  # NEW
        conversation_history=chat.messages
    )

You are ChatGPT with access to:

## Recent Context (Short-term Memory):
- Project Context: User is working on a SvelteKit application with FastAPI backend (discussed 3 days ago)
- Communication Style: Prefers detailed technical explanations with code examples
- Ongoing Topics: Docker deployment, RAG implementation, TypeScript integration
- Recent Decisions: Chose PostgreSQL over MongoDB for vector storage (2 days ago)

## Current Knowledge Base (RAG):
[Standard RAG context...]

## Conversation History:
[Current chat messages...]

Based on this context, respond naturally while maintaining awareness of the user's ongoing projects and preferences.

Memory Types and Examples

Previous Chat: "Help me implement authentication in my SvelteKit app"
Current Chat: "How do I add rate limiting?"
Memory Bridge: Knows you're still working on the same SvelteKit project

Memory: User prefers TypeScript examples over JavaScript
Memory: User wants production-ready code with error handling
Auto-adapts: Provides TypeScript examples with comprehensive error handling

Memory: Working on SecondBrain chat application
Memory: Using Azure OpenAI, PostgreSQL, Docker
Current Query: "How to optimize performance?"
Context: Knows to suggest database indexing, caching, Azure-specific optimizations

Memory: Decided against using WebSocket in favor of SSE (1 week ago)
Current Query: "Should I implement real-time features?"
Response: References previous decision and builds on established architecture

Implementation Phases

Privacy and Performance Considerations

User Experience Enhancements

[Memory: Working on Docker deployment for SecondBrain app]
Based on your recent work with Docker, here's how to optimize your container build times...

Benefits Over Alternatives

🚀 Next Steps to Add RAG

✅ Already Implemented

📝 How to Use Current Features

🔜 Still To Do

🤝 Contributing

Project Structure Guide

Development Guidelines

Key Integration Points

📄 License

🆘 Troubleshooting

Common Issues

Debug Commands

# Check backend logs
docker-compose -f docker-compose.dev.yml logs backend

# Test Ollama connectivity
curl http://localhost:11434/api/tags

# Test backend health
curl http://localhost:8000/api/v1/health

# Check database connection
docker-compose -f docker-compose.dev.yml exec postgres psql -U secondbrain -d secondbrain -c "\dt"

# View environment variables
docker-compose -f docker-compose.dev.yml exec backend env | grep -E "(LLM_PROVIDER|OLLAMA_|AZURE_)"