← Back to DocsView raw

Technical Specifications

Detailed requirements and user flows

Chatbot Creation Framework - Technical Specifications

Project Overview

A comprehensive chatbot creation framework that enables users to build and deploy multimodal RAG chatbots with document processing capabilities, integrating with n8n workflows for advanced document processing and vector storage.

Core Features

1. User Management

  • Multi-provider Authentication: Email, Google, GitHub via NextAuth.js
  • User Profiles: Basic profile management with avatar support
  • Session Management: Secure JWT-based sessions with refresh tokens

2. Project Management

  • Project Creation: Wizard-based chatbot project setup
  • Configuration Management: System prompts, n8n URLs, processing settings
  • Resource Provisioning: Automatic database and storage bucket creation
  • Project Dashboard: Overview of all user projects with status indicators

3. Document Processing

  • Multi-format Support: PDF, Markdown, DOCX, TXT (extensible)
  • File Upload: Drag-and-drop interface with progress tracking
  • Processing Pipeline: Integration with n8n workflows for RAG processing
  • Image Extraction: Automatic extraction and storage of document images
  • Status Tracking: Real-time processing status updates

4. Chat Interface

  • Real-time Messaging: WebSocket-based chat with typing indicators
  • Conversation History: Persistent chat history with search capabilities
  • Multi-session Support: Multiple concurrent conversations per project
  • Rich Responses: Support for text, images, and structured data

5. Infrastructure Management

  • Database Provisioning: Automatic PostgreSQL database creation per project
  • Storage Management: MinIO bucket creation and lifecycle management
  • Resource Monitoring: Usage tracking and alerts
  • Backup & Recovery: Automated backup strategies

Technical Requirements

Frontend Requirements

  • Framework: Next.js 14+ with App Router
  • Language: TypeScript 5+
  • UI Library: Tailwind CSS + Headless UI or shadcn/ui
  • State Management: Zustand or React Query for server state
  • Real-time: Socket.io-client for WebSocket connections
  • File Upload: React Dropzone with progress tracking
  • Charts: Chart.js or Recharts for analytics

Backend Requirements

  • Runtime: Node.js 18+
  • Framework: Next.js API Routes
  • Database: PostgreSQL 15+ with pgvector extension
  • ORM: Prisma or Drizzle ORM
  • Storage: MinIO client for S3-compatible storage
  • Caching: Redis for session and data caching
  • Queue: Bull/BullMQ for background job processing
  • Real-time: Socket.io for WebSocket connections

Infrastructure Requirements

  • Containerization: Docker & Docker Compose
  • Database: PostgreSQL with vector extensions
  • Cache: Redis 7+
  • Storage: MinIO server
  • Reverse Proxy: Nginx for load balancing
  • Monitoring: Health check endpoints and metrics

User Flows

1. User Onboarding Flow

flowchart TD
    A[Landing Page] --> B[Sign Up/Login]
    B --> C[Choose Auth Provider]
    C --> D[Complete Profile]
    D --> E[Dashboard]
    E --> F[Create First Project]
    F --> G[Project Configuration]
    G --> H[Upload First Document]
    H --> I[Start Chatting]

2. Project Creation Flow

flowchart TD
    A[Dashboard] --> B[New Project Button]
    B --> C[Project Details Form]
    C --> D[System Prompt Configuration]
    D --> E[n8n Integration Setup]
    E --> F[Resource Provisioning]
    F --> G{Provisioning Success?}
    G -->|Yes| H[Project Created]
    G -->|No| I[Error Handling]
    I --> J[Retry/Manual Setup]
    H --> K[Upload Documents]

3. Document Processing Flow

flowchart TD
    A[Document Upload] --> B[File Validation]
    B --> C[Store in MinIO]
    C --> D[Create Processing Job]
    D --> E[Trigger n8n Workflow]
    E --> F[n8n Processing]
    F --> G[Extract Text & Images]
    G --> H[Generate Embeddings]
    H --> I[Store in Vector DB]
    I --> J[Update Job Status]
    J --> K[Notify User]

4. Chat Interaction Flow

flowchart TD
    A[User Message] --> B[Message Validation]
    B --> C[Store Message]
    C --> D[Vector Search]
    D --> E[Retrieve Context]
    E --> F[Generate Response]
    F --> G[Store Response]
    G --> H[Send to User]
    H --> I[Update UI]

API Specifications

Authentication APIs

// POST /api/auth/signup
interface SignupRequest {
  email: string;
  password: string;
  name: string;
}

// POST /api/auth/signin
interface SigninRequest {
  email: string;
  password: string;
}

Project Management APIs

// GET /api/projects
interface ProjectsResponse {
  projects: ChatbotProject[];
  totalCount: number;
}

// POST /api/projects
interface CreateProjectRequest {
  name: string;
  description?: string;
  systemPrompt?: string;
  n8nChatUrl?: string;
  n8nWebhookUrl?: string;
}

// PUT /api/projects/[id]
interface UpdateProjectRequest {
  name?: string;
  description?: string;
  systemPrompt?: string;
  n8nChatUrl?: string;
  n8nWebhookUrl?: string;
}

// DELETE /api/projects/[id]
// Returns: { success: boolean, message: string }

Document Management APIs

// POST /api/projects/[id]/documents
interface UploadDocumentRequest {
  file: File;
  filename: string;
}

interface UploadDocumentResponse {
  documentId: string;
  uploadUrl: string;
  processingJobId: string;
}

// GET /api/projects/[id]/documents
interface DocumentsResponse {
  documents: Document[];
  totalCount: number;
}

// DELETE /api/documents/[id]
// Returns: { success: boolean, message: string }

// POST /api/documents/[id]/process
interface ProcessDocumentRequest {
  forceReprocess?: boolean;
}

Chat APIs

// POST /api/projects/[id]/chat
interface ChatRequest {
  message: string;
  conversationId?: string;
  context?: Record<string, any>;
}

interface ChatResponse {
  response: string;
  conversationId: string;
  messageId: string;
  context?: Record<string, any>;
}

// GET /api/projects/[id]/conversations
interface ConversationsResponse {
  conversations: Conversation[];
  totalCount: number;
}

// WebSocket: /api/projects/[id]/ws
interface WebSocketMessage {
  type: 'message' | 'typing' | 'status';
  payload: any;
  conversationId: string;
}

Data Models

User Model

interface User {
  id: string;
  email: string;
  name?: string;
  image?: string;
  createdAt: Date;
  updatedAt: Date;
}

ChatbotProject Model

interface ChatbotProject {
  id: string;
  userId: string;
  name: string;
  description?: string;
  systemPrompt?: string;
  n8nChatUrl?: string;
  n8nWebhookUrl?: string;
  databaseName: string;
  minioBucketName: string;
  status: 'active' | 'inactive' | 'processing';
  createdAt: Date;
  updatedAt: Date;
}

Document Model

interface Document {
  id: string;
  chatbotProjectId: string;
  filename: string;
  originalFilename: string;
  fileType: string;
  fileSize: number;
  minioPath: string;
  processingStatus: 'pending' | 'processing' | 'completed' | 'failed';
  processingError?: string;
  uploadedAt: Date;
  processedAt?: Date;
}

Message Model (Per-chatbot DB)

interface Message {
  id: string;
  conversationId: string;
  role: 'user' | 'assistant' | 'system';
  content: string;
  metadata?: Record<string, any>;
  createdAt: Date;
}

Embedding Model (Per-chatbot DB)

interface Embedding {
  id: string;
  documentId: string;
  chunkId: string;
  content: string;
  metadata?: Record<string, any>;
  embedding: number[]; // Vector array
  createdAt: Date;
}

Configuration Management

Environment Variables

# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/chatbot_framework
REDIS_URL=redis://localhost:6379

# MinIO
MINIO_ENDPOINT=localhost
MINIO_PORT=9000
MINIO_ACCESS_KEY=minioaccess
MINIO_SECRET_KEY=miniosecret
MINIO_USE_SSL=false

# NextAuth
NEXTAUTH_URL=http://localhost:3000
NEXTAUTH_SECRET=your-secret-key

# n8n Integration
N8N_BASE_URL=http://localhost:5678
N8N_API_KEY=your-n8n-api-key

# File Upload
MAX_FILE_SIZE=50MB
ALLOWED_FILE_TYPES=pdf,md,docx,txt

# Rate Limiting
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_WINDOW=900000 # 15 minutes

Application Configuration

interface AppConfig {
  database: {
    maxConnections: number;
    connectionTimeout: number;
  };
  upload: {
    maxFileSize: number;
    allowedTypes: string[];
    uploadPath: string;
  };
  processing: {
    maxConcurrentJobs: number;
    jobTimeout: number;
    retryAttempts: number;
  };
  chat: {
    maxMessageLength: number;
    conversationTimeout: number;
    maxContextLength: number;
  };
}

Security Specifications

Authentication Security

  • JWT tokens with short expiration (15 min access, 7 day refresh)
  • Password hashing with bcrypt (12 rounds minimum)
  • Account lockout after failed attempts
  • Email verification for new accounts

API Security

  • Rate limiting per user/IP
  • Request validation with Zod schemas
  • CORS configuration for frontend only
  • API key authentication for chatbot endpoints

File Upload Security

  • File type validation (whitelist approach)
  • Virus scanning integration
  • Size limits per file and per user
  • Secure file storage with access controls

Database Security

  • Connection pooling with limited connections
  • Prepared statements for SQL injection prevention
  • Row-level security for multi-tenant data
  • Encryption at rest for sensitive data

Performance Specifications

Response Time Requirements

  • API responses: < 200ms (95th percentile)
  • File upload: < 30s for 10MB files
  • Document processing: < 5 minutes for 100-page PDF
  • Chat responses: < 3 seconds including RAG retrieval

Scalability Targets

  • Support 1000+ concurrent users
  • Handle 10,000+ documents per project
  • Process 100+ projects simultaneously
  • Store 1TB+ of documents and vectors

Caching Strategy

  • Redis for session data (TTL: 24 hours)
  • Query result caching (TTL: 5 minutes)
  • Static asset caching (TTL: 1 year)
  • Vector search result caching (TTL: 1 hour)

Error Handling

Error Categories

  1. Validation Errors: Input validation failures
  2. Authentication Errors: Login/permission failures
  3. Processing Errors: Document processing failures
  4. System Errors: Database/storage connectivity issues
  5. Rate Limit Errors: Too many requests

Error Response Format

interface ErrorResponse {
  error: {
    code: string;
    message: string;
    details?: Record<string, any>;
    timestamp: string;
  };
}

Monitoring & Alerting

  • Application performance monitoring (APM)
  • Error rate tracking and alerting
  • Database performance monitoring
  • Storage usage monitoring
  • User activity analytics

This technical specification provides the detailed foundation needed to implement the chatbot creation framework with all necessary components, security measures, and performance considerations.