Computer VisionFrame Latency: 30–50 ms/frameLive App: Hosted on Render

AI Visual Memory & 3D Spatial Tracking System

Persistent visual memory, object detection, appearance Re-ID, FAISS indexing, and semantic natural language search.

Verified Milestone:30–50 ms/frame inference · Real-time ByteTrack + CLIP Re-ID · Live on Render
Launch Live ApplicationView Repository
Technologies:OpenCVYOLOv11ByteTrackCLIPFAISSFastAPIReactDockerWebSocketsSQLite
AI Visual Memory System
AI Visual Memory System System Interface

Problem Statement

Traditional surveillance and camera systems record gigabytes of raw pixels without semantic understanding, making retrospective event search tedious and manual.

Engineering Goals

  • Achieve real-time processing (30-50 ms/frame) for multi-object tracking.
  • Re-identify objects across camera view occlusions using visual appearance embeddings.
  • Enable natural language semantic search over tracked object memories.
  • Provide a live interactive React monitoring dashboard over WebSockets.

Execution Pipeline & Mechanics

01

Frame Ingestion & Detection

FastAPI stream receiver feeds frames into YOLOv11 for bounding box proposal.

02

Trajectory Tracking

ByteTrack associates bounding boxes across frames to maintain unique object IDs.

03

Appearance Embedding

Cropped object thumbnails are encoded via CLIP visual encoder.

04

Memory Bank Indexing

Embeddings are stored in FAISS alongside timestamps, camera IDs, and spatial coordinates.

05

Semantic Query Retrieval

Natural language queries are embedded and matched against FAISS vectors to highlight historical locations.

Engineering Challenges & Design Decisions

Decision: ByteTrack over DeepSORT

Rationale: ByteTrack utilizes low-score detection boxes to maintain tracking during partial occlusions with significantly lower computational overhead.

Alternative Rejected: DeepSORT with heavy appearance descriptor networks.

Core Architectural Highlights

Video streams are processed frame-by-frame by OpenCV and YOLOv11. Detected bounding boxes are tracked via ByteTrack. Crop thumbnails are embedded by CLIP into a FAISS index. FastAPI coordinates WebSocket feeds to the React frontend.

•Reported 30–50 ms per frame processing performance.
•Integrated homography matrix calculations for 2D plane spatial localization.
•Deployed public live web application to Render.

Testing & Empirical Evaluation

Benchmarked across diverse synthetic and real indoor camera feeds with varied lighting.

Result #1
30–50 ms average processing latency per frame
Result #2
Continuous tracking through short occlusions up to 30 frames

Security & Reliability Defenses

Local thumbnail storage with automated retention and deletion policies.

Key Engineering Learnings

  • •Filtering low-confidence detection noise before embedding generation prevents polluting the FAISS memory bank with false positives.