Architecture Case Study

Trading Card Manager

A massive-scale data processing ecosystem managing 24M+ trading cards. It integrates a distributed database architecture with high-speed scraping to automate inventory management and eBay listings.

#Django#Flask#JavaScript#Celery#MySQL (Galera)
trading-card-manager.viewport
Trading Card Manager Software Architecture & UI System
Explore Architecture

Executive Overview

Managing dynamic collectible inventories at multi-million scale introduces extreme database engineering challenges. When dealing with over 24 million individual trading card items—each undergoing real-time market price fluctuation, grading classification, and multi-channel marketplace distribution—conventional relational database architectures fail under lock contention and search bottlenecks.

Trading Card Manager solves high-volume e-commerce throughput by pairing a 3-node Galera Multi-Master MySQL replication cluster with ultra-fast Typesense in-memory indexing. This hybrid architecture guarantees ACID-compliant transactional consistency for inventory allocation while delivering sub-millisecond fuzzy search performance across tens of millions of records.

By utilizing ProxySQL as an intelligent load balancing database routing tier, database write transactions are dynamically segregated from high-frequency search and scrape ingestion reads, preventing lock degradation during peak scraping workflows.

High-Throughput Scraping & Processing Architecture

To keep pace with volatile global resale markets (eBay and specialized auctions), the backend relies on a distributed high-speed ingestion architecture:

  • Asynchronous Scraper Engine (Flask & Celery): Orchestrates thousands of worker tasks to continuously collect pricing trends, auctions, and historical valuation metadata across external marketplaces.
  • Hybrid Microservices Ecosystem: Integrates lightweight Python (Flask/Celery) processing workers with solid enterprise web frameworks to separate background ETL (Extract, Transform, Load) pipelines from interactive frontend dashboard routing.
  • Containerized Infrastructure (Docker): The entire ecosystem—spanning application microservices, ProxySQL nodes, Galera clusters, and Typesense indexing servers—is orchestrated within Docker containers for zero-downtime scalability and plug-and-play DevOps portability.

Strategic Value & Operational ROI

Beyond database performance, the system automates end-to-end e-commerce logistics. It matches newly ingested warehouse card batches against historical valuation indexes, formats listing specifications, and auto-exports synchronized CSV feeds directly to eBay. Internal operations tools automate employee grading accuracy tracking and performance calculation, transforming a complex manual cataloging operation into an enterprise automated data factory.

Technical Innovation

Key Architectural Capabilities

Scraper Engine :

Ingests massive card datasets in real-time via Flask & Celery.

Instant Search :

Real-time, fuzzy search over 24M+ synced records via Typesense.

Listing Workflow :

Automates card matching, approval, and CSV export for eBay.

Employee Ops :

Tracks processing accuracy and automatically calculates performance bonuses.

Execution & Design

The Engineering Approach

01.

Designed the high-speed scraping engine and inter-service communication protocols.

02.

Implemented the ACID-compliant internal tool for card processing and inventory management.

03.

Managed the seamless synchronization between MySQL primary storage and Typesense indices.

Measurable Outcome

Quantifiable Impact & Results

Managed 24M+ records using Galera 3-node multi-master replication with ProxySQL.
Achieved millisecond-level search capability by syncing massive datasets with Typesense.
Engineered a hybrid micro-service architecture (Laravel + Flask) for modular scalability.
Dockerized the entire multi-service ecosystem for plug-and-play deployment.