PythonSeleniumMySQLDocker

Open Tender Scraper

A production-grade web scraper that extracts high volumes of open-tender data around the clock under strict connection quotas — engineered for reliability with connection pooling, atomic transactions, and adaptive backoff.

Overview

A procurement-intelligence platform is only as good as its data feed, but the source portals impose strict connection limits and change layout often, so naive scraping breaks or gets throttled.

Open Tender Scraper is the microservice that keeps that feed flowing: a supervised, constraint-aware scraper that runs 24/7 and lands validated records in a normalized relational schema. It is a backend-only microservice of EZBid, a procurement-intelligence platform.

What I Built

I built it in Python with Selenium for browser automation and MySQL for storage, packaged in Docker for repeatable deployment. The whole design centres on operating reliably inside tight resource constraints.

Constraint-aware connection pooling that maximises throughput while respecting the portal's strict connection quota.

Atomic transactions across a normalized multi-table schema, preventing partial or corrupt updates.

Automated pagination & multi-step navigation to traverse large result sets reliably.

24/7 supervised execution with intelligent backoff and process locking to prevent overlap.

Validation & deduplication with a custom exception hierarchy and context-managed resource cleanup.