Django Web Scraping: Run Scrapers Inside Your Web App

When scraping isn't a one-off script but part of a full web service — with a UI, a schedule, database storage, and background processing — Django comes into its own. The framework gives you an ORM for storage, management commands to launch jobs, and, together with Celery, background and periodic tasks.

This article is a practical follow-up to our overview piece on Python web scraping. The basics (requests, BeautifulSoup, encodings) are covered there; here the focus is Django web scraping — integrating collection into a Django project.

Table of contents

1. Why scrape inside Django
2. Fetching a page
3. Libraries for parsing content
4. Storing results in models
5. Running the scraper: management commands and Celery
6. Encodings in Django
7. Concurrency and background tasks
8. Proxies
9. Scraping through Tor
10. HTTPS/SSL
11. Working with cookies
12. Response status and headers
13. Storing URLs and queues via the ORM
14. Pros and cons

1. Why scrape inside Django

Django is the right call when scraping is only part of a product:

• a product/job/news aggregator with a web storefront;
• price monitoring with history in the database and a dashboard;
• a scheduled, regular catalog refresh;
• an admin panel to manage sources and review results.

Django gives you, out of the box: an ORM (storage and deduplication), an admin site (managing sources), a migrations system, and — with Celery — background and periodic execution so that heavy scraping doesn't block your web requests.

2. Fetching a page

The HTTP client in Django is no different from ordinary Python — reach for requests or httpx. Keep the scraping logic out of your views: views should be fast, so push network requests into a service layer or Celery tasks.

# parser/services.py
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (compatible; MyParserBot/1.0)",
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch_page(url: str) -> str | None:
    try:
        resp = requests.get(url, headers=HEADERS, timeout=15)
        resp.raise_for_status()
        resp.encoding = resp.apparent_encoding
        return resp.text
    except requests.RequestException as exc:
        # log it and don't crash the app
        import logging
        logging.getLogger("parser").warning("fetch failed %s: %s", url, exc)
        return None

The golden rule: never run a long scrape directly inside a view — the user will wait and the web server worker will be blocked. The request goes onto a queue and the response returns immediately.

3. Libraries for parsing content

Inside the service you use the same tools as anywhere else — BeautifulSoup for convenience, lxml for speed:

from bs4 import BeautifulSoup

def parse_products(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "lxml")
    items = []
    for card in soup.select(".product-card"):
        items.append({
            "title": card.select_one(".title").get_text(strip=True),
            "price": card.select_one(".price").get_text(strip=True),
            "url": card.select_one("a")["href"],
        })
    return items

A detailed walkthrough of the parsers lives in Python web scraping and lxml. If your source is tables, see scraping tables with BeautifulSoup. If an API returns JSON, see parsing JSON.

4. Storing results in models

Django's strength is the ORM. Describe a model, and deduplication, filtering, and update history become trivial.

# parser/models.py
from django.db import models

class Product(models.Model):
    source_url = models.URLField(unique=True)   # uniqueness = protection against dupes
    title = models.CharField(max_length=500)
    price = models.DecimalField(max_digits=10, decimal_places=2, null=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    def __str__(self):
        return self.title

Saving with dedup protection via update_or_create:

from .models import Product

def save_products(items: list[dict]):
    for item in items:
        Product.objects.update_or_create(
            source_url=item["url"],
            defaults={"title": item["title"], "price": item["price"]},
        )

update_or_create updates an existing record or creates a new one — perfect for re-running the scrape.

5. Running the scraper: management commands and Celery

A management command — for manual and cron runs

# parser/management/commands/run_parser.py
from django.core.management.base import BaseCommand
from parser.services import fetch_page, parse_products, save_products

class Command(BaseCommand):
    help = "Runs the catalog scraper"

    def add_arguments(self, parser):
        parser.add_argument("--url", required=True)

    def handle(self, *args, **options):
        html = fetch_page(options["url"])
        if html:
            items = parse_products(html)
            save_products(items)
            self.stdout.write(self.style.SUCCESS(f"Saved {len(items)} products"))

Run it with python manage.py run_parser --url https://example.com/catalog. A command like this is easy to hang off the system cron.

Celery — for background and periodic tasks

# parser/tasks.py
from celery import shared_task
from .services import fetch_page, parse_products, save_products

@shared_task(bind=True, max_retries=3, default_retry_delay=60)
def parse_url_task(self, url):
    html = fetch_page(url)
    if html is None:
        raise self.retry()       # retry on failure
    items = parse_products(html)
    save_products(items)
    return len(items)

Periodic runs are configured with django-celery-beat right in the admin — for example, refresh the catalog every 6 hours. Celery's retry gives you resilience against transient network failures without hand-rolled error handling.

6. Encodings in Django

The problem is the same as in any Python scraper: a mis-decoded response, showing up as mojibake (garbled characters). The fix is the same too — resp.apparent_encoding, or working with resp.content directly (covered in detail in the encodings section of Python web scraping).

Django-specific note: make sure the database uses UTF-8 (for PostgreSQL, the UTF8 encoding; for MySQL, utf8mb4) — otherwise non-ASCII text breaks at the save stage, not at scraping. In a new project this is the default, but with an older database it's worth checking.

7. Concurrency and background tasks

Django projects rarely use raw threads — for parallelism there's Celery with a pool of workers. Start several workers and you get parallel queue processing without hand-managing threads:

celery -A myproject worker --concurrency=8 --loglevel=info

If you need to quickly walk a list of URLs inside one task, a ThreadPoolExecutor works well. For very high concurrency inside a task you can use async — but in Django that requires care with the ORM (sync_to_async). That topic is covered in asynchronous web scraping in Python.

8. Proxies

Proxies plug in at the service layer — Django has nothing to do with it. It's convenient to store a proxy list in a model and flag the healthy ones:

class Proxy(models.Model):
    address = models.CharField(max_length=200)   # http://user:pass@ip:port
    is_active = models.BooleanField(default=True)
    fail_count = models.IntegerField(default=0)

The service then picks a random active proxy, and on error it increments fail_count, deactivating the proxy once a threshold is passed. The general rotation techniques are in our guide to proxies for scraping.

9. Scraping through Tor

Tor plugs in as a SOCKS5 proxy (socks5h://127.0.0.1:9050) in the same fetch_page. There's nothing Django-specific about it. On a Django server the Tor daemon runs as a separate process (a systemd service).

10. HTTPS/SSL

Certificate validation happens at the requests layer. In production keep verify=True. If your server has outdated root certificates, update certifi inside the project's virtual environment.

11. Working with cookies

For scraping behind a login use requests.Session inside the task. If you need to reuse a session across Celery tasks, persist the cookie jar (for example, serialized in Redis or in a model) and restore it before the request.

12. Response status and headers

Log the response code and decide whether to retry the task. In Celery this maps elegantly onto the retry mechanism:

@shared_task(bind=True, max_retries=5)
def fetch_task(self, url):
    resp = requests.get(url, timeout=15)
    if resp.status_code == 429:
        delay = int(resp.headers.get("Retry-After", 60))
        raise self.retry(countdown=delay)
    if resp.status_code in (403, 503):
        raise self.retry(countdown=120)   # probably a ban, wait longer
    resp.raise_for_status()
    return resp.text

This turns anti-bot blocks (429/403/503) into managed, delayed retries rather than lost data.

13. Storing URLs and queues via the ORM

In Django a frontier is conveniently implemented as a table with statuses — it survives a restart and is visible in the admin.

class CrawlURL(models.Model):
    STATUS = [("new", "new"), ("in_progress", "in_progress"),
              ("done", "done"), ("failed", "failed")]
    url = models.URLField(unique=True)     # unique = automatic dedup
    status = models.CharField(max_length=20, choices=STATUS, default="new")
    attempts = models.IntegerField(default=0)
    created_at = models.DateTimeField(auto_now_add=True)

A worker picks up "new" URLs, marks them in_progress, processes them, and moves them to done/failed. The unique index on url rules out duplicates. For high-load projects, keep the real queue in Redis / the Celery broker and store only results and history in the database.

14. Pros and cons of scraping in Django

Pros:

• The ORM takes care of storage, deduplication, and migrations.
• A built-in admin — manage sources and review data.
• Celery + beat — reliable background and periodic tasks with retries.
• One codebase: scraper and web storefront in a single project.

Cons:

• Overkill for one-off scripts — a plain .py file is simpler.
• Requires infrastructure: a broker (Redis/RabbitMQ) and Celery workers.
• Async in Django still comes with caveats (the ORM is largely synchronous).
• For crawling millions of pages, the specialized Scrapy is more efficient.

Let us run the pipeline for you

Wiring up Django, Celery, proxies, and monitoring — and keeping it all alive as target sites change — is ongoing work. If you'd rather own the product and skip the plumbing, scraping.pro can deliver the data straight into your Django models as a managed data extraction service or data as a service, so your app just reads from the tables.