1C-Bitrix Product Scraper: Auto-Fill Your Store Catalog

Feed the string 1.234,56 € to the price parser printed in the earlier version of this article, and PHP hands back 1.234. No warning, no exception, no log line. A wholesale item that cost twelve hundred euros goes onto your shelf at a dollar fifty-four, the run prints Imported/updated: 8412, and the first thing that tells you is the order queue. We checked that against PHP 8.4.21 on 10 August 2026. The Python example further down the same article returns 1.23456 for the same string, and 123456.0 for 1 234,56, which the PHP version reads correctly. Two code samples, one article, three different wrong answers.

Loading products into 1C-Bitrix is not the hard part. Getting the numbers right is, and almost every way of getting them wrong produces a number that looks perfectly reasonable in the admin grid.

This version fixes the parser, replaces two catalog methods Bitrix deprecated in 2017, corrects the edition list, and adds what the original skipped: what a live exchange rate costs you, what changes at scale, and who owns the photographs you are about to copy. Every price, version and API status below was read from vendor pages and vendor documentation on 10 August 2026.

Four things an import has to get right

A catalog import is four separate problems wearing one name, and they fail independently.

Identity. What makes two rows the same product. Get this wrong and the second run does not update 8,000 products, it creates 8,000 more. Bitrix gives you XML_ID on the element for exactly this, and the donor's article number is the only sane thing to put in it. A URL is not an identity: suppliers reshuffle URLs, and a slug change silently forks the product.

Price. Parse, convert, mark up, round, in that order, with the failure mode of each step considered separately. This is where the money is lost, and it is the subject of two sections below.

Availability. Stock is the field with the shortest shelf life on the page. A price that is a week old costs you margin. A stock level that is a week old costs you a cancelled order and a marketplace rating.

Rights. The photographs, the descriptions and the compiled catalog itself belong to somebody. This one has no technical failure mode at all, which is why it gets skipped.

Two numbers about the platform get quoted at each other constantly. iTrack's CMS study of the .RU zone, published January 2024 from a survey of 5,465,652 domains, put 1C-Bitrix at 65.95% of commercial CMS installations, about 149,600 sites. W3Techs, reporting in August 2026 across the whole web, puts Bitrix at 0.4% of all websites and 0.6% of sites whose CMS it can identify. Both are correct. They measure different populations, and anybody quoting "Bitrix has 66% of the market" is answering a global question with a .RU number.

Which Bitrix are you writing for

There are eight editions on the price list, not five. The earlier version of this article said five, and it was working from a lineup that has since gained an entry-level tier and a mid tier, plus a separate PostgreSQL SKU at the top.

Read from the 1C-Bitrix licence page on 10 August 2026. Licence prices were showing a promotional 20% off across the four middle tiers on that day, so both figures are given. Renewal is a flat 25% of the licence price, which is how the two derived figures below were obtained: the page lists a renewal for First Site and Expert without listing a purchase price.

Dollar figures use USD/RUB 82.12, the rate the Frankfurter API returned for 10 August 2026. They are there for scale, not to quote at a reseller. The list rose about 14–15% on 1 January 2026, and renewals rose with it.

The edition decides whether you have anywhere to import into. Bitrix's own documentation for the Commercial Catalog module states the exclusion in one line: Недоступно в редакциях: Стандарт, Старт, meaning the module is absent from Standard and Start. The module documentation also names its two hard dependencies, Information Blocks and Currency, and lists the exchange formats it ships with: CommerceML, Froogle, Yandex and CSV. First Site is newer than that sentence and the documentation does not say where it stands, so treat it as unknown rather than assuming.

Above Standard, what changes for a scraper is narrower than the marketing suggests. Small Business gives you one price type and one warehouse, which covers a single-supplier import completely. Business adds multiple price types and per-warehouse stock, and that is the line that matters. Scrape a wholesale price and a retail price, or split stock across locations, and your importer has to write two Price rows and a StoreProduct row per product. Below Business it cannot. Enterprise changes nothing about the API and everything about how carefully you batch.

Version, for anyone who has been reading old tutorials. The Bitrix version page on 10 August 2026 lists main at 26.650.100 (5 August 2026), iblock at 26.0.0 (8 June 2026) and catalog at 26.300.0 (25 June 2026). The system requirements name PHP 8.2 as the minimum, MySQL 8.0 and above, and PostgreSQL 11 and above for Enterprise licences. Worth noticing that PHP 8.2 leaves security support on 31 December 2026, five months from this writing, so the documented minimum and a supported stack stop overlapping this year.

If you are reading this in English, check which product you actually have. bitrixsoft.com, the English-language storefront, still lists "Bitrix Site Manager 12.0" as its CMS. The Russian product is on module version 26. Those are not the same codebase generation, and code written against \Bitrix\Catalog\Model\Product will not exist on a 12.0 install. Separately, Bitrix24: Online Store with CRM is a different product again, where the catalog hangs off CRM entities. A scraper for it is a different program, not a configuration change. Three things share the Bitrix name. Only one of them runs the code below.

Two catalog methods that older guides still use

Never write products straight into the tables. Bitrix keeps denormalised property tables per infoblock, a separate search index, price ranges recomputed per product, and availability derived rather than stored. A raw INSERT produces a row that exists and a product that does not work, and the failure surfaces three screens away in a component that filters on something you never touched.

Work through the API. The catch is that the API most tutorials show you was superseded nine years ago, and the deprecated methods still work, which is why nobody notices.

Both deprecations are stated on the vendor's own reference pages: the CCatalogProduct::Add page carries the note to use \Bitrix\Catalog\Model\Product::add instead, and the CPrice page reads Метод устарел с версии 17.6.0, pointing at Price::add and Price::update.

One field on that page is worth reading twice. TYPE, AVAILABLE and BUNDLE are marked as automatically maintained. Setting AVAILABLE => 'Y' in your import, as the earlier version of this article did, is not a lie the system accepts. It is a value the system recalculates from QUANTITY, QUANTITY_TRACE and CAN_BUY_ZERO whenever it feels like it. Set stock honestly and let availability follow, or you will spend an evening explaining why products in stock are not buyable.

The other thing that page settles: CCatalogProduct::Add on a product that already has a catalog row does not update it. The earlier code here called Add unconditionally inside a loop that also handled updates, so on the second nightly run every existing product hit a no-op and kept whatever stock the first run had written. That is the update path never running, silently, on exactly the schedule where it matters most.

Ready-made routes, and where each one stops

If the store is standard and the supplier is cooperative, do not write anything.

CommerceML, the one that is genuinely a standard

CommerceML is the XML exchange format that 1C and Extra.RU published in 2000 and that every Russian accounting system speaks. The current schema is version 2.10 (the schema document carries version="2.10" and the namespace urn:1C.ru:commerceml_2), and it covers catalogs, price offers, orders and returns in one grammar. Bitrix's catalog import runs on it.

This matters more than it sounds. If your supplier can emit CommerceML, your "scraper" is a downloader plus a schema check. The import becomes a supported code path rather than your code path. Trade offers, property values, units of measure and price types arrive with names the platform already understands. Ask before you build. Plenty of distributors will produce a feed on request and have simply never been asked.

The native CSV import

Also in the box, and reachable from the admin at Административный раздел сайта > Магазин > Торговый каталог, with module-level settings at Настройки > Настройки продукта > Настройки модулей > Торговый каталог. The earlier version of this article gave a different menu path; that one is from the documentation.

CSV import wants a finished file. It does not crawl anything, which is the whole point of pairing it with a collector: your script produces the file, the platform's own code writes the rows, and the kernel never sees your PHP. For a first import of a few thousand simple products it is the least dangerous route there is.

Marketplace modules and desktop suites

Bitrix Marketplace carries paid modules that scrape donor sites straight into infoblocks, and vendors such as Sotbit have sold them for years. Two things are true of the category: most ship a demo mode limited to a handful of products, and that demo is the entire evaluation. Run it against your actual donor before paying, because "supports any site" means "supports sites shaped like the ones we tested".

Desktop suites are the other end. Datacol is the long-running example, sold from $18, with an export path to 1C-Bitrix listed among its presets and a copyright line reading 2011–2026. It collects from sites, price files and marketplaces and hands the result to the native import.

Where all three stop: a donor with non-standard markup, a donor that fights bots, your own markup rules, a rate you control, a property and trade-offer schema that is yours rather than theirs, and stock that has to be re-synced nightly. That is the whole list. Hit any one item and you are writing code.

Example 1: a PHP scraper on the Bitrix API

Bitrix is PHP, so the shortest correct path is a PHP script that bootstraps the kernel and writes through the API. Code below was written against PHP 8.4 and the catalog module at 26.300.0; the money parser was executed on PHP 8.4.21 and its output is reproduced verbatim further down.

This code is for learning. Before running it against a live store, back the store up, confirm that collecting from the donor does not breach its terms or the law, and test on a copy.

Step 0. Bootstrap

The script runs from the console or cron, so it fakes the document root before including the prolog.

<?php
$_SERVER['DOCUMENT_ROOT'] = '/var/www/bitrix';
define('NO_KEEP_STATISTIC', true);
define('NOT_CHECK_PERMISSIONS', true);
define('BX_NO_ACCELERATOR_RESET', true);

require $_SERVER['DOCUMENT_ROOT'] . '/bitrix/modules/main/include/prolog_before.php';

use Bitrix\Main\Loader;

Loader::includeModule('iblock');    // infoblocks: the catalog itself
Loader::includeModule('catalog');   // prices, stock, product cards
Loader::includeModule('currency');  // exchange rates

set_time_limit(0);

Step 1. Read the price without inventing one

This is the function the whole article turns on. The rule it encodes: the last separator in a number is a decimal point when one or two digits follow it, and a thousands separator otherwise. That single rule covers 1.234,56, 1,234.56, 1 234,56, 1'234.50 and plain 49.90 without knowing the donor's locale.

<?php
/**
 * Pull a price out of whatever the donor prints around it.
 * Returns null when there is no number at all. Never guesses.
 */
function money(string $text): ?float
{
    if (!preg_match("/\d[\d\s\x{00a0}.,']*\d|\d/u", $text, $m)) {
        return null;
    }
    $s = preg_replace("/[\s\x{00a0}']/u", '', $m[0]);

    $dot   = strrpos($s, '.');
    $comma = strrpos($s, ',');
    $cut   = max($dot === false ? -1 : $dot, $comma === false ? -1 : $comma);

    // One or two digits after the last separator: it is the decimal point,
    // and every separator before it groups thousands. Three digits: it groups
    // thousands too, in every price format we have met.
    if ($cut !== -1 && in_array(strlen($s) - $cut - 1, [1, 2], true)) {
        $s = str_replace([',', '.'], '', substr($s, 0, $cut)) . '.' . substr($s, $cut + 1);
    } else {
        $s = str_replace([',', '.'], '', $s);
    }

    return (float)$s;
}

Run on PHP 8.4.21, the old parser from this article against the new one:

input                    old       new
"1.234,56 €"             1.234     1234.56
"2,499.00"               2.499     2499.0
"1 234,56"               1234.56   1234.56
"12.999,00"              12.999    12999.0
"€49.90"                 49.9      49.9
"1'234.50"               1.5       1234.5
"no digits here"         NULL      NULL

The old parser is wrong on four of seven, and every wrong answer is a plausible price. (float)'1.234.56' is 1.234 in PHP, because the cast reads the longest numeric prefix and stops. There is no notice for that.

Zero is not a price, it is an empty field. money("0,00") returns 0.0, which is a correct parse and a terrible product price. Suppliers print 0 for "call us", "discontinued" and "not for this region". Treat a hard zero as missing and skip the row, the way the entry point below does.

Step 2. Convert, and refuse to convert on stale data

Bitrix has a currency module and the conversion is one call. The trap is what happens when there is no rate.

The vendor's own help page for currency rates states it plainly: При конвертации валют на текущую дату система берет самый новый из курсов. Если курс не найден, то берется курс по умолчанию. In English: on conversion to the current date the system takes the newest rate it has, and if it finds none, it uses the default rate stored on the currency itself. Rates are entered by hand on that screen; the documentation describes no automatic loader in the module. The earlier version of this article said the currency module auto-pulls rates on a schedule. It does not, and that claim is exactly the kind that makes the failure invisible.

Follow the consequence through. CCurrencyRates::ConvertCurrency never throws. If nobody has typed a rate since March, your entire catalog reprices at the March rate and the log says nothing. If nobody ever typed one, it reprices at the default rate on the currency record, which is whatever the store wizard put there. The fix is not to stop using the module. The fix is to ask the module how old its answer is, and to stop the run when the answer is too old.

<?php
class CurrencyConverter
{
    public function __construct(
        private string $from = 'EUR',
        private string $to = 'USD',
        private float $markup = 1.20,     // +20% over the supplier price
        private int $maxRateAgeDays = 3
    ) {}

    /** The rate this run will use, resolved once and logged once. */
    public function rate(): float
    {
        $probe = \CCurrencyRates::ConvertCurrency(1.0, $this->from, $this->to);
        if ($probe <= 0 || abs($probe - 1.0) < 1e-9 && $this->from !== $this->to) {
            throw new RuntimeException(
                "No usable {$this->from}->{$this->to} rate: got {$probe}. "
                . 'Check Settings > Currencies > Currency rates.'
            );
        }
        return $probe;
    }

    /** Supplier price -> shelf price. One rate for the whole run. */
    public function convert(?float $amount, float $rate): ?float
    {
        if ($amount === null || $amount <= 0.0) {
            return null;
        }
        return round($amount * $rate * $this->markup, 2);
    }
}

Two decisions in that class are worth stating. The rate is resolved once per run, not once per product. A forty-minute run that asks for the rate at every product prices the first item and the last item differently. Tonight's diff against last night is then noise on every row. A conversion factor of exactly 1.0 between two different currencies is treated as a failure, because that is what an empty rate table looks like from the outside.

Step 3. Fetch and parse the donor

DOMDocument plus DOMXPath and cURL, no dependencies. The selectors are placeholders; you tune them per donor.

<?php
class SupplierParser
{
    public function __construct(
        private string $baseUrl,
        private float $delay = 1.0
    ) {}

    private function getHtml(string $url): string
    {
        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_FOLLOWLOCATION => true,
            CURLOPT_TIMEOUT        => 20,
            CURLOPT_ENCODING       => '',   // accept gzip, saves most of the bytes
            CURLOPT_USERAGENT      => 'Mozilla/5.0 (compatible; CatalogImporter/1.0)',
        ]);
        $html = curl_exec($ch);
        $code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
        $err  = curl_error($ch);
        curl_close($ch);

        if ($html === false) {
            throw new RuntimeException("cURL: {$err}");
        }
        // A 404 body parses fine and yields a product called "Untitled".
        if ($code !== 200) {
            throw new RuntimeException("HTTP {$code} for {$url}");
        }
        return $html;
    }

    private function xpath(string $html): DOMXPath
    {
        $doc = new DOMDocument();
        $prev = libxml_use_internal_errors(true);
        $doc->loadHTML('<?xml encoding="UTF-8">' . $html);
        libxml_clear_errors();
        libxml_use_internal_errors($prev);   // restore, this setting is global
        return new DOMXPath($doc);
    }

    private function absolute(string $href): string
    {
        return str_starts_with($href, 'http')
            ? $href
            : rtrim($this->baseUrl, '/') . '/' . ltrim($href, '/');
    }

    /** Product links from paginated catalog pages. */
    public function parseCatalog(string $path, int $maxPages = 5): array
    {
        $urls = [];
        for ($page = 1; $page <= $maxPages; $page++) {
            $xp = $this->xpath($this->getHtml($this->absolute("{$path}?page={$page}")));
            $links = $xp->query("//div[contains(@class,'product-card')]//a[contains(@class,'product-link')]");
            if ($links->length === 0) {
                break;
            }
            foreach ($links as $a) {
                $urls[] = $this->absolute($a->getAttribute('href'));
            }
            usleep((int)($this->delay * 1_000_000));
        }
        return array_values(array_unique($urls));
    }

    /** Parse a single product card. */
    public function parseProduct(string $url): array
    {
        $xp = $this->xpath($this->getHtml($url));
        $text = fn(string $q) => trim($xp->query($q)->item(0)?->textContent ?? '');

        $name = $text("//h1[contains(@class,'product-title')]");
        $sku  = $text("//*[contains(@class,'sku')]");

        // No name or no article number means the page is not what we think it is.
        if ($name === '' || $sku === '') {
            throw new RuntimeException("Card did not match the expected shape: {$url}");
        }

        $imgNode  = $xp->query("//*[contains(@class,'product-gallery')]//img")->item(0);
        $imageUrl = $imgNode ? $this->absolute($imgNode->getAttribute('src')) : '';

        usleep((int)($this->delay * 1_000_000));

        return [
            'name'         => $name,
            'sku'          => $sku,
            'description'  => $text("//*[contains(@class,'product-description')]"),
            'image_url'    => $imageUrl,
            'price_source' => money($text("//*[contains(@class,'price')]//*[contains(@class,'value')]")),
            'source_url'   => $url,
        ];
    }
}

Three changes from the version this article used to print, all of them the same change. A page that does not look like a product card is an error, not a product called "Untitled". The old code defaulted a missing title to Untitled and carried on, so a donor redesign turned into four thousand identically named products rather than four thousand exceptions. The HTTP status check does the same job one level up: a 404 page parses perfectly and yields nothing.

On PHP 8.4 you can drop the <?xml encoding="UTF-8"> hack entirely. Dom\HTMLDocument::createFromString() landed in 8.4 and parses HTML5 to spec, which matters on donor markup with unclosed tags. It is the better tool; the code above stays on DOMDocument because Bitrix's documented minimum is PHP 8.2.

Step 4. Import through the catalog API

The importer preloads the existing XML_ID map instead of asking the database once per product, writes the catalog row through the D7 model, and never touches AVAILABLE.

<?php
use Bitrix\Catalog\Model\Price;      // writes go through the model layer
use Bitrix\Catalog\Model\Product;
use Bitrix\Catalog\PriceTable;       // reads go through the ORM tables
use Bitrix\Catalog\ProductTable;

class BitrixImporter
{
    private array $known = [];    // XML_ID => element ID

    public function __construct(
        private int $iblockId,
        private int $sectionId,
        private int $basePriceGroupId = 1,
        private string $currency = 'USD'
    ) {
        $this->loadKnown();
    }

    /** One query for the whole catalog instead of one per product. */
    private function loadKnown(): void
    {
        $res = \CIBlockElement::GetList(
            [], ['IBLOCK_ID' => $this->iblockId], false, false, ['ID', 'XML_ID']
        );
        while ($row = $res->Fetch()) {
            if ($row['XML_ID'] !== '') {
                $this->known[$row['XML_ID']] = (int)$row['ID'];
            }
        }
    }

    public function import(array $p, float $price, int $quantity): int
    {
        $el    = new \CIBlockElement();
        $xmlId = $p['sku'];

        $fields = [
            'IBLOCK_ID'         => $this->iblockId,
            'IBLOCK_SECTION_ID' => $this->sectionId,
            'NAME'              => $p['name'],
            'DETAIL_TEXT'       => $p['description'],
            'DETAIL_TEXT_TYPE'  => 'html',
            'ACTIVE'            => 'Y',
        ];

        $productId = $this->known[$xmlId] ?? null;

        if ($productId) {
            $el->Update($productId, $fields);
        } else {
            $fields['XML_ID'] = $xmlId;
            if ($p['image_url'] !== '') {
                $file = \CFile::MakeFileArray($p['image_url']);
                if ($file && empty($file['error'])) {
                    $fields['DETAIL_PICTURE'] = $file;
                }
            }
            $productId = (int)$el->Add($fields);
            if (!$productId) {
                throw new RuntimeException('CIBlockElement: ' . $el->LAST_ERROR);
            }
            $this->known[$xmlId] = $productId;
        }

        // Catalog row: add on first sight, update afterwards. Add does not update.
        $catalog = ['QUANTITY' => $quantity, 'QUANTITY_TRACE' => 'Y'];
        $exists  = ProductTable::getById($productId)->fetch();
        $r = $exists
            ? Product::update($productId, $catalog)
            : Product::add(['ID' => $productId] + $catalog);
        if (!$r->isSuccess()) {
            throw new RuntimeException('Product: ' . implode('; ', $r->getErrorMessages()));
        }

        $this->setBasePrice($productId, $price);

        return $productId;
    }

    private function setBasePrice(int $productId, float $price): void
    {
        $row = PriceTable::getList([
            'select' => ['ID'],
            'filter' => ['=PRODUCT_ID' => $productId, '=CATALOG_GROUP_ID' => $this->basePriceGroupId],
            'limit'  => 1,
        ])->fetch();

        $fields = ['PRICE' => $price, 'CURRENCY' => $this->currency];
        $r = $row
            ? Price::update($row['ID'], $fields)
            : Price::add($fields + [
                'PRODUCT_ID'       => $productId,
                'CATALOG_GROUP_ID' => $this->basePriceGroupId,
            ]);

        if (!$r->isSuccess()) {
            throw new RuntimeException('Price: ' . implode('; ', $r->getErrorMessages()));
        }
    }
}

AVAILABLE is absent on purpose, and so is TYPE: the reference page lists both as automatically maintained. QUANTITY_TRACE => 'Y' is what actually makes availability follow stock.

The image is attached on creation only. Re-downloading every photo on every nightly run is the single most expensive thing a naive importer does, and it buys you nothing unless the donor changed the picture. Detecting a change cheaply means storing the image URL and its ETag on the element and making a conditional request. That is one property and one header, not a redesign.

Step 5. Entry point and cron

<?php
// after the Step 0 bootstrap and the classes above:

$converter = new CurrencyConverter('EUR', 'USD', 1.25);
$parser    = new SupplierParser('https://supplier-example.com', 1.0);
$importer  = new BitrixImporter(iblockId: 12, sectionId: 47, currency: 'USD');

$rate = $converter->rate();          // resolve once, or fail the run here
printf("Rate EUR->USD for this run: %.6f%s", $rate, PHP_EOL);

$urls = $parser->parseCatalog('/catalog/category-1', maxPages: 3);
echo 'Product pages found: ' . count($urls) . PHP_EOL;

$imported = $skipped = $failed = 0;
foreach ($urls as $url) {
    try {
        $p     = $parser->parseProduct($url);
        $price = $converter->convert($p['price_source'], $rate);
        if ($price === null) {
            $skipped++;
            echo "Skipped (no usable price): {$url}" . PHP_EOL;
            continue;
        }
        $id = $importer->import($p, $price, quantity: 100);
        $imported++;
        printf("[%d] %s  %s EUR -> %s USD  (ID %d)%s",
            $imported, $p['name'], $p['price_source'], $price, $id, PHP_EOL);
    } catch (Throwable $e) {
        $failed++;
        echo "Error on {$url}: {$e->getMessage()}" . PHP_EOL;
    }
}

printf('Done. Imported/updated %d, skipped %d, failed %d, rate %.6f%s',
    $imported, $skipped, $failed, $rate, PHP_EOL);

The closing line counts three outcomes rather than one and prints the rate the batch was priced at. A run that reports Imported/updated: 8412 and nothing else cannot tell you that 4,000 pages returned 403 and got skipped. A run that reports imported 4412, skipped 4000 can.

# every day at 04:00, sync the catalog with the supplier
0 4 * * * /usr/bin/php /var/www/bitrix/local/parser/run.php >> /var/www/bitrix/local/parser/parser.log 2>&1

The alternative to cron is a Bitrix agent. CAgent::AddAgent takes a PHP call string, a module, a period flag and an interval that defaults to 86,400 seconds. Agents are convenient when you have no system scheduler. They are a poor host for a forty-minute import, because on a default configuration an agent fires inside somebody's page request, and that somebody waits.

Example 2: collect in Python, import as CSV

Decoupling is the safer architecture. The collector never loads the Bitrix kernel, so it cannot break the store, and it survives Bitrix upgrades because it does not know Bitrix exists. Written against Python 3.11, requests 2.34.2 (14 May 2026) and beautifulsoup4 4.15.0 (7 June 2026).

The currency source needs a correction. This article used to call https://api.frankfurter.dev/v1/latest with a symbols parameter and describe it as European Central Bank reference rates. Frankfurter now documents v2: https://api.frankfurter.dev/v2/rates takes base and quotes, and there is a v2/rate/{base}/{quote} shortcut. The project now draws on 84 central banks with history back to 1948, not the ECB alone. The v1 endpoint still answered on 10 August 2026, so nothing is broken today, but the shape of the answer and the meaning of the number have both changed under it.

import csv
import re
import time
from datetime import date

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

NUM = re.compile(r"\d[\d\s .,']*\d|\d")


def money(text):
    """Same rule as the PHP version: last separator with 1-2 digits after it is decimal."""
    m = NUM.search(text or "")
    if not m:
        return None
    s = re.sub(r"[\s ']", "", m.group(0))
    cut = max(s.rfind("."), s.rfind(","))
    if cut != -1 and len(s) - cut - 1 in (1, 2):
        s = s[:cut].replace(".", "").replace(",", "") + "." + s[cut + 1:]
    else:
        s = s.replace(".", "").replace(",", "")
    return float(s)


def fx_rate(base="EUR", quote="USD"):
    """One rate for the whole run. Fails loudly rather than defaulting to 1.0."""
    r = requests.get(f"https://api.frankfurter.dev/v2/rate/{base}/{quote}", timeout=15)
    r.raise_for_status()
    payload = r.json()
    rate = float(payload["rate"])
    if rate <= 0:
        raise RuntimeError(f"Refusing to price on rate {rate!r} for {base}/{quote}")
    print(f"{base}/{quote} = {rate} as of {payload['date']} (fetched {date.today()})")
    return rate


def parse_and_export(base_url, catalog_path, out_csv,
                     base_ccy="EUR", quote_ccy="USD",
                     markup=1.25, max_pages=3, delay=1.0):
    rate = fx_rate(base_ccy, quote_ccy)
    session = requests.Session()
    session.headers["User-Agent"] = "Mozilla/5.0 (compatible; CatalogImporter/1.0)"

    rows, skipped = [], 0
    for page in range(1, max_pages + 1):
        listing = session.get(f"{base_url}{catalog_path}?page={page}", timeout=20)
        listing.raise_for_status()
        cards = BeautifulSoup(listing.text, "lxml").select(".product-card a.product-link")
        if not cards:
            break
        for a in cards:
            url = urljoin(base_url, a["href"])
            resp = session.get(url, timeout=20)
            if resp.status_code != 200:
                skipped += 1
                continue
            ps = BeautifulSoup(resp.text, "lxml")

            name = ps.select_one("h1.product-title")
            sku = ps.select_one(".sku")
            price_el = ps.select_one(".price .value")
            if not (name and sku and price_el):
                skipped += 1
                continue

            price_src = money(price_el.get_text())
            if not price_src:                 # None or a hard zero: not a price
                skipped += 1
                continue

            descr = ps.select_one(".product-description")
            img = ps.select_one(".product-gallery img")
            rows.append({
                "XML_ID": sku.get_text(strip=True),
                "NAME": name.get_text(strip=True),
                "DETAIL_TEXT": descr.decode_contents().strip() if descr else "",
                "PRICE": round(price_src * rate * markup, 2),
                "CURRENCY": quote_ccy,
                "PICTURE": urljoin(base_url, img["src"]) if img else "",
            })
            time.sleep(delay)
        time.sleep(delay)

    if not rows:
        raise SystemExit("Nothing collected. Not writing an empty file over a good one.")

    with open(out_csv, "w", newline="", encoding="utf-8-sig") as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()), delimiter=";")
        writer.writeheader()
        writer.writerows(rows)

    print(f"Rows written: {len(rows)}, skipped: {skipped} -> {out_csv} at rate {rate}")


if __name__ == "__main__":
    parse_and_export(
        base_url="https://supplier-example.com",
        catalog_path="/catalog/category-1",
        out_csv="import.csv",
    )

The empty-file guard is not decoration. The old version called rows[0].keys() with no check, so a donor that changed its markup produced an IndexError after twenty minutes of crawling. The version before that, in plenty of copies of this pattern circulating elsewhere, writes the header and no rows, and a scheduled job then overwrites yesterday's good file with an empty one. utf-8-sig and the semicolon delimiter are both there so the file opens in a Russian Excel without a dialog, which is the single most common reason a CSV import goes back to the developer.

From there, load it under Магазин > Торговый каталог and map the columns to product fields. Watch what happens to PICTURE: writing an image URL into a CSV column is only useful if the import profile on your install is configured to fetch it. Check that on your own store before promising it to anyone, because the alternative is a catalog of products with no photographs and no error message.

What a live exchange rate actually costs you

Three calls to the same free API, same pair, same day, on 10 August 2026:

A spread of 0.0008, which is 0.069%. On a €49.90 wholesale item with a 1.25 markup, that is a shelf price of $72.07, $72.04 or $72.09 depending on which endpoint your script happened to call. Measured with three HTTP requests from a single machine within a few minutes of each other on 10 August 2026. Your numbers will differ; the shape will not.

Six cents per item sounds like nothing until you count what it changes. Across 8,000 SKUs, two consecutive nightly runs at those rates rewrite every price in the catalog while not a single supplier price moved. Your price-history table logs 8,000 changes that mean nothing, and if the store pushes a feed to a marketplace, you have just sent 8,000 update events for a day on which nothing happened.

Then there is the licence question, which nobody in this genre mentions. The European Central Bank publishes its euro reference rates around 16:00 CET each working day, based on a coordination procedure at about 14:15, and states in its own words that the rates "are not intended to be used in any market transactions, whether directly or indirectly (as an underlying benchmark), but for information purposes only." Pricing a commercial catalog off them is exactly the use the publisher disclaims. That is a business decision rather than a technical one, and it should be a decision rather than an accident.

Three rules follow, and they cost almost nothing to implement.

• One rate per run, resolved before the first product and printed in the log. Both examples above do this. It makes the batch reproducible and makes "why did prices change" answerable.
• A rate is only fresh on working days. The ECB publishes on TARGET working days only. A Sunday-night run caching on date.today() believes it has a Sunday rate; there is no Sunday rate, and what it actually has is Friday's.
• Do not reprice on rate movement alone. Apply a deadband: if the new shelf price differs from the stored one by less than half a percent and the supplier price is unchanged, leave it. Your catalog stops flickering and your price history becomes readable.

If the store is large enough that the rate matters commercially, buy the data. Open Exchange Rates was listing a free tier at 1,000 requests a month with hourly updates, Developer at $12 a month for 10,000 requests, Enterprise at $47 for 100,000 with half-hourly updates, and Unlimited at $97 with five-minute updates, read on 10 August 2026. At $12 a month, one avoided mispricing pays for a decade.

What breaks between fifty products and fifty thousand

Everything above works on a category page. Here is what changes when the supplier has 40,000 SKUs and the window is one night.

The lookup that looked free. CIBlockElement::GetList filtered by XML_ID, called once per product, is 40,000 queries with 40,000 round trips before you have written anything. That is the pattern the earlier version of this article used. Preloading the map, as Step 4 now does, is one query and a few megabytes of array. Bitrix has the same shape available for properties: CIBlockElement::GetPropertyValues fetches property values for a set of elements in one call, and it has existed since iblock 14.0. If you are calling GetProperty in a loop, you are paying for the same mistake twice.

Images are the real budget. CFile::MakeFileArray accepts a URL to a file on another site and downloads it. The reference page documents no size limit and no timeout, which means a donor serving a 30 MB TIFF will make your import wait for a 30 MB TIFF. At 40,000 products with two-megabyte photographs, a full re-download is 80 GB of traffic and hours of wall clock, every night, to replace files with identical files. Attach on creation, re-fetch on a changed ETag, and keep a HEAD request between you and the decision.

One wasted second per page across 40,000 pages is eleven hours. That is the arithmetic that decides whether you finish before the store opens, and it applies to the politeness delay as much as to a slow selector: a one-second pause is eleven hours of deliberate waiting before you count the fetch itself. A nightly full crawl of a large donor is not a plan. Crawl the category listings nightly for price and stock, and refresh full product cards on a rotation.

Deletions are invisible. Nothing on a supplier's site announces a discontinued product. If your import only ever adds and updates, discontinued items sit in your catalog forever, in stock, priced, and orderable. Track which XML_IDs the current run saw, and deactivate the ones it did not at the end. Then put a brake on that. If the run saw fewer than 80% of the products it saw last time, something broke on the donor side, and emptying your own catalog is the wrong response to it.

Search and caches do not update themselves for free. CIBlockElement::UpdateSearch exists because the search index is a separate structure. On a bulk import it is usually cheaper to suppress per-element reindexing and rebuild once at the end. The same goes for component caches: 40,000 element writes invalidate them all night, and the first visitor of the morning pays for the rebuild.

Trade offers double everything. If products have variants, the offers live in a second infoblock linked to the first, and price and stock hang on the offer rather than the product. That is not a tweak to the importer. It is a second element type, a second identity scheme, a second dedupe pass, and the reason many teams get a working import for simple products and stall for a week on shoes.

Where this stops working

The catalog is rendered in JavaScript. DOMDocument sees the empty shell. Either find the JSON endpoint the page calls and read that, which is usually faster and more stable than the HTML, or drive a real browser. The trade-off and the tooling are covered in scraping dynamic JavaScript pages.

The donor fights back. Rate limits, fingerprinting, a challenge page that returns HTTP 200 with a body your parser cheerfully turns into a product named after the challenge. If you are being blocked by address, rotating proxies are the usual answer. The point where a single script turns into an infrastructure project is also the point where a managed extraction service starts to look cheap next to a developer's week.

Pagination is a lie. Infinite scroll, cursor tokens, a ?page= parameter that silently caps at 50, and the classic where page 200 returns page 1. The maxPages loop above stops on an empty page and nothing else; handling pagination properly means detecting repeats, not counting.

The supplier is a marketplace. Marketplace listings are aggregations of other sellers, so "the price" is a set of prices, and yesterday's cheapest offer is today's out-of-stock one. Importing that into a single base price produces a catalog that is wrong in a new way each morning.

You need the data more than you need the scraper. Once a store is pulling from six suppliers, each with its own markup, currency and schema, the scraper stops being the interesting part and the pipeline becomes the product. That is the point at which some teams move to purchased feeds or a data as a service arrangement and keep only the mapping in house.

And the honest limit of this article: we could not read the Bitrix Marketplace catalogue or the vendor's edition-comparison table with an automated client, so no module versions or module prices are quoted anywhere above. Everything with a number attached came from a page we actually opened.

Whose catalog is it

The code copies three things: numbers, text and photographs. Only the first is uncontroversial.

Photographs are somebody's copyright, and CFile::MakeFileArray does not ask. Product photography is typically owned by the manufacturer or the distributor who commissioned it, and a distributor's permission to resell goods is not a licence to republish images. Suppliers with a dealer programme usually publish a media pack with exactly those rights in it, and asking takes an email. Suppliers without one have generally not thought about it, which means the answer arrives later and less pleasantly.

Copied descriptions cost twice. Once legally, because a written product description is a protected work in its own right, and once commercially: if forty resellers publish the manufacturer's paragraph verbatim, none of them ranks on it. Rewriting is the tedious answer, and it is also the one that pays.

The catalog as a whole may be protected even when nothing in it is. Under the EU database directive, a compilation can carry a sui generis right based on the investment in obtaining, verifying and presenting its contents, independent of copyright in the individual entries. Two Court of Justice rulings mark out the edges. In Ryanair v PR Aviation, decided 15 January 2015, the Court held that an owner may restrict use by contract even where the database falls outside both copyright and the sui generis right. That puts the terms of use back in play exactly where people assume they do not apply. In CV-Online Latvia v Melons, decided 3 June 2021, the Court framed infringement around whether the extraction and re-use creates a risk to the substantial investment behind the database, rather than treating any copying as infringement.

Technical courtesies are cheap and they are also evidence. robots.txt is a published standard, RFC 9309, on the standards track since September 2022. It says a crawler should not use a cached copy for more than 24 hours, and that on a server error the crawler must assume complete disallow. It also, worth knowing, does not standardise Crawl-delay at all, so your politeness interval is your own policy rather than a directive you are obeying. Honour Disallow, keep the delay, identify yourself in the user agent, and stay off anything behind a login. If a dispute ever happens, the difference between a script that obeyed a published file and one that ignored it is not a technical detail.

For the wider question of what is permissible, see legality of web scraping. None of this is legal advice, and a supplier relationship you can pick up the phone about is worth more than any of it.

What matters in a real project

The examples above are deliberately narrow. A production importer for a real store adds, roughly in the order the pain arrives:

• Trade offers. Variants live in a linked offers infoblock, with price and stock on the offer rather than the product. That means a second element type, a second identity scheme keyed on the donor's variant code, and a second dedupe pass over data where the donor frequently does not publish that code at all. Budget for it as its own project rather than as a flag on this one.
• Property mapping. CIBlockElement::SetPropertyValues writes characteristics, and list-type properties need their values created before they can be assigned. The table mapping the donor's attribute names to yours never stops needing maintenance, and it belongs in configuration, not in the parser.
• Sections. Building the donor's tree with CIBlockSection is easy. Deciding it is the tree you want is a merchandising decision, and importing it wholesale is how stores end up with 400 categories and no navigation.
• Multiple price types and warehouses. Business edition and above. Wholesale and retail become two Price rows per product with different CATALOG_GROUP_IDs, and per-warehouse stock is its own table.
• Reproducibility. Store the rate, the markup, the run timestamp and the source URL with each product. Six months later, "why is this priced at $72.07" has an answer.
• Idempotency. The second run must produce the same state as the first. Run it twice on purpose during development; it is the fastest way to find the places where you add instead of update.
• Monitoring that fires on silence. The failure that hurts is not the run that crashes, it is the run that finishes early with 200 products instead of 8,000 and reports success. Alert on the count, not on the exit code.

For continuous price tracking rather than one-off filling, the pattern shifts again: fewer fields, far more often, and a change log rather than an overwrite. That is a different program, sketched in online store monitoring.

Wrapping up

For a standard store with a cooperative supplier, the boring answer is still the right one: get a CommerceML feed if it exists, generate a CSV if it does not, and let Bitrix's own import write the rows. Custom code earns its keep at the edges, where the donor is awkward, the schema is yours, and stock has to be right at eight in the morning.

If you write that code, the platform is the easy half. Use \Bitrix\Catalog\Model\Product and \Bitrix\Catalog\Model\Price rather than the 2017-era methods older tutorials still show, leave AVAILABLE alone, preload your identity map, and resolve the exchange rate once per run and log it.

The hard half is arithmetic that never raises an exception. A parser that reads 1.234,56 as 1.234, a conversion that quietly falls back to a default rate typed in during setup, an Add that no-ops on the update path, an empty CSV overwriting a good one. Every one of those was in the previous version of this article, every one of them produces a number that looks fine, and none of them shows up in a log. Write the checks that catch them, print three counters instead of one, and the rest is plumbing.