URL Parsing: What It Is and How to Parse URLs in Any Language

URL parsing (from parse — "to break down") is the act of splitting a web address into its separate parts: protocol, domain, port, path, query parameters, and fragment. From a raw string like https://example.com/catalog?page=2 you get a structured object whose fields you can read one by one and change when needed.

At first glance, parsing a URL looks trivial — "just split the string on some characters." In practice, a URL has a formal specification (RFC 3986) with plenty of subtleties: percent-encoding of special characters, relative links, optional address parts, normalization. That is why nearly every language ships a built-in URL parser, and writing your own with regular expressions is almost never the right move.

What a URL is made of

Let's take a maximally "complete" address so we can see every component:

https://user:pass@example.com:8080/catalog/items?page=2&sort=price#reviews
└─┬─┘   └──┬───┘ └────┬────┘└─┬─┘└─────┬──────┘ └──────┬──────┘ └──┬──┘
scheme  userinfo    host    port      path             query     fragment

These are exactly the parts a parser pulls out. From there you can work with them: read the value of a specific GET parameter, swap the domain, drop the fragment, or rebuild a new URL from the modified components.

Why you need URL parsing

URL parsing shows up in almost any application that works with web addresses. The most common scenarios:

• Extracting query parameters. You need to know which catalog page the user is on (?page=5), which filter is applied, or which UTM tag came in. A parser turns the query string into a convenient dictionary.
• Validating and sanitizing links. Before you store a user-supplied URL in a database or drop it into a redirect, you want to confirm the scheme is allowed (not javascript:, say) and the domain is on an allowlist.
• Routing. Web frameworks parse the path and parameters to decide which handler to call.
• Analytics and logging. To group hits by domain or path, addresses are first broken into parts.
• Web scraping and crawling. This is where URL parsing matters most — more on that below.

URL parsing when building crawlers

A crawler walks a site by following links from page to page. If you want the full picture of how crawling differs from scraping and from parsing itself, start with our explainer on scraping vs. parsing vs. crawling. In short: the crawler handles traversal and page discovery, the scraper handles data extraction, and parsing handles breaking down the content it gets back — including the addresses.

The main trap when crawling a site is that links in HTML are almost always relative, not absolute. In the markup you'll see:

<a href="/about">About us</a>
<a href="../catalog/phones">Phones</a>
<a href="page2.html">Next</a>
<a href="#reviews">Reviews</a>

A bot can't follow /about or page2.html directly — those aren't complete addresses. To keep crawling, the relative link has to be resolved to an absolute form using the current page's address (the base URL). That's exactly what URL parsing does:

This operation is called URL resolution, and nearly every language has a ready-made function for it: urljoin in Python, the new URL(href, base) constructor in JavaScript, URI.resolve() in Java, ResolveReference() in Go.

Normalization and killing duplicates

Beyond resolving to absolute form, a crawler needs to normalize addresses so it doesn't visit the same page multiple times. Typical moves:

• drop the #fragment — it points to a spot inside a page, not a separate resource (/page#top and /page are the same page);
• lowercase the scheme and domain;
• remove the default port (:443 for https);
• sort or filter the query parameters (for example, strip UTM tags).

After normalization, absolute URLs go into a set of already-visited addresses — so the bot doesn't loop and doesn't make redundant requests. If you scrape at any real volume, this is the same data normalization discipline that keeps your output clean downstream.

URL parser examples

Below is how to break a URL into parts and how to resolve a relative link using the built-in tools of popular languages.

Python

The standard library handles everything through urllib.parse.

from urllib.parse import urlparse, urljoin, urldefrag, parse_qs

url = "https://example.com:8080/catalog/items?page=2&sort=price#reviews"
parsed = urlparse(url)

print(parsed.scheme)    # https
print(parsed.hostname)  # example.com
print(parsed.port)      # 8080
print(parsed.path)      # /catalog/items
print(parsed.query)     # page=2&sort=price
print(parsed.fragment)  # reviews

# Query parameters -> dict
print(parse_qs(parsed.query))  # {'page': ['2'], 'sort': ['price']}

# Resolving a relative link (for a crawler)
base = "https://example.com/catalog/items/"
print(urljoin(base, "../about"))    # https://example.com/catalog/about
print(urljoin(base, "/contacts"))   # https://example.com/contacts
print(urljoin(base, "page2.html"))  # https://example.com/catalog/items/page2.html

# Drop the fragment during normalization
print(urldefrag("https://example.com/page#reviews")[0])  # https://example.com/page

JavaScript (browser and Node.js)

The built-in URL class works the same in the browser and in Node.js. It resolves relative links too — pass the base address as the second argument.

const url = new URL("https://example.com:8080/catalog/items?page=2&sort=price#reviews");

console.log(url.protocol);  // "https:"
console.log(url.hostname);  // "example.com"
console.log(url.port);      // "8080"
console.log(url.pathname);  // "/catalog/items"
console.log(url.search);    // "?page=2&sort=price"
console.log(url.hash);      // "#reviews"

// Query parameters
console.log(url.searchParams.get("page"));  // "2"

// Resolving a relative link
const base = "https://example.com/catalog/items/";
console.log(new URL("../about", base).href);   // "https://example.com/catalog/about"
console.log(new URL("/contacts", base).href);  // "https://example.com/contacts"

PHP

Basic parsing is done by parse_url(), and parse_str() turns the query string into an array.

$url = "https://example.com:8080/catalog/items?page=2&sort=price#reviews";
$parts = parse_url($url);

echo $parts['scheme'];    // https
echo $parts['host'];      // example.com
echo $parts['port'];      // 8080
echo $parts['path'];      // /catalog/items
echo $parts['fragment'];  // reviews

parse_str($parts['query'], $query);
print_r($query);          // ['page' => '2', 'sort' => 'price']

PHP has no built-in urljoin equivalent, so to resolve relative links in crawlers people usually add a library — for example league/uri or the Guzzle helpers (GuzzleHttp\Psr7\UriResolver).

Java

The standard library's java.net.URI class handles addresses, and resolve() gives you relative-link resolution directly.

import java.net.URI;

URI uri = URI.create("https://example.com:8080/catalog/items?page=2&sort=price#reviews");

System.out.println(uri.getScheme());    // https
System.out.println(uri.getHost());      // example.com
System.out.println(uri.getPort());      // 8080
System.out.println(uri.getPath());      // /catalog/items
System.out.println(uri.getQuery());     // page=2&sort=price
System.out.println(uri.getFragment());  // reviews

// Resolving a relative link
URI base = URI.create("https://example.com/catalog/items/");
System.out.println(base.resolve("../about"));   // https://example.com/catalog/about
System.out.println(base.resolve("/contacts"));  // https://example.com/contacts

Go

The standard library's net/url package covers both parsing and link resolution (ResolveReference).

package main

import (
    "fmt"
    "net/url"
)

func main() {
    u, _ := url.Parse("https://example.com:8080/catalog/items?page=2&sort=price#reviews")

    fmt.Println(u.Scheme)               // https
    fmt.Println(u.Hostname())           // example.com
    fmt.Println(u.Port())               // 8080
    fmt.Println(u.Path)                 // /catalog/items
    fmt.Println(u.RawQuery)             // page=2&sort=price
    fmt.Println(u.Query().Get("page"))  // 2

    // Resolving a relative link
    base, _ := url.Parse("https://example.com/catalog/items/")
    ref, _ := url.Parse("../about")
    fmt.Println(base.ResolveReference(ref))  // https://example.com/catalog/about
}

A practical example: collect every link on a page

Let's put it together — the classic crawler step: parse the HTML, pull every link, and resolve them all to absolute form. Here's a Python version with requests and BeautifulSoup:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urldefrag

def extract_links(page_url: str) -> set[str]:
    response = requests.get(page_url, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")

    links = set()
    for a in soup.find_all("a", href=True):
        # 1. resolve the relative link to absolute
        absolute = urljoin(page_url, a["href"])
        # 2. drop the fragment so we don't create duplicates
        absolute, _ = urldefrag(absolute)
        # 3. keep only http(s) links
        if absolute.startswith(("http://", "https://")):
            links.add(absolute)

    return links

for link in extract_links("https://example.com/catalog/"):
    print(link)

Here URL parsing does two key jobs: urljoin turns relative hrefs into full addresses the bot can follow, and urldefrag keeps /page and /page#top from being counted as different pages.

Common mistakes

• Parsing URLs with regular expressions. Because the spec has so many edge cases, a hand-rolled parser is almost always incomplete. Use the built-in libraries.
• Forgetting about relative links. Without resolving to absolute form, a crawler trips on the first internal link.
• Not normalizing addresses. Without dropping the fragment, ordering parameters, and lowercasing, the bot crawls the same pages in circles.
• Trusting the scheme without checking. Before a redirect or before inserting a link, make sure the scheme is safe (http/https) — otherwise you open the door to XSS via javascript:.

Conclusion

URL parsing is a basic but important operation: it turns an address string into a structure that's easy to work with. In ordinary applications it's for extracting parameters, validation, and routing; in crawlers it becomes critical — without resolving relative links to absolute form and without normalization, walking a site simply isn't possible. The good news is that nearly every language ships a reliable built-in tool for this, so there's no wheel to reinvent.

If you'd rather skip the plumbing entirely and just receive clean, structured data, scraping.pro offers crawling and data extraction as a service — we handle the URL frontier, normalization, and delivery so you get the fields you asked for, not a pile of raw HTML.