requirements.txt vs pyproject.toml

Mental Model

View requirements.txt as a sequence of shell commands for pip, including install instructions, --index-url flags, and file:// paths. pyproject.toml, on the other hand, is a structured, declarative configuration file that defines your project's metadata and build requirements in a standardized, machine-readable format.

Rule: Always use pyproject.toml to declare project metadata and build requirements instead of relying on legacy parsing of requirements.txt.

The Setup

You are setting up a shared internal Python library. You want developers to easily install it while ensuring your build tools (like linters, compilers, and test runners) use the exact same declarative metadata.

What Does This Print?

⚠ Broken code

Python

# setup.py (Old way) or manual requirements.txt parsing inside setup.py
# This is fragile and breaks modern standards PEP 517 / PEP 518.
from setuptools import setup

with open("requirements.txt") as f:
    # Reading requirements.txt inside setup.py is a classic anti-pattern
    install_requires = f.read().splitlines()

setup(
    name="my-legacy-package",
    version="1.0.0",
    install_requires=install_requires,
)

What happens if you run pip install on this package when requirements.txt contains flags like --index-url or local file references?

The Output

What actually happens

ValueError: Invalid requirement: '--index-url'

The execution fails because requirements.txt is an instruction file containing arbitrary command-line flags meant for the pip installer, not a list of declarative package dependencies. Parsing this file inside setup.py breaks standard package installation pipelines because abstract metadata fields cannot process these command-line flags. This creates fragile build configurations that crash during modern containerization and publishing steps. Standardizing on pyproject.toml separates installer-specific configuration from the explicit package metadata, ensuring compliance with modern build standards. It also prevents execution of arbitrary logic during the metadata extraction phase of a package, resolving a long-standing security and compatibility issue in the packaging ecosystem.

Why Python Does This

Historically, package metadata in Python was defined imperatively inside setup.py files. This required executing Python code simply to discover a package's name and dependencies. To fix this, the community introduced PEP 518 and PEP 621, which standardized pyproject.toml as a declarative configuration file. The build backend reads metadata statically without executing untrusted code. By defining project requirements inside pyproject.toml, you allow the installer to verify the dependency graph before setting up the build environment. This removes the need for custom parsing scripts and ensures interoperability across all packaging tools.

The Fix

✓ Corrected pattern

Python

# pyproject.toml — the modern, declarative alternative.
# Abstract dependencies only: package names and version constraints.
# No --index-url flags, no -r includes, no arbitrary pip instructions.

PYPROJECT_CONTENT = """
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my-modern-package"
version = "1.0.0"
dependencies = [
    "fastapi>=0.110.0",
    "pydantic>=2.0"
]
"""

# Verify that abstract deps parse cleanly (no ValueError from pip flags):
from packaging.requirements import Requirement

for dep in ["fastapi>=0.110.0", "pydantic>=2.0"]:
    req = Requirement(dep)
    print(f"Valid dep: {req.name} {req.specifier}")
# Valid dep: fastapi >=0.110.0
# Valid dep: pydantic >=2.0

pyproject.toml explicitly defines project dependencies using a standardized TOML format (e.g., in the [project] or [tool.poetry.dependencies] sections), making them machine-readable and parsable by build backends and dependency managers without needing to interpret pip-specific command-line arguments.

How This Fails in Real Systems

A financial calculations service had an automated deployment pipeline that parsed requirements.txt into an installer. A developer added a -r dev-requirements.txt flag into the file. The setup helper crashed during compilation, blocking a critical security patch deployment for 6 hours because the build toolchain could not handle CLI-only arguments in metadata fields.

Key Takeaway

Always use pyproject.toml to declare project metadata and build requirements instead of relying on legacy parsing of requirements.txt.

Common mistake: Developers treat requirements.txt as a declarative list of package dependencies suitable for programmatic parsing, rather than an instruction script meant exclusively for the pip installer which can contain arbitrary CLI flags.