Applied Data Science in Finance

Week 2 – Data collection and web scraping

Jérôme Deyris & Thomas Rigou

2026-11-16

Outline

  1. A bit of theory: reflections on data collection
  2. Retrieving existing corpora
    1. Private data in economics
    2. Downloading open datasets
    3. Querying free APIs
  3. Web scraping
    1. Theory and packages
    2. Simple cases
    3. Complex case
    4. A loop in a loop (in a loop)
    5. Dealing with PDFs
  4. Conclusion

1. Theory

Reflexivity on your data

  • Ask yourself what data you need
    • Can you separate the needle from the haystack? Is the needle even in there? Is it the right needle to answer your research question(s)?
    • Is your haystack representative of the whole field? What about what you do not have access to? Are there biases? Is it a good proxy?

A corpus is a sample

Choosing documents is choosing a sample: start from a population and a quantity of interest

  • Who do you want to learn about? What do you want to measure?
  • e.g. “How do investors perceive the ECB?” ≠ “What do people say about the ECB on X/Twitter?”

Most text corpora are found data: produced by institutions for their own purposes, not by a sampling design (Grimmer et al., 2022, ch. 4)

  • Same idea as Groves’s organic data (session 1)

The question is not “is my corpus biased?” (it is) but how, and what that does to your answer.

Four types of bias (Grimmer et al., 2022, ch. 4)


1. Resource

Who can afford to produce, record and store texts?

2. Incentive

Who wants to hide, perform or destroy?

3. Medium

What does the format let through, and what does it lose?

4. Retrieval

How were documents selected from a larger collection?

1. Resource bias

  • Texts are expensive to produce, record and store: corpora over-represent those with more resources
    • Large central banks publish more, translate into English, keep better archives, are collected by the BIS in their institutional repository
    • Archives / repositories keep what someone found worth (or not too cumbersome) keeping
  • An example: more documents later than earlier
    • Almost every multi-decade collection grows over time
    • A rising count of “climate” speeches, or of “fed pressures” may just reflect a rising count of speeches or newspapers

Normalize (shares rather than counts) and ask who is missing, not only what is there.

2. Incentive bias

  • Individuals and institutions have incentives to perform, hide or destroy what could cast them in a bad light
  • A more “finance” relevant example: Transparency changes what gets said
    • FOMC transcripts: until late 1993, members thought their debates were private; since then, transcripts are published with a 5-year lag
    • After the change, newer members speak less, ask fewer questions and sound more like the Chair — but also bring more data to the table (Hansen et al., 2018)
    • Schonhardt-Bailey, Cheryl (2013) interviewed FOMC members about how candid the transcripts really are

There’s also the case of ECB diaries and forbidden meetings during the silent periods!

3. Medium bias

  • The medium shapes the content: what is recorded, and what is lost
    • Transcripts lose tone of voice, hesitations, body language ; or may even drop interventions entirely
    • The same press conference moves markets through the chair’s voice, beyond the words (Gorodnichenko et al., 2023)
  • An example: Formats and platforms change over time
    • Twitter: 140 characters until late 2017, then 280

    • Transcription practices evolve / differ

      • e.g. in the French Parliament, see Banque de France hearings or Algan et al. (2025)

4. Retrieval bias

  • Corpora are often built by querying a larger collection: keywords, search engines, APIs
    • These search methods are often black boxes that are difficult to keep tractable
      • case sensitivity, sampling, “most relevant” results
    • This leads to the worst type of bias, one that is not even known by the researcher
  • A few examples:

    • Keyword searches systematically miss relevant documents and catch irrelevant ones
      • The EPU index: economy + uncertainty + policy terms (Baker et al., 2016)
      • Fed pressures in Factiva or ProQuest by Bonaparte (2026)

Read documentation about how queries work in closed-source environments, check what your query may miss, and be transparent about how your corpus was built (and its limitations)

2. Retrieving existing corpora

A. Private (text) data in economics

A long term trend

  • Confidential Census microdata: from 0.21% of published economics papers in 1991 to 1.27% in 2019 (Nagaraj et al., 2025)
    • These papers are 50% more likely to appear in a top-5 journal, and 80% more cited in policy documents
  • Private company data: from about 4% of papers in 1980 to over 8% in 2023, especially in finance and industrial organization (Garg & Fetzer, 2025)

Access shapes research: when a secure data center opens nearby, economists’ use of confidential data more than doubles, and their top publications rise by 18% (Nagaraj & Tranchero, 2023)

Concerns about replicability and transparency

This is not without problems

  • Replications are rare in general: less than 0.1% of publications in the top-50 economics journals (1974–2014); they “bring little recognition for their authors despite the substantial work they entail” (Czibor et al., 2019)
  • Confidential data blocks reproduction (even more!): in AEJ: Applied, 25% of articles are fully reproducible, 43% among those with non-confidential data (Herbert et al., 2021)
  • Few can even try: confidential data are used mostly by researchers at top institutions with prior top publications (Nagaraj et al., 2025)

As a result, the use of private data facilitates fraud…

Too good to be true

  • Nov 2024: an MIT PhD student’s paper: an AI tool raised discoveries (+44%) and patents (+39%) in the R&D lab of an unnamed firm
  • Dec 2024: Acemoglu and Autor praise it
    • Acemoglu: “It’s fantastic”
    • Autor: “I was floored”

Wall Street Journal, 29 December 2024

The house of cards falls

  • Jan 2025: a computer scientist doubts it: only 3M and Corning employ that many materials scientists, and both deny it
  • May 2025: MIT has “no confidence in the provenance, reliability or validity of the data”
  • Nov 2025: the student admits faking a data-use agreement: “a huge and embarrassing act of dishonesty”
    • Autor: “There is no world where this makes any sense”
    • Acemoglu: “In the future, of course, people will become more paranoid in economics”

In this course, we’ll stick with public data ;)

B. Downloading open datasets

Open textual data: companies / producers

Without any exhaustivity, a few ideas on where / what to look for:

  • Annual reports (10-K), 1993–2020, already split by section (business, risk factors, MD&A…)
  • Cleaned 10-K and 10-Q files, with the finance dictionaries
    • SRAF, University of Notre Dame (Loughran & McDonald)
  • Patents: full text of US patents since 1976
    • PatentsView (USPTO); used by Kelly et al. (2021) to date breakthrough innovations
  • Job ads in US newspapers, 1940–2000, with the tasks they describe

Open textual data: policymakers / regulators

Open textual data: consumers / workers

  • Financial complaints, in the consumer’s own words
  • Product reviews, 1996–2023, with product descriptions
  • Retail investors on Reddit: r/WallStreetBets during GameStop
  • Answers to open-ended survey questions
    • Replication packages, e.g. households’ explanations of inflation: paper · data

Ready-made datasets rarely fit exactly

  • Where to look: the replication packages of papers, on openICPSR (AEA), Harvard Dataverse, Zenodo, Hugging Face
  • But often…
    • Not quite what you need: another scope, unit or period than your research question
    • Out of date: EDGAR-CORPUS stops in 2020, the Stanford Congress text in 2017, the ConvoKit FOMC corpus in 2008
    • Built in ways you must check: e.g. BIS central bankers’ speeches have missing metadata and uneven coverage (cbspeeches.com)
    • The underlying data is not shared: licences, confidentiality, privacy (→ part A)

C. Querying free APIs

What is an API?

  • API (Application Programming Interface): a door that a data producer opens for programs, not for humans
    • You send a request, the server sends back structured data (usually JSON): no web page to parse
  • Like a restaurant: you read the menu (the documentation), you order (the request), the kitchen serves (the response)
  • Many producers of text have one: the CFPB (complaints), the Guardian (articles), the US Congress (bills, debates)…

API first: if an API exists, use it. It is faster, cleaner and usually legal.

Requests and responses

The request (what you send)

  • A method: GET = “give me this”
  • A URL: where, and what exactly
  • Headers: who you are, your key, the format you accept

The response (what you get back)

  • A status code: did it work?
  • Headers: date, type of content…
  • A body: the content (HTML for a web page, JSON for an API)
Code Meaning
200 OK
403 Forbidden: missing or wrong identification or key
404 Not found: wrong URL
429 Too many requests: slow down

Build requests: read the doc!

Every API has its own syntax: you only learn it from its documentation

What to look for?

  • The base URL: the address of the API
  • The parameters: their names and accepted values (search_term, size…)
  • Access rules: key, identification, rate limits, pages of results
  • The fields in the answer and what they mean (product, issue, company…)
  • The licence: what you may do with the data

Build requests: an example

https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/?search_term=overdraft&size=5
  • https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/ → the base URL: the CFPB complaint API, version 1
  • Endpoint: what comes after the base URL = which service of the API you call
    • here, nothing after v1/: the API’s root, i.e. the search over all complaints
  • ? → then the query parameters, as key=value pairs separated by &
    • search_term=overdraft: full-text search · size=5: number of results returned

Paste the URL in your browser: the answer is not a web page but JSON.

Build requests: in Python

import requests

my_identity = {"User-Agent": "Jane Doe jane.doe@univ-paris1.fr"}   # who you are (an honest identity)

# base URL
api_url = "https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/"
my_params = {"search_term": "overdraft", "size": 5}   # the query parameters, as a dictionary

raw_response = requests.get(api_url, params=my_params, headers=my_identity)   # send the request (GET)
print(raw_response.status_code)                       # did it work?
200
  • requests writes the ?search_term=overdraft&size=5 part for you
  • headers tells the server who we are: we send it every time (more in part 3)
  • 200 means it worked: the answer has arrived, as JSON text

Parse answers: JSON

The whole answer (top level)

{
  "took": 119,
  "timed_out": false,
  "_shards": {"total": 5, "successful": 5, ...},
  "hits": {
    "total": {"value": 21862, "relation": "eq"},
    "max_score": null,
    "hits": [ {...}, {...}, {...}, {...}, {...} ]
  },
  "aggregations": {"product": {...}, "company": {...}, ...},
  "_meta": {"license": "CC0", ...}
}

How to read it

  • { } an object: named fields, "key": value
  • [ ] an array: an ordered list of values
  • Values: text "...", numbers, true/false, null (empty)

One complaint (the first of the 5 hits)

{
  "_index": "complaint-public-v1",
  "_id": "9322415",
  "_score": 13.282445,
  "_source": {
    "product": "Prepaid card",
    "date_sent_to_company": "2024-06-21T19:10:11.000Z",
    "issue": "Problem with overdraft",
    "sub_product": "Payroll card",
    "zip_code": "74133",
    "tags": null,
    "complaint_id": "9322415",
    "timely": "Yes",
    "company_response": "Closed with explanation",
    "submitted_via": "Referral",
    "company": "ADP Inc.",
    "date_received": "2024-06-20T17:04:28.000Z",
    "state": "OK",
    "company_public_response": null,
    "sub_issue": "Overdraft charges"
  },
  "highlight": {...},
  "sort": [13.282445, "9322415"]
}

Parse answers: in Python

.json() translates JSON into Python

JSON Python
{ } object dictionary
[ ] array list
"text" str
12, 13.28 int, float
true, null True, None
  • Dictionary → values by key: ["hits"]
  • List → values by position: [0], Python counts from 0
  • We go down the tree, one step at a time
structured_response = raw_response.json()          # JSON text -> Python dictionary
complaints = structured_response["hits"]["hits"]   # the list of 5 complaints
first_complaint = complaints[0]["_source"]         # the first one, its fields
print(first_complaint["company"], "|", first_complaint["issue"])   # ADP Inc. | Problem with overdraft

APIs: pros and cons

Advantages

  • Easy and legal access to data
  • Structured output, easy to process
  • Often a ready-made package exists

Limits

  • You only get what the provider wants to share
  • Rate limits, quotas, sometimes a price
  • APIs can close or change overnight
    • The “APIcalypse”: Facebook, then Twitter/X closed free research access
  • Results often come in pages; documentation can be thin
  • Opaque selection (→ retrieval bias)

3. Web scraping

A. Theory and packages

What is web scraping?

The extraction and copying of data from a website into a structured format using a computer program.

  • We all scrape by hand: copy, paste, download
  • A program does it faster and more exhaustively
  • Use cases: press releases and speeches, reports and PDFs, metadata, Wikipedia tables…

Step by step

1. Identify the URL

of the page to analyse

2. Analyse the source code

of the page

(right click → View page source)

3. Locate the information

which tags? which attributes?

4. Store it

in variables, files, a database

Steps 2 and 3 go back and forth: look at the source, try, look again.

Source code & HTML

  • Open the list of S&P 500 companies on Wikipedia, then right click → View page source (Ctrl+U / ⌘+Option+U): this is the HTML code of the page
  • HTML is made of tags (opened and closed), with attributes, nested in a tree
<table class="wikitable sortable" id="constituents">        <!-- the table: its id is "constituents" -->
  <tr> <th>Symbol</th> <th>Security</th> <th>GICS Sector</th> ... </tr>   <!-- the header row -->
  <tr>                                                       <!-- one row = one firm -->
    <td><a href="https://www.nyse.com/quote/XNYS:MMM">MMM</a></td>  <!-- a cell, with a link -->
    <td><a href="/wiki/3M">3M</a></td>                       <!-- the firm's name -->
    <td>Industrials</td>                                     <!-- its sector -->
    ...
  • h1, h2…: titles · p: paragraph · a: link · div: block · table, tr, td: tables

Inspect a page in your browser

  • Try it on the list of S&P 500 companies: right click on a firm’s name
  • Right click on an element → Inspect (or F12)
    • The HTML of that element is highlighted
    • Note its tag (p, a, div…) and its attributes (class, id)
  • Right click → View page source
    • The raw HTML, as your program will download it
  • Text in the inspector but not in the source? It is loaded by JavaScript (→ To go further)

Required packages

requests

  • downloads the page
  • sends the request, returns the answer (HTML, JSON, PDF…)

BeautifulSoup (package bs4)

  • reads the page
  • turns the HTML into a tree we can search


  • Both are already installed in Google Colab (and Anaconda)
  • Otherwise: pip install requests beautifulsoup4
import requests
from bs4 import BeautifulSoup

requests

  • requests is a Python module for sending HTTP requests: it asks a server for a page, like your browser
  • Some websites block programs. Solution: send a header with a User-Agent (who you are)
    • we send it every time from now on: it is good practice
import requests

my_identity = {"User-Agent": "Jane Doe jane.doe@univ-paris1.fr"}   # who you are
page_url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"

raw_response = requests.get(page_url, headers=my_identity)   # ask the server for the page
print(raw_response.status_code)                              # 200 = OK

page_html = raw_response.text                                # the HTML code, as one long text
print(page_html[:500])                                       # its first 500 characters

BeautifulSoup

  • A Python library for parsing HTML and XML documents
  • It produces a tree that can be used to search for elements
  • Here, the tree of exercise 1’s page

%%{init: {"themeVariables": {"fontSize": "22px"}}}%%
flowchart LR
  html --> head
  html --> body
  head --> title["title<br/>First example"]
  body --> p["p<br/>I want to get this text"]
  body --> h1["h1<br/>But not this one"]
  body --> h2["h2<br/>Neither this one"]
  body --> h3["h3<br/>But I want this one"]

BeautifulSoup: find and find_all

  • find → the first tag that matches (or None) · find_all → all the tags that match: a list
Filter on… Example What you get
a tag name page.find("title") the first (usually only) <title> tag
a tag name page.find_all("tr") all the rows of all the tables
several tag names page.find_all(["p", "h3"]) all the <p> and <h3> tags
a tag + an attribute page.find("table", {"id": "constituents"}) the table whose id is “constituents”
a tag + an attribute page.find_all("a", {"class": "art"}) the links whose class is “art”
a tag + two attributes page.find_all("a", {"class": "art", "href": "article-1-xyz.html"}) the links with both
  • Then read what you found: tag.text → its text · tag["href"] → the value of an attribute

BeautifulSoup: select, with CSS selectors

  • select does the same as find_all, with the CSS selectors of web developers (select_one = find)
  • . = a class · # = an id · a space = “inside”
With find / find_all With select What you get
page.find_all("a") page.select("a") all the links
page.find_all("a", {"class": "art"}) page.select("a.art") the links of class “art”
page.find("table", {"id": "constituents"}) page.select_one("#constituents") the table of id “constituents”
page.find("main").find_all("a") page.select("main a") the links inside <main>
page.find("table", {"id": "constituents"}).find_all("tr") page.select("#constituents tr") the rows of that table

In the browser: right click → Inspect → right click on the tag → Copy → Copy selector: a ready-made CSS selector.

BeautifulSoup with requests

import requests
import pandas as pd
from bs4 import BeautifulSoup

my_identity = {"User-Agent": "Jane Doe jane.doe@univ-paris1.fr"}
page_url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
raw_response = requests.get(page_url, headers=my_identity)          # 1. download the page
page = BeautifulSoup(raw_response.text, "html.parser")              # 2. parse the HTML into a tree

sp500_table = page.find("table", {"id": "constituents"})            # 3. find the table...
all_firms = []
for row in sp500_table.find_all("tr")[1:]:                          #    ...each of its rows (except the header)
    cells = row.find_all("td")                                      #    ...the cells of this row
    all_firms.append((cells[0].text.strip(), cells[1].text.strip(), cells[2].text.strip()))

sp500 = pd.DataFrame(all_firms, columns=["symbol", "name", "sector"])   # 4. store it in a table
print(sp500.shape)
sp500.head(3)
(503, 3)
  symbol                 name       sector
0    MMM                   3M  Industrials
1    AOS          A. O. Smith  Industrials
2    ABT  Abbott Laboratories  Health Care

B. Simple cases

Exercise 1: get data from a simple website

On thomas-renault.com/side/ex1.html, collect the texts “I want to get this text” and “But I want this one”, but not the others.

  1. Go to the URL
  2. Look at the source code: right click → View page source, Ctrl+U, or F12
  3. Using BeautifulSoup, extract the text

Exercise 1: the plan

  1. Look at the source: which tags hold the texts we want? Which tags hold the others?
  2. Download the page: requests.get(page_url, headers=my_identity)
  3. Parse it: BeautifulSoup(raw_response.text, "html.parser")
  4. Extract the right tags: page.find_all(...)
  5. Store their text in a list: .text, .append()

Exercise 1: solution

import requests
from bs4 import BeautifulSoup

my_identity = {"User-Agent": "Jane Doe jane.doe@univ-paris1.fr"}
page_url = "https://www.thomas-renault.com/side/ex1.html"
raw_response = requests.get(page_url, headers=my_identity)   # download the page
page = BeautifulSoup(raw_response.text, "html.parser")   # parse the HTML into a tree


Solution 1: two loops

wanted_texts = []
for tag in page.find_all("p"):      # the <p> tags
    wanted_texts.append(tag.text)
for tag in page.find_all("h3"):     # the <h3> tags
    wanted_texts.append(tag.text)

Solution 2: one loop instead of two

wanted_texts = []
for tag in page.find_all(["p", "h3"]):   # <p> or <h3>
    wanted_texts.append(tag.text)
print(wanted_texts)  →  ['I want to get this text', 'But I want this one']

Exercise 2: get specific data from a simple website

On thomas-renault.com/side/ex2.html, get only the links to the four articles, not the “Contact us” link.

  • Open the page, then look at its source: what distinguishes the article links from the other one?

Exercise 2: the plan

  1. Look at the source: all the links are <a> tags; what differs between them?
  2. Download and parse the page, as in exercise 1
  3. Extract only the right links: three possible filters
    • where they are: inside <main>
    • what their address contains: "article-"
    • which class they have: "art"
  4. Get their address: link["href"]

Exercise 2: solution

page_url = "https://www.thomas-renault.com/side/ex2.html"
raw_response = requests.get(page_url, headers=my_identity)
page = BeautifulSoup(raw_response.text, "html.parser")


1. Inside <main>

links = []
main = page.find("main")
for link in main.find_all("a"):
    links.append(link["href"])
print(links)

2. URL contains “article-”

links = []
for link in page.find_all("a"):
    if "article-" in link["href"]:
        links.append(link["href"])
print(links)

3. Class “art”

links = []
for link in page.find_all("a",
        {"class": "art"}):
    links.append(link["href"])
print(links)
['article-1-xyz.html', 'article-2-ert.html', 'article-3-sdf.html', 'article-4-vbn.html']

C. A real world case

A real world case

Many reasons why real world tasks will be more complex:

  1. HTML will be much more complex
  2. You will try to gather more than one field, and have to clean your data
  3. Data will be scattered on many pages rather than one
  4. Final text may be stored in PDFs rather than in plain HTML

We’ll see (most of) these problems in a real world example, scraping the publications of the Banque de France.

Step 1: Same old

What is the first step? Same as before!

  1. Request and parse the page:
    • download it (with requests and a User-Agent)
    • parse it (with BeautifulSoup and a "html.parser")
import requests
from bs4 import BeautifulSoup

my_identity = {"User-Agent": "Jane Doe jane.doe@univ-paris1.fr"}
list_url = "https://www.banque-france.fr/en/publications-and-statistics/publications"
raw_response = requests.get(list_url, headers=my_identity)    # download the page
print(raw_response.status_code)                              # did it work?
200
list_page = BeautifulSoup(raw_response.text, "html.parser")   # parse it into a tree
print(list_page.find("title").text)                           # quick check: the page's title
Publications | Banque de France

Step 2. Same old

  • Right click on a publication → Inspect
  • The whole card is highlighted: it is a link
    • tag: <a>
    • class: "card" (among others: card-vertical, rounded…)
    • href: the address of the publication (relative)
  • Further down inside the card (for later):
    • the title: an <h3>
    • the date: a <small>

Step 2. Same old

This informs the next coding step, using BeautifulSoup and find_all!

all_cards = list_page.find_all("a", {"class": "card"})       # one card per publication
print(len(all_cards))
12

Build a loop: for each card, keep its link (the card is an <a>: its href)

all_publications_hrefs = []                             # an empty list, to store the results
for card in all_cards:                                  # the loop
    all_publications_hrefs.append(card["href"])         # the link of the card: add and repeat

print(len(all_publications_hrefs))
print(all_publications_hrefs[0])
12
/en/publications-and-statistics/publications/why-dont-fuel-prices-always-strictly-follow-global-oil-prices

Are we done yet?

Let’s try to use the first link we collected:


print(all_publications_hrefs[0])
/en/publications-and-statistics/publications/why-dont-fuel-prices-always-strictly-follow-global-oil-prices


raw_response = requests.get(all_publications_hrefs[0], headers=my_identity)
MissingSchema: Invalid URL '/en/publications-and-statistics/publications/why-dont-fuel-prices-always-strictly-follow-global-oil-prices': No scheme supplied. Perhaps you meant https:///en/publications-and-statistics/...?


Python failed. Paste this link in your browser: it fails too, it is not a complete web address.

Completing URLs

Our links start with /: they are relative, only the end of the address.

https://www.banque-france.fr

the site: missing, we add it

+

/en/publications-and-statistics/publications/why-dont-...

the path: what card["href"] gives us

  • Your browser knows which site you are on: it completes the link for you

  • requests does not: requests.get("/en/...") fails

    → we have to add the site’s base URL in front

publication_url = "https://www.banque-france.fr" + card["href"]     # site + path = full (absolute) URL
https://www.banque-france.fr/en/publications-and-statistics/publications/why-dont-fuel-prices-always-strictly-follow-global-oil-prices

Step 3. Add metadata

The title: an <h3>

The date: a <small>

Same loop, with new lines for the title and the date:

all_publications = []
for card in all_cards:
    publication_url = "https://www.banque-france.fr" + card["href"]
    publication_title = card.find("h3").text.strip()     # NEW: the title
    publication_date = card.find("small").text.strip()   # NEW: the date
    all_publications.append((publication_date, publication_title, publication_url))   # one tuple

print(len(all_publications))
print(all_publications[0])
12
('13th of October 2026', "Why don't fuel prices always strictly follow global oil prices?", 'https://www.banque-france.fr/en/publications-and-statistics/publications/why-dont-fuel-prices-always-strictly-follow-global-oil-prices')

Step 4. Export as a DataFrame

all_publications is a list of tuples: that is already a table in disguise.

  • one tuple = one row (one publication)
  • one position in the tuple = one column (date, title, url)
import pandas as pd

bdf_publications = pd.DataFrame(all_publications, columns=["date", "title", "url"])   # name the 3 columns
print(bdf_publications.shape)                  # (rows, columns)
bdf_publications.head(1)                       # the first row
(12, 3)
date title url
0 13th of October 2026 Why don’t fuel prices always strictly follow global oil prices? https://www.banque-france.fr/en/publications-and-statistics/…
bdf_publications.to_csv("bdf_publications.csv", index=False)   # save it as a CSV file (plain text)

A CSV is a plain text file that every tool opens. Unlike an Excel cell (32,767 characters at most), it can hold the full texts we will collect later.

Two problems remaining

However, we are not done. What’s next? Discuss.

What did we retrieve?

  • The 12 latest publications (page 1 of the list)
  • For each: its date, title and link
  • All in a table, saved as a CSV

What is missing?

  1. Problem 1: the other 106 pages of the list
  2. Problem 2: the content of each publication (full text, PDF), which is not on the list page

Let’s fix them one at a time!

D. A loop in a loop (in a loop)

Problem 1. Not 1, but 107 pages

Our previous code only retrieved the 12 latest publications

  • At the bottom of the page, notice the “Next page” button…
  • Let’s look at the URL when we paginate
https://www.banque-france.fr/en/publications-and-statistics/publications?page=0     ← page 1 (12 publications)
https://www.banque-france.fr/en/publications-and-statistics/publications?page=1     ← page 2 (12 more)
https://www.banque-france.fr/en/publications-and-statistics/publications?page=2     ← page 3
...
https://www.banque-france.fr/en/publications-and-statistics/publications?page=106   ← the last page

We must find a way to repeat our code for every page

The plan: a loop inside a loop

Our code for one page (the inner loop):

Outer loop: for page_number in range(107): → download and parse each list page, ?page=0, ?page=1… → all_cards

for card in all_cards:                                       # INNER loop: the 12 publications of ONE page
    publication_url = "https://www.banque-france.fr" + card["href"]
    publication_title = card.find("h3").text.strip()
    publication_date = card.find("small").text.strip()
    all_publications.append((publication_date, publication_title, publication_url))
  • Every publication, from every page, goes into the same list all_publications → pd.DataFrame → CSV

The code for one page does not change: we put it inside an outer loop over the pages.

The pagination loop

Only the outer loop for now: download and parse each list page.

import time

for page_number in range(107):                                     # ALL the pages: 107 requests...
    page_url = list_url + "?page=" + str(page_number)              # ...?page=0, ?page=1, ?page=2
    raw_response = requests.get(page_url, headers=my_identity)
    list_page = BeautifulSoup(raw_response.text, "html.parser")
    
# DOES IT WORK? We can try if on each page, we do find 12 publications! 

    all_cards = list_page.find_all("a", {"class": "card"})         # the 12 cards of this page
    print(page_number, len(all_cards))
    time.sleep(1)                                                  # be polite: wait between two pages
import time

for page_number in range(3):                                       # TEST: only pages 0, 1, 2
    page_url = list_url + "?page=" + str(page_number)              # ...?page=0, ?page=1, ?page=2
    raw_response = requests.get(page_url, headers=my_identity)
    list_page = BeautifulSoup(raw_response.text, "html.parser")
    
# DOES IT WORK? We can try if on each page, we do find 12 publications! 

    all_cards = list_page.find_all("a", {"class": "card"})         # the 12 cards of this page
    print(page_number, len(all_cards))
    time.sleep(1)                                                  # be polite: wait between two pages
0 12
1 12
2 12

The two loops

import time

all_publications = []                                              # created ONCE, before the loops
for page_number in range(3):                                       # OUTER LOOP over the list pages: 0, 1, 2
    print("Page", page_number)
    raw_response = requests.get(list_url + "?page=" + str(page_number), headers=my_identity)
    list_page = BeautifulSoup(raw_response.text, "html.parser")

    for card in list_page.find_all("a", {"class": "card"}):        # INNER LOOP: same code as on one page
        publication_title = card.find("h3").text.strip()
        publication_url = "https://www.banque-france.fr" + card["href"]
        publication_date = card.find("small").text.strip()
        all_publications.append((publication_date, publication_title, publication_url))

    time.sleep(1)                                                  # be polite: wait between two pages

bdf_publications = pd.DataFrame(all_publications, columns=["date", "title", "url"])
bdf_publications.to_csv("bdf_publications.csv", index=False)
print(bdf_publications.shape)                                      # (36, 3)

Problem 2. The text is on another page

We now have the date, title and link of every publication… but not their text

  • The text is on each publication’s own page: we must open every link
https://www.banque-france.fr/en/publications-and-statistics/publications/why-dont-fuel-prices-...   ← its text
https://www.banque-france.fr/en/publications-and-statistics/publications/increasing-the-...        ← its text
...                                                                                               ← one page per publication
  • So: one more loop, over all the links we collected

Full texts, step 1: test on one publication

first_url = bdf_publications["url"][0]                               # the first link of our table
raw_response = requests.get(first_url, headers=my_identity)
publication_page = BeautifulSoup(raw_response.text, "html.parser")

text_block = publication_page.find("div", {"class": "field--name-field-espaces2-paragraphes"})   # the text
print(text_block.text.strip()[:200])                                 # its first 200 characters
Chart 1: Fuel prices (including VAT) in France and global oil prices (Brent) since January 2026 (in euro per litre on the left, in euro per barrel on the right) ...

Full texts, step 2: the third loop

all_texts = []
for publication_url in bdf_publications["url"]:                      # LOOP 3: every link of the table
    raw_response = requests.get(publication_url, headers=my_identity)
    publication_page = BeautifulSoup(raw_response.text, "html.parser")
    text_block = publication_page.find("div", {"class": "field--name-field-espaces2-paragraphes"})
    if text_block is None:                                           # no text? keep an empty value
        all_texts.append(None)
    else:
        all_texts.append(text_block.text.strip())
    time.sleep(1)                                                    # one page per second

bdf_publications["text"] = all_texts                                 # a new column
print(bdf_publications["text"].isna().sum(), "texts missing")        # check!
0 texts missing

E. Dealing with PDFs

How complete are the texts we retrieved?

bdf_publications["text_length"] = bdf_publications["text"].str.len()   # number of characters
bdf_publications[["title", "text_length"]].head(5)
                                               title  text_length
0  Why don't fuel prices always strictly follow g...        10411
1  France's balance of payments and international...         3990
2  Increasing the attractiveness and vibrancy of ...         3457
3  Generative AI: the first changes in the labour...        11368
4  DSGE Models in Central Banks: Progress and Pro...         3981
  • Blog posts: the whole text is on the page (about 10,000 characters)
  • Working papers, bulletins, reports: only a summary (3,000–4,000 characters)… the full text is in the PDF

For most publications, the real text is in the PDF. Let’s look at where the PDF is.

A page with only a summary

Example: the working paper Monetary Policy and the Wealth Distribution (24th of September 2026)

  • On the page: only a non-technical summary (about 5,000 characters)
  • The paper itself: a PDF (2.3 MB), behind the “Download” button

Our scraper only captured the summary!

To get the whole text: find the PDF link, download the file, then read it with Python.

Saving a PDF from a URL

  • A link to a PDF is a URL like any other: requests can save it to your disk
  • But a PDF is not text: use .content (the raw bytes) and write it with "wb" (write bytes)
import requests

pdf_url = "https://www.ecb.europa.eu/pub/pdf/annex/ECB_2022_17_f_sign.en.pdf"
raw_response = requests.get(pdf_url, headers=my_identity)   # download the file
with open("ecb_report.pdf", "wb") as output_file:       # open a new file, to write bytes
    output_file.write(raw_response.content)             # .content: bytes, not text


This downloads the PDF, but we then need to read its content to add it to our dataframe!

Opening a PDF in Python

  • pypdf: a free, pure-Python PDF library (split, merge, crop… and extract text)
from pypdf import PdfReader                              # in Colab: !pip install pypdf

pdf_reader = PdfReader("ecb_report.pdf")                 # open the PDF
pdf_text = ""                                            # an empty text
for pdf_page in pdf_reader.pages:                        # for each page...
    pdf_text = pdf_text + pdf_page.extract_text() + "\n" # ...add its text

print(len(pdf_reader.pages), "pages")
print(pdf_text[:1000])


Please note two limits:

  • Extraction is lossy: tables, columns, footnotes, headers (→ medium bias)
  • Scanned documents need OCR first (e.g. Tesseract)

Final boss exercise

Enrich the table bdf_publications (the 36 publications of the first 3 pages) with the full PDFs.

Add two new columns, and keep the summaries (the text column):

  • pdf_file: the name of the downloaded PDF file (e.g. WP1064.pdf)
  • pdf_content: the text of the PDF
  1. Loop over the url column: on each page, find the “Download” button (class card-download)
  2. Download the PDF into a folder bdf_pdfs (requests, .content, "wb"), keep its file name
  3. Read it with pypdf: all the pages, in one text
  4. No PDF? None in both columns. Then save the table (to_csv)

Conclusion

Good practices

  1. Test small, then loop: 3 pages, then 10, then all
  2. Check each answer (status_code, number of items found, missing values): sites change, your code will break
  3. Be polite: time.sleep, an honest User-Agent, the terms of use and robots.txt
  4. Save as you go: raw pages and PDFs, a CSV every N pages
  5. Document the collection (date, URLs, code) and share the code and URLs, not always the texts (copyright, GDPR for personal data)

To go further

Resources

Other packages you may need

  • Selenium, Playwright: a real browser (JavaScript, logins)
  • Scrapy: large crawls (many pages, retries)
  • pandas.read_html: HTML tables in one line
  • pdfplumber, PyMuPDF: PDFs;
  • Tesseract: OCR

References

Algan, Y., Renault, T., & Subtil, H. (2025). La Fièvre parlementaire: Ce monde où l’on catche! CEPREMAP.
Atalay, E., Phongthiengtham, P., Sotelo, S., & Tannenbaum, D. (2020). The Evolution of Work in the United States. American Economic Journal: Applied Economics, 12(2), 1–34. https://doi.org/10.1257/app.20190070
Baker, S. R., Bloom, N., & Davis, S. J. (2016). Measuring Economic Policy Uncertainty*. The Quarterly Journal of Economics, 131(4), 1593–1636. https://doi.org/10.1093/qje/qjw024
BIS. (2024). Central bankers’ speeches. Bank of International Settlements.
Bonaparte, Y. (2026). Independence under pressure: Quantifying presidential influence on the federal Reserve. Finance Research Letters, 102, 110042. https://doi.org/10.1016/j.frl.2026.110042
Czibor, E., Jimenez-Gomez, D., & List, J. A. (2019). The Dozen Things Experimental Economists Should Do (More of). Southern Economic Journal, 86(2), 371–432. https://doi.org/10.1002/soej.12392
ECB. (2019). Speeches dataset. In ECB. https://www.ecb.europa.eu/press/key/html/downloads.en.html.
Garg, P., & Fetzer, T. (2025). Causal Claims in Economics (arXiv:2501.06873). arXiv. https://doi.org/10.48550/arXiv.2501.06873
Gorodnichenko, Y., Pham, T., & Talavera, O. (2023). The Voice of Monetary Policy. American Economic Review, 113(2), 548–584. https://doi.org/10.1257/aer.20220129
Grimmer, J., Roberts, M. E., & Stewart, B. M. (2022). Text as data: A new framework for machine learning and the social sciences. Princeton University Press.
Hansen, S., McMahon, M., & Prat, A. (2018). Transparency and Deliberation Within the FOMC: A Computational Linguistics Approach*. The Quarterly Journal of Economics, 133(2), 801–870. https://doi.org/10.1093/qje/qjx045
Herbert, S., Kingi, H., Stanchi, F., & Vilhuber, L. (2021). The reproducibility of economics research: A case study.
King, G., Pan, J., & Roberts, M. E. (2013). How Censorship in China Allows Government Criticism but Silences Collective Expression. American Political Science Review, 107(2), 326–343. https://doi.org/10.1017/S0003055413000014
Nagaraj, A., Stipanicic, F., & Tranchero, M. (2025). Restricted-Access Data in Economics: Adoption, Diffusion, and Impact of US Census Bureau’s Microdata. Harvard Data Science Review, (Special Issue 6).
Nagaraj, A., & Tranchero, M. (2023). How Does Data Access Shape Science? The Impact of Federal Statistical Research Data Centers on Economics Research (Working {{Paper}} No. 31372). National Bureau of Economic Research. https://doi.org/10.3386/w31372
Schonhardt-Bailey, Cheryl. (2013). Deliberating American Monetary Policy.

Thank you!

jerome.deyris@univ-paris1.fr

Appendix: final boss, solution

Final boss, solution (1/2): the loop

import os
from pypdf import PdfReader

os.makedirs("bdf_pdfs", exist_ok=True)                                  # a folder for the PDFs
all_pdf_files = []                                                      # one value per publication
all_pdf_contents = []
for publication_url in bdf_publications["url"]:                         # the 36 publications
    raw_response = requests.get(publication_url, headers=my_identity)
    publication_page = BeautifulSoup(raw_response.text, "html.parser")
    pdf_button = publication_page.find("a", {"class": "card-download"})  # the "Download" button

    if pdf_button is None:                                              # no PDF: empty values
        all_pdf_files.append(None)
        all_pdf_contents.append(None)
    else:
        pdf_url = "https://www.banque-france.fr" + pdf_button["href"]   # relative: add the site
        pdf_file_name = pdf_url.split("/")[-1]                          # e.g. WP1064.pdf
        raw_pdf_response = requests.get(pdf_url, headers=my_identity)
        with open("bdf_pdfs/" + pdf_file_name, "wb") as output_file:    # save the raw file
            output_file.write(raw_pdf_response.content)

        pdf_reader = PdfReader("bdf_pdfs/" + pdf_file_name)             # open it
        pdf_content = ""
        for pdf_page in pdf_reader.pages:                                 # each page...
            pdf_content = pdf_content + pdf_page.extract_text() + "\n"  # ...add its text
        all_pdf_files.append(pdf_file_name)
        all_pdf_contents.append(pdf_content)
    time.sleep(1)                                                       # be polite

Final boss, solution (2/2): the new columns

bdf_publications["pdf_file"] = all_pdf_files                            # two new columns
bdf_publications["pdf_content"] = all_pdf_contents
bdf_publications.to_csv("bdf_publications.csv", index=False)            # save the enriched table

print(bdf_publications["pdf_file"].isna().sum(), "PDFs missing")        # check!
bdf_publications[["title", "text", "pdf_file", "pdf_content"]].head()
2 PDFs missing
  • The table now holds both: the web page text (often a summary) and the pdf_content (the full paper)