Data Collection Process

1. Collecting Player Stats

Player statistics were scraped using Selenium and BeautifulSoup. Selenium was used to handle dynamic content on the FBref website. For example, the following code was used to scrape data for the 2020-2021 La Liga season:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
import pandas as pd

# URL for La Liga 2020-2021 player stats
url = 'https://fbref.com/en/comps/12/2020-2021/stats/2020-2021-La-Liga-Stats'

# Setup Selenium with headless Chrome
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(options=chrome_options)

# Open the URL and extract the page source
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'html.parser')

# Find the player stats table by its ID
table = soup.find('table', {'id': 'stats_standard'})
df = pd.read_html(str(table))[0]  # Parse the table into a DataFrame

# Save the data to a CSV file
df.to_csv('la_liga_2020_2021_player_stats.csv', index=False)

This process was repeated for the 2017-2018, 2018-2019, 2019-2020, and 2020-2021 seasons, resulting in four separate files. The files were later combined into a single dataset: Player_stats.csv.

2. Collecting Team Stats

Team statistics were collected using BeautifulSoup and requests. We fetched URLs for each team, scraped relevant tables, and combined the data. Here’s an example of extracting team stats for the 2020-2021 La Liga season:

from bs4 import BeautifulSoup
import requests

# Fetch team URLs
response = requests.get('https://fbref.com/en/comps/12/2020-2021/2020-2021-La-Liga-Stats')
soup = BeautifulSoup(response.text, 'html.parser')
team_links = [f"https://fbref.com{a['href']}" for a in soup.find_all('a', href=True) if '/squads/' in a['href']]

# Scrape "Scores & Fixtures" and "Shooting" tables for each team
for team_url in team_links:
    team_page = requests.get(team_url)
    team_soup = BeautifulSoup(team_page.text, 'html.parser')
    scores_fixtures = pd.read_html(str(team_soup), match="Scores & Fixtures")[0]
    # Process and save the data...

This process was also repeated for four seasons and the resulting files were merged into Team_Stats.csv.

3. Collecting Additional Team Data

In addition to the initial Team_Stats.csv dataset, another dataset named Team Dataset 2 (originally cleaned_dataset_no_index.csv) was created. This dataset aggregates and enhances the initial team statistics with additional cleaned and processed metrics. It combines player and team-level information for a more comprehensive analysis.

import pandas as pd

# Load the cleaned dataset
cleaned_data = pd.read_csv('cleaned_dataset_no_index.csv')

# Display the first few rows of the dataset
print(cleaned_data.head())

The Team Dataset 2 was used alongside the original Team_Stats.csv for advanced team-level analysis, providing deeper insights into team dynamics and player contributions.