Web Scraping with BeautifulSoup: Extracting Article Titles and Links

Web scraping is a powerful technique for extracting data from websites. BeautifulSoup is a Python library designed to make this process easier by parsing HTML and XML documents. This challenge will guide you through using BeautifulSoup to extract article titles and their corresponding links from a given webpage.

Problem Description

You are tasked with writing a Python script that uses BeautifulSoup to scrape a specified webpage and extract all article titles and their associated links. The script should take a URL as input, fetch the HTML content of the page, parse it using BeautifulSoup, and then identify all <a> tags within <h2> tags (representing article titles and links). The script should then print the title and link for each article found in a user-friendly format.

Key Requirements:

• Fetch HTML: The script must be able to fetch the HTML content of the provided URL.
• Parse HTML: The script must use BeautifulSoup to parse the fetched HTML content.
• Identify Articles: The script must locate all <h2> tags containing <a> tags within them. These are assumed to represent article titles and links.
• Extract Data: The script must extract the text content of the <h2> tag (the article title) and the href attribute of the <a> tag (the article link).
• Output: The script must print the extracted title and link for each article in the format: "Title: [Article Title]\nLink: [Article Link]\n".

Expected Behavior:

The script should gracefully handle cases where the webpage is unavailable or the expected HTML structure is not found. If no articles are found, it should print a message indicating that no articles were found.

Edge Cases to Consider:

• Invalid URL: The provided URL might be invalid or unreachable.
• Missing HTML Structure: The webpage might not contain the expected <h2> and <a> tags.
• Relative Links: The href attribute might contain relative links instead of absolute URLs. (For simplicity, assume absolute URLs are used in this challenge).
• Encoding Issues: The webpage might use a different character encoding.

Examples

Example 1:

Input: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
Output:
Title: Beautiful Soup 4
Link: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
Title: Installation
Link: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installation
Title: Getting Started
Link: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#getting-started
... (and so on for all articles on the page)

Explanation: The script successfully fetches the BeautifulSoup documentation page, parses the HTML, and extracts the titles and links from the <h2> tags containing <a> tags.

Example 2:

Input: https://www.example.com (a simple webpage with no article structure)
Output:
No articles found.

Explanation: The script fetches the example.com page, but since it doesn't contain the expected <h2> and <a> tags, it correctly reports that no articles were found.

Example 3:

Input: https://invalid-url (an invalid URL)
Output:
Error fetching URL. Please check the URL and try again.

Explanation: The script attempts to fetch the invalid URL and catches the resulting exception, printing an error message.

Constraints

• URL Length: The input URL should be no longer than 2048 characters.
• HTML Size: The HTML content of the webpage should be less than 10MB.
• Error Handling: The script must handle potential errors such as invalid URLs and network connection issues gracefully.
• Libraries: You are allowed to use only the requests and BeautifulSoup4 libraries.

Notes

• You will need to install the requests and beautifulsoup4 libraries before running the script: pip install requests beautifulsoup4
• Consider using a try-except block to handle potential errors during the URL fetching process.
• The specific HTML structure of the target webpage is assumed to be consistent. This challenge focuses on the BeautifulSoup parsing aspect, not on robustly handling varying HTML structures.
• Focus on extracting the text from the <h2> tag and the href attribute from the <a> tag. You don't need to perform any further processing on the extracted data.