Web Scraping and API Integration
After mastering the basics of Python, it’s time to unlock its power in real-world applications. In Stage 2, we dive into web scraping and API integration—two essential skills that allow you to collect data, automate research, and connect your Python apps to the internet. Whether you want to track product prices, pull live weather updates, or fetch stock data, this stage will show you how.
1. What Is Web Scraping and Why Is It Important?
Web scraping is the process of using code to extract data from websites automatically. With Python, you can gather massive amounts of information in minutes that would otherwise take hours to copy manually.
Use cases:
- Monitor prices on e-commerce platforms (e.g., Amazon, Tokopedia)
- Collect job listings from multiple websites
- Extract news headlines or articles for analysis
Legal Tip: Always check a site’s robots.txt and terms of service before scraping.
2. Setting Up Web Scraping with BeautifulSoup
Let’s start by installing the libraries:
pip install requests beautifulsoup4
Now try scraping article titles from a news site:
import requests
from bs4 import BeautifulSoup
url = 'https://news.ycombinator.com/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
for title in soup.find_all('a', class_='storylink'):
print(title.text)
This simple script will display the top headlines from Hacker News.
3. Scraping Dynamic Sites with Selenium
Some websites use JavaScript to load content dynamically. In this case, use Selenium:
pip install selenium
Example with Selenium:
from selenium import webdriver
from time import sleep
driver = webdriver.Chrome()
driver.get('https://quotes.toscrape.com/js/')
sleep(2)
quotes = driver.find_elements_by_class_name('text')
for quote in quotes:
print(quote.text)
driver.quit()
Selenium is slower but more powerful for modern, JS-heavy websites.
4. Accessing Data via Public APIs
Many services offer structured data through APIs (Application Programming Interfaces), which are more efficient and ethical than scraping.
Example: Fetching weather data from OpenWeatherMap
import requests
api_key = 'your_api_key'
city = 'Jakarta'
url = f'https://api.openweathermap.org/data/2.5/weather?q={city}&appid={api_key}&units=metric'
response = requests.get(url)
data = response.json()
print(f"Temperature in {city}: {data['main']['temp']}°C")
Using APIs allows you to integrate live data into your apps with minimal effort.
5. Combine Web Scraping + API for Smart Automation
Once you’ve mastered both skills, start combining them:
- Scrape trending topics → Feed them into an AI model
- Monitor competitors’ prices → Send alerts via Telegram API
- Fetch data from multiple APIs → Store and analyze it with Pandas
These automations save time, increase efficiency, and open doors to predictive analysis.
Conclusion: You’re Now Connected to the Web!
With web scraping and API integration under your belt, you’re now equipped to interact with the vast ecosystem of online data. These tools turn Python from a scripting language into a powerful automation engine.
👉 Next up: Stage 3 — Data Analysis and Visualization for Beginners.
