Multithreading vs Multiprocessing in Python

Overview

Concurrency in programming allows multiple tasks to run simultaneously, improving performance and efficiency. In Python, multithreading and multiprocessing are two primary approaches to achieve concurrency. While they may seem similar, they serve distinct purposes and are suited for different tasks. This blog dives into the differences, use cases, and real-world examples of multithreading and multiprocessing in Python, helping you choose the right tool for your project.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Multithreading

Multithreading involves running multiple threads within the same process. A thread is a lightweight unit of execution that shares the same memory space as other threads in the process. Python’s threading module enables multithreading, but the Global Interpreter Lock (GIL) in CPython limits true parallel execution for CPU-bound tasks. This makes multithreading ideal for I/O-bound tasks, where threads spend time waiting for external resources like network responses or file operations.

Key Characteristics of Multithreading

Threads share memory, reducing overhead but requiring synchronization (e.g., locks) to avoid race conditions.
Best for I/O-bound tasks like web scraping or downloading files.
Limited by the GIL for CPU-bound tasks, preventing full CPU utilization.

Multiprocessing

Multiprocessing creates multiple independent processes, each with its own memory space and Python interpreter. The multiprocessing module in Python allows true parallelism by bypassing the GIL, making it suitable for CPU-bound tasks like data processing or mathematical computations. However, inter-process communication (IPC) and process creation introduce higher overhead than threads.

Key Characteristics of Multiprocessing

Processes are isolated, eliminating the need for locks but requiring IPC mechanisms like pipes or queues.
It is ideal for CPU-bound tasks like image processing or machine learning model training.
Higher memory and startup overhead due to separate memory spaces.

Multithreading vs Multiprocessing: A Comparison

To understand when to use each approach, let’s compare them across key dimensions:

table

Example 1: Web Scraping (Multithreading)

Imagine you’re building a tool to scrape product prices from multiple e-commerce websites. This is an I/O-bound task because most of the time is spent waiting for HTTP responses. Multithreading shines here, as threads can handle multiple requests concurrently while sharing the same memory for storing results.

Here’s a simplified Python script using the threading module to scrape multiple URLs:

import threading
import requests
from queue import Queue
urls = ["https://example.com/page1", "https://example.com/page2"]
results = Queue()
def scrape_url(url):
    response = requests.get(url)
    results.put((url, response.status_code))
threads = []
for url in urls:
    t = threading.Thread(target=scrape_url, args=(url,))
    threads.append(t)
    t.start()
for t in threads:
    t.join()
while not results.empty():
    url, status = results.get()
    print(f"URL: {url}, Status: {status}")

import threading

import requests

from queue import Queue

urls = ["https://example.com/page1", "https://example.com/page2"]

results = Queue()

def scrape_url(url):

response = requests.get(url)

results.put((url, response.status_code))

threads = []

for url in urls:

t = threading.Thread(target=scrape_url, args=(url,))

threads.append(t)

t.start()

for t in threads:

t.join()

while not results.empty():

url, status = results.get()

print(f"URL: {url}, Status: {status}")

In this example, each thread fetches a web page, and the results are collected in a thread-safe queue. Multithreading reduces the total time by overlapping network delays.

multithread

Example 2: Image Processing (Multiprocessing)

Suppose you’re developing an application to resize thousands of images for a photo gallery. Image resizing is a CPU-bound task, as it involves intensive computations. Multiprocessing is ideal here, as each process can utilize a separate CPU core to process images in parallel.

Here’s a sample script using the multiprocessing module:

from multiprocessing import Pool
from PIL import Image
import os
def resize_image(image_path):
    img = Image.open(image_path)
    img = img.resize((100, 100))
    img.save(f"resized_{os.path.basename(image_path)}")
    return f"Processed {image_path}"
image_paths = ["image1.jpg", "image2.jpg", "image3.jpg"]
if __name__ == "__main__":
    with Pool(processes=3) as pool:
        results = pool.map(resize_image, image_paths)
    for result in results:
        print(result)

from multiprocessing import Pool

from PIL import Image

import os

def resize_image(image_path):

img = Image.open(image_path)

img = img.resize((100, 100))

img.save(f"resized_{os.path.basename(image_path)}")

return f"Processed {image_path}"

image_paths = ["image1.jpg", "image2.jpg", "image3.jpg"]

if __name__ == "__main__":

with Pool(processes=3) as pool:

results = pool.map(resize_image, image_paths)

for result in results:

print(result)

In this script, the Pool class distributes image paths across multiple processes, each resizing an image independently. This approach maximizes CPU utilization and speeds up the task.

multithread2

Conclusion

Multithreading and multiprocessing are powerful tools for concurrency in Python, but they cater to different needs. Multithreading excels in I/O-bound scenarios like web scraping, where waiting is the bottleneck.

Multiprocessing shines in CPU-bound tasks like image processing, leveraging multiple cores for true parallelism. Understanding their strengths, limitations, and real-world applications enables you to make informed decisions to optimize your Python programs.

Drop a query if you have any questions regarding Multithreading or multiprocessing and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.