Multithreading vs Multiprocessing in Python

Overview

Concurrency in programming allows multiple tasks to run simultaneously, improving performance and efficiency. In Python, multithreading and multiprocessing are two primary approaches to achieve concurrency. While they may seem similar, they serve distinct purposes and are suited for different tasks. This blog dives into the differences, use cases, and real-world examples of multithreading and multiprocessing in Python, helping you choose the right tool for your project.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Multithreading

Multithreading involves running multiple threads within the same process. A thread is a lightweight unit of execution that shares the same memory space as other threads in the process. Python’s threading module enables multithreading, but the Global Interpreter Lock (GIL) in CPython limits true parallel execution for CPU-bound tasks. This makes multithreading ideal for I/O-bound tasks, where threads spend time waiting for external resources like network responses or file operations.

Key Characteristics of Multithreading

Threads share memory, reducing overhead but requiring synchronization (e.g., locks) to avoid race conditions.
Best for I/O-bound tasks like web scraping or downloading files.
Limited by the GIL for CPU-bound tasks, preventing full CPU utilization.

Multiprocessing

Multiprocessing creates multiple independent processes, each with its own memory space and Python interpreter. The multiprocessing module in Python allows true parallelism by bypassing the GIL, making it suitable for CPU-bound tasks like data processing or mathematical computations. However, inter-process communication (IPC) and process creation introduce higher overhead than threads.

Key Characteristics of Multiprocessing

Processes are isolated, eliminating the need for locks but requiring IPC mechanisms like pipes or queues.
It is ideal for CPU-bound tasks like image processing or machine learning model training.
Higher memory and startup overhead due to separate memory spaces.

Multithreading vs Multiprocessing: A Comparison

To understand when to use each approach, let’s compare them across key dimensions:

table

Example 1: Web Scraping (Multithreading)

Imagine you’re building a tool to scrape product prices from multiple e-commerce websites. This is an I/O-bound task because most of the time is spent waiting for HTTP responses. Multithreading shines here, as threads can handle multiple requests concurrently while sharing the same memory for storing results.

Here’s a simplified Python script using the threading module to scrape multiple URLs:

import threading
import requests
from queue import Queue
urls = ["https://example.com/page1", "https://example.com/page2"]
results = Queue()
def scrape_url(url):
    response = requests.get(url)
    results.put((url, response.status_code))
threads = []
for url in urls:
    t = threading.Thread(target=scrape_url, args=(url,))
    threads.append(t)
    t.start()
for t in threads:
    t.join()
while not results.empty():
    url, status = results.get()
    print(f"URL: {url}, Status: {status}")

import threading

import requests

from queue import Queue

urls = ["https://example.com/page1", "https://example.com/page2"]

results = Queue()

def scrape_url(url):

response = requests.get(url)

results.put((url, response.status_code))

threads = []

for url in urls:

t = threading.Thread(target=scrape_url, args=(url,))

threads.append(t)

t.start()

for t in threads:

t.join()

while not results.empty():

url, status = results.get()

print(f"URL: {url}, Status: {status}")

In this example, each thread fetches a web page, and the results are collected in a thread-safe queue. Multithreading reduces the total time by overlapping network delays.

multithread

Example 2: Image Processing (Multiprocessing)

Suppose you’re developing an application to resize thousands of images for a photo gallery. Image resizing is a CPU-bound task, as it involves intensive computations. Multiprocessing is ideal here, as each process can utilize a separate CPU core to process images in parallel.

Here’s a sample script using the multiprocessing module:

from multiprocessing import Pool
from PIL import Image
import os
def resize_image(image_path):
    img = Image.open(image_path)
    img = img.resize((100, 100))
    img.save(f"resized_{os.path.basename(image_path)}")
    return f"Processed {image_path}"
image_paths = ["image1.jpg", "image2.jpg", "image3.jpg"]
if __name__ == "__main__":
    with Pool(processes=3) as pool:
        results = pool.map(resize_image, image_paths)
    for result in results:
        print(result)

from multiprocessing import Pool

from PIL import Image

import os

def resize_image(image_path):

img = Image.open(image_path)

img = img.resize((100, 100))

img.save(f"resized_{os.path.basename(image_path)}")

return f"Processed {image_path}"

image_paths = ["image1.jpg", "image2.jpg", "image3.jpg"]

if __name__ == "__main__":

with Pool(processes=3) as pool:

results = pool.map(resize_image, image_paths)

for result in results:

print(result)

In this script, the Pool class distributes image paths across multiple processes, each resizing an image independently. This approach maximizes CPU utilization and speeds up the task.

multithread2

Conclusion

Multithreading and multiprocessing are powerful tools for concurrency in Python, but they cater to different needs. Multithreading excels in I/O-bound scenarios like web scraping, where waiting is the bottleneck.

Multiprocessing shines in CPU-bound tasks like image processing, leveraging multiple cores for true parallelism. Understanding their strengths, limitations, and real-world applications enables you to make informed decisions to optimize your Python programs.

Drop a query if you have any questions regarding Multithreading or multiprocessing and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.

FAQs

1. How can I share data between processes in multiprocessing?

ANS: – You can use multiprocessing.Queue, multiprocessing.Pipe, or shared memory, to communicate between processes.

2. Is multithreading faster than multiprocessing?

ANS: – It depends on the task. Multithreading is faster for I/O-bound tasks, while multiprocessing is better for CPU-intensive operations.

WRITTEN BY Aiswarya Sahoo

Aiswarya is a Data Engineer at CloudThat, with a strong focus on designing and building scalable data pipelines and cloud-based solutions. He is skilled in working with big data tools and technologies such as PySpark, AWS Glue, AWS Lambda, Amazon S3, and Amazon RDS. Aiswarya has a solid understanding of data processing, ETL workflows, and optimizing data systems for performance and reliability. In his free time, he enjoys exploring advancements in cloud computing, experimenting with new data tools, and staying updated with industry trends.