14 items tagged “concurrency”
2024
This month in Servo: parallel tables and more (via) New in Servo:
Parallel table layout is now enabled (@mrobinson, #32477), spreading the work for laying out rows and their columns over all available CPU cores. This change is a great example of the strengths of Rayon and the opportunistic parallelism in Servo's layout engine.
The commit landing the change is quite short, and much of the work is done by refactoring the code to use .par_iter().enumerate().map(...) - par_iter() is the Rayon method that allows parallel iteration over a collection using multiple threads, hence multiple CPU cores.
Free-threaded CPython is ready to experiment with! The Python 3.13 beta releases that include a "free-threaded" version that removes the GIL are now available to test! A team from Quansight Labs, home of the PyData core team, just launched py-free-threading.github.io to help document the new builds and track compatibility with Python's larger ecosystem.
Free-threading mode will not be enabled in Python installations by default. You can install special builds that have the option enabled today - I used the macOS installer and, after enabling the new build in the "Customize" panel in the installer, ended up with a /usr/local/bin/python3.13t binary which shows "Python 3.13.0b3 experimental free-threading build" when I run it.
Here's my TIL describing my experiments so far installing and running the 3.13 beta on macOS, which also includes a correction to an embarrassing bug that Claude introduced but I failed to catch!
2023
Real Multithreading is Coming to Python—Learn How You Can Use It Now (via) Martin Heinz provides a detailed tutorial on trying out the new Per-Interpreter GIL feature that’s landing in Python 3.12, which allows Python code to run concurrently in multiple threads by spawning separate sub-interpreters, each with their own dedicated GIL.
It’s not an easy feature to play with yet! First you need to compile Python yourself, and then use APIs that are generally only available to C code (but should hopefully become available to Python code itself in Python 3.13).
Martin’s workaround for this is ingenious: it turns out the Python test.support package provides utility functions to help write tests against interpreters, and Martin shows how to abuse this module to launch, run and cleanup interpreters using regular Python code.
He also demonstrates test.support.interpreters.create_channel(), which can be used to create channels with receiver and sender ends, somewhat similar to Go.
2022
PEP 554 – Multiple Interpreters in the Stdlib: Shared data (via) Python 3.12 hopes to introduce multiple interpreters as part of the Python standard library, so Python code will be able to launch subinterpreters, each with their own independent GIL. This will allow Python code to execute on multiple CPU cores at the same time while ensuring existing code (and C modules) that rely on the GIL continue to work.
The obvious question here is how data will be shared between those interpreters. This PEP proposes a channels mechanism, where channels can be used to send just basic Python types between interpreters: None, bytes, str, int and channels themselves (I wonder why not floats?)
2021
The GIL and its effects on Python multithreading (via) Victor Skvortsov presents the most in-depth explanation of the Python Global Interpreter Lock I’ve seen anywhere. I learned a ton from reading this.
2012
Can Scala gain wider usage than Java any time soon?
No, because Scala is harder to master than Java.
[... 54 words]What server do I need to handle 1000+ users simultaneously while they can post messages, upload pictures, and other similar stuff on a website based on PHP and mySQL?
You don’t need to handle 1,000 users simultaneously: you need to build something and ship it and start the process of discovering what you can build that will attract that many users. Seriously: don’t even start worrying about that kind of scale until you know you’re going to need it.
[... 138 words]2010
Distributed lock on top of memcached. A simple Python context manager (taking advantage of the with statement) that implements a distributed lock using memcached to store lock state: “memcached_lock can be used to ensure that some global data is only updated by one server”. Redis would work well for this kind of thing as well.
2009
dustin’s gomemcached (via) A memcached server written in Go, an experiment by memcached maintainer Dustin Sallings.
The Go Programming Language. A brand new systems programming language, designed by Robert Griesemer and Unix/Plan 9 veterans Rob Pike and Ken Thompson and funded by Google. Concurrency is supported by lightweight communicating processes called goroutines. “It feels like a dynamic language but has the speed and safety of a static language.”
Testing Django Views for Concurrency Issues. Neat decorator for executing a Django view under high concurrency in your unit tests, to help spot errors caused by database race conditions that should be executed inside a transaction.
A Curious Course on Coroutines and Concurrency. David Beazley’s sequel to last year’s mind-expanding “Generator Tricks for System Programmers”, perfect for if you’ve ever puzzled over what exactly you can use Python’s generator-based coroutine support for.
django-springsteen and Distributed Search. Will Larson’s Django search library currently just talks to Yahoo! BOSS, but is designed to be extensible for other external search services. Interestingly, it uses threads to fire off several HTTP requests in parallel from within the Django view.
2007
Mac OS X Leopard: Multicore. “... NSOperation, a breakthrough new API that optimizes applications for the world of multicore processing.”