In this blog post, we’ll discuss thread-safety, why it’s important and how to write thread-safe django code, especially for bulk operations like management commands. We’ll take a simple example - get or create.
Thread-safety means that our code can be run in multiple threads and behave as expected. The reason that code can be unsafe with regard to threads is because we’ll be manipulating shared memory (e.g. database) from the threads and there’s a chance of a race-condition which will produce unexpected results.
To avoid this, we have the option of using read-write locks, transactions etc.
We’ll look at some simple examples and try to understand these options.
The usual way:
Let’s consider a management command that syncs data from another source (e.g. API,
remote database etc.. The correct way to do this would be to use the built-in
django utility -
Update: Updated the command to run each arg in a thread
class MyThread(Thread): def __init__(self, my_id): super(MyThread, self).__init__(name=my_id) self.my_id = my_id def run(self): instance, created = MyModel.objects.get_or_create(my_id=my_id) print '%s %s' % (instance.id, created) instance.delete() return class Command(BaseCommand): args = '<my_id my_id ...>' help = 'Get or create instace of mymodel with my_id' def handle(self, *args, **options): for my_id in args: thread = MyThread(my_id=my_id) thread.start()
In this command, we’ll be using the command line arg my_id get a MyModel instance if it exists, else, we create one.
Note: we’ll be discarding the object for simplicity.
Now, this management command (let’s call it sync_myapp) is thread-safe because
we’re just using the built-in
get_or_create to do our database call.
To test this, run the command with same arg repeated multiple times:
$> python manage.py sync_myapp 1 1 1 1 1
This will create five threads which will simultaneously try to get or create an instance with my_id = 1
$> 1 True $> 1 False $> 1 False $> 1 False
Note that all the threads are successful, even though they were working on the same database at the same time.
Now, even though get_or_create is the way to go, we might need to customize a
few things which are outside the scope of
get_or_create. For example, let’s
say we need to do something special just before creating a new instance.
Let’s assume our code was:
def run(self): created = False try: instance = MyModel.objects.get(my_id=my_id) except MyModel.DoesNotExist: something_special() instance = MyModel.objects.create(my_id=my_id) created = True instance.delete() return
If we try to test it with the above command, you’ll probably get:
Exception in thread 1: Traceback (most recent call last): .... IntegrityError: duplicate key value violates unique constraint "myapp_mymodel_my_id_key" DETAIL: Key (my_id)=(1) already exists.
This indicates that the try except block isn’t thread-safe.
Let’s try to fix this problem.
Update: I’ve removed the section dealing with locks since it’s only useful when dealing with shared memory in python processes, it’s not applicable to databases.
Using database transactions:
We use django’s built-in
transactions module as follows:
Update: We only need to wrap the create command in a transaction
def run(self): created = False try: instance = MyModel.objects.get(my_id=self.my_id) except MyModel.DoesNotExist: try: something_special() with transaction.commit_on_success(): instance = MyModel.objects.create(my_id=self.my_id) created = True except IntegrityError: instance = MyModel.objects.get(my_id=self.my_id) print '%s %s' % (instance.id, created) instance.delete() return
Now, the transaction will be committed only if there’s no error within the
context block, so we can be sure that only one thread gets the go-ahead for
In addition to the context manager, django also has options for using savepoints, manually commits, rollbacks etc.
If you’re using MySQL, refer to this open issue on problem with
Using database transactions, we can avoid data integrity issues and write thread-safe code which can be run in parallel without any issues.
Thank you for reading the Agiliq blog. This article was written by Javed on Aug 21, 2013 in threads .
You can subscribe ⚛ to our blog.
We love building amazing apps for web and mobile for our clients. If you are looking for development help, contact us today ✉.
Would you like to download 10+ free Django and Python books? Get them here