Writing thread-safe django - get_or_create

By : Javed Khan

In this blog post, we'll discuss thread-safety, why it's important and how to write thread-safe django code, especially for bulk operations like management commands. We'll take a simple example - get or create.

Thread-safety:

Thread-safety means that our code can be run in multiple threads and behave as expected. The reason that code can be unsafe with regard to threads is because we'll be manipulating shared memory (e.g. database) from the threads and there's a chance of a race-condition which will produce unexpected results.

To avoid this, we have the option of using read-write locks, transactions etc.

We'll look at some simple examples and try to understand these options.

The usual way:

Let's consider a management command that syncs data from another source (e.g. API, remote database etc.. The correct way to do this would be to use the built-in django utility - get_or_create:

Update: Updated the command to run each arg in a thread

class MyThread(Thread):

    def __init__(self, my_id):
        super(MyThread, self).__init__(name=my_id)
        self.my_id = my_id

    def run(self):
        instance, created = MyModel.objects.get_or_create(my_id=my_id)
        print '%s %s' % (instance.id, created)
        instance.delete()
        return


class Command(BaseCommand):
    args = '<my_id my_id ...>'
    help = 'Get or create instace of mymodel with my_id'

    def handle(self, *args, **options):
        for my_id in args:
            thread = MyThread(my_id=my_id)
            thread.start()

In this command, we'll be using the command line arg my_id get a MyModel instance if it exists, else, we create one.

Note: we'll be discarding the object for simplicity.

Now, this management command (let's call it sync_myapp) is thread-safe because we're just using the built-in get_or_create to do our database call.

To test this, run the command with same arg repeated multiple times:

$> python manage.py sync_myapp 1 1 1 1 1

This will create five threads which will simultaneously try to get or create an instance with my_id = 1

$> 1 True
$> 1 False
$> 1 False
$> 1 False

Note that all the threads are successful, even though they were working on the same database at the same time.

Now, even though get_or_create is the way to go, we might need to customize a few things which are outside the scope of get_or_create. For example, let's say we need to do something special just before creating a new instance.

The problem:

Let's assume our code was:

def run(self):
    created = False
    try:
        instance = MyModel.objects.get(my_id=my_id)
    except MyModel.DoesNotExist:
        something_special()
        instance = MyModel.objects.create(my_id=my_id)
        created = True
    instance.delete()
    return

If we try to test it with the above command, you'll probably get:

Exception in thread 1:
Traceback (most recent call last):
....
IntegrityError: duplicate key value violates unique constraint "myapp_mymodel_my_id_key"
DETAIL:  Key (my_id)=(1) already exists.

This indicates that the try except block isn't thread-safe.

Let's try to fix this problem.

Update: I've removed the section dealing with locks since it's only useful when dealing with shared memory in python processes, it's not applicable to databases.

Using database transactions:

We use django's built-in transactions module as follows:

Update: We only need to wrap the create command in a transaction

def run(self):
    created = False
    try:
        instance = MyModel.objects.get(my_id=self.my_id)
    except MyModel.DoesNotExist:
            try:
                something_special()
                with transaction.commit_on_success():
                    instance = MyModel.objects.create(my_id=self.my_id)
                created = True
            except IntegrityError:
                instance = MyModel.objects.get(my_id=self.my_id)
    print '%s %s' % (instance.id, created)
    instance.delete()
    return

Now, the transaction will be committed only if there's no error within the context block, so we can be sure that only one thread gets the go-ahead for create call.

In addition to the context manager, django also has options for using savepoints, manually commits, rollbacks etc.

https://docs.djangoproject.com/en/1.5/topics/db/transactions

Caveats:

If you're using MySQL, refer to this open issue on problem with get_or_create:

https://code.djangoproject.com/ticket/13906

Conclusion:

Using database transactions, we can avoid data integrity issues and write thread-safe code which can be run in parallel without any issues.


Related Posts


Can we help you build amazing apps? Contact us today.

Topics : django python threads

Comments

Patryk Zawadzki 21st Aug., 2013

Except that… parallel does not use threads. What you test for is race conditions, not thread safety. Thread safety involves heaps of other problems like shared memory, locking and reentrancy.

commmenttor
Owais Lone

Like Patryk said, this is avoiding race conditions, not thread safety.

>>> Now, this management command (let's call it sync_myapp) is thread-safe because we're just using the built-in get_or_create to do our database call.

get_or_create is built in but it has no way of avoiding race conditions. What it does internally is fetch a row, and insert the row if nothing was fetched. It is the same as loading, catching DoesNotExist and then inserting inside the exception handler block.

Check https://github.com/django/django/blob/f7290581fe2106c08d97215ab93e27cf6b27e100/django/db/models/query.py#L408

To handle the race condition, you need to wrap get_or_create inside managed DB transactions or do some locking in the DB (row-level/table-level; whatever makes sense) itself and not in application code.

commmenttor
Javed Khan

@Patryk thanks, I've removed parallel and added threads for testing it. Yes, this blog post only covers dealing with race conditions.

@Owais There are transactions in place to avoid race conditions, see:

https://github.com/django/django/blob/f7290581fe2106c08d97215ab93e27cf6b27e100/django/db/models/query.py#L449

Though, it looks like it depends on the transaction level. See https://code.djangoproject.com/ticket/13906#comment:38

commmenttor
Owais Lone

Look closely. It is creating savepoints, not managing a transaction. It is only specifying how much to rollback when you (the developer) choose to rollback. You still need to manage transactions.

Read up on savepoints https://docs.djangoproject.com/en/dev/topics/db/transactions/#savepoints

commmenttor
Javed Khan

@Owais,

I should have mentioned I'm testing this on django 1.5 and postgresql 9.1. Here's the test I'm running:

https://gist.github.com/tuxcanfly/6339390

If you comment the get_or_create and uncomment the try..except block, you should see IntegrityErrors

I think, with get_or_create, we don't need to manage the transaction because of django's default commit behavior which commits on save(). In case of an IntegrityError, get_or_create rolls back to the savepoint, so it handles the race condition.

commmenttor
Under floden

Thanks a ton so much for just a incredibly intelligent and also refreshing write-up. Great job!

commmenttor
casey evans

Thread-safe django is very much helpful to us cause we have bulk operation. Thank You for this wonderful information and easy to understand command in your scenario.

commmenttor
Qu'est-ce que la mesure de Ray Ban Wayfarer Lunettes de soleil

Writing thread-safe django - get_or_create - Agiliq Blog | Django web app development

commmenttor
nike air force 1 mid kids

Sneak A Peek At THESE Guys

commmenttor
ray ban pas cher homme

Casque [canal audio original en ligne Zhongguancun] Bluetooth est d&#233;sormais beaucoup de la vie quotidienne des gens parmi le mat&#233;riel indispensable pour &#233;couter de la musique, mais si un kit oreillette Bluetooth est un peu pr&#233;matur&#233; de jeunes gens d&#233;vou&#233;s, et maintenant beaucoup de jeunes amis Dessins comme une exag&#233;ration, d'ajuster les prix &#233;lev&#233;s des produits de grandes casque Bluetooth cool, mais pas pour les exigences de haute qualit&#233;, tel casque Bluetooth est contraire &#224; la notion de bonne qualit&#233; avec un casque Bluetooth, cette fois comme ya Le jour ce n'est pas exag&#233;r&#233;, ce ABH302 accorder plus d'attention &#224; son conception et de performance les utilisateurs de casque Bluetooth sont devenus de bons amis dans tous les jours d'&#233;coute casque Bluetooth de choix de produits de qualit&#233;, ce casque Bluetooth est situ&#233; dans le district de Haidian, Beijing Zhongguancun &#233;lectronique 4920 Artini City II Prix casque de magasin est de 298 yuans, Bloc A, les pr&#233;f&#233;rences de l'utilisateur peuvent sembler bonne &#224; conna&#238;tre la prochaine audition pour stocker le casque o&#249; son fort &#224; la fin. Casque avec microphone con&#231;u pour permettre Les utilisateurs peuvent plus facilement utiliser leurs t&#233;l&#233;phones avec de grandes oreilles sur la taille des boutons de contr&#244;le pour assurer le traitement de l'oreillette, vous pouvez dire que ce soit la conception globale ou la manipulation de l'oreillette en place pour faire tr&#232;s [quote Rechercher] casques offrent &#187;.

commmenttor
cheap nike jerseys

Shop for nfl Jerseys from china, cheap nfl Jerseys for sale with free shipping.

commmenttor
jordan femme 2012 chiffre romain

Basic Silkworm SetupSince silkworms do not drink water, they get their needed moisture from the food they eat, you will need a covered container to raise them in. The container should be almost air tigt to prevent the food from drying out, but have small holes to allow air exchange.

commmenttor
nike fitness ruh&#225;zat

to the end-user, Hardt says. There are many craft retail outlets as well as pottery studios that sell off clay-based pots that you could decorate with your own personal design. " or "Will you go out with me?
nike fitness ruh&#225;zat http://www.visionext.hu/olcsonike.asp?id=965

commmenttor
Hogan Interactive Nere 351234

nowe buty nike 234141 Air Max 1 Femme Pas Cher 113412 nike sko herre 544002 mac eyeshadow all that glitters 512153 Hogan Traditional Uomo 033124 Ballerine Hogan 350323
Hogan Interactive Nere 351234 http://www.madou-ballonvaarten.be/userfiles-madou/IsabelMarantBobby/Hogan_Interactive_Nere_351234.asp

commmenttor
replique sac a main hermes

Love the shoe! What a deal! FAST shipping!

commmenttor
Rivenditori Hogan Torino 200552

free run nike norge 345044 Ducati Oakley Sunglasses 404401 design nike free run 5 345503 nike free roshe run women 214003 Isabel Marant Sebay Suede Ankle Boots 414345
Rivenditori Hogan Torino 200552 http://www.adgar.com/Flash/HoganShoesOnlinen/Rivenditori_Hogan_Torino_200552.asp

commmenttor
Sneakers Isabel Marant

Interactive Hogan buy mac cosmetics cheap Nike Air Max 90 Blanche nike air max
Sneakers Isabel Marant http://www.avatre.it/club/IsabelMarantBobby/Isabel_Marant_2013_341455.asp

commmenttor
nike air max trainers cheap

and you are making $30,000 now at a job you hate that is a good deal. Also like to have cut baits such as crayfish meat. "Within that a financial crisis individuals are seriously keep locate outside equipment, provisional break free of from your importance within hard concrete realities akin to whole life. So stay put and get ready to view the whole world in no time with virtual reality!
nike air max trainers cheap http://www.craftspace.co.uk/shelanu/default.asp?id=59

commmenttor
fit flops outlet

Design dit hus for krystal dekorationer er absolut en sjov aktivitet. Spring design skal vise skinnende og frisk følelse fra sæsonen. I år giver konventionelle krystal dekorationer dagens perspektiv. Her er nogle unikke krystal dekorationer, som du kan bruge i din house.Generate foråret lysere i dit hus med en smuk ferie desk oprettelse. Søndag frokoster ornamenterede sammen med behagelige krystal dekorationer vil være det vigtigste element i selve dekorationer festligheder. Tænk over lyse nuancer til ferie skrivebord. Vælg 2 eller 3 nuancer, der stimulerer forår for eksempel chokolade brun og stor grøn og stole på dem i din ferie placering konfigurationer. Bruge de traditionelle hvide farvede middagsretter og klare cup steward, der spreder lys til ethvert blændende look. Tilføj meget dekorationer som keramik Tilfreds krystal! I enkle ikke-traditionelle toner som lys grålig samt stor røget blå. Du kan også holde tingene let og øge de enkle dekorationer af din ejendom med en mættet bord oprettelse. Opret en iøjnefaldende ferie position oprettelse ved hjælp af en forholdsvis neutral tone middag service og krystal briller. Brug desk tilbehør såsom lysestage stande og dokumentere servietringe i brændkammer eller endda anvendt stål. Tilføj en skinnende pop af farve til denne let oprettelse ved at tilføje et fokus på skinnende gule påskeliljer inde i en hvid-farvet keramik container.Giving dine venner og nære slægtninge giver er halvt fornøjelig af krystal dekorationer sjov. Præsenter dine kære noget, som vil guide skønheden i foråret til deres hjem. Plant containere tillade genindfoersel til vidunderlige påske tid giver. Du kan vælge nem at vedligeholde grimme plantekasser, du kan bruge indendørs. Plastic mel plante opbevaringsbeholdere i flotte farver og designs kunne placeres på skærmen vindueskarme samt i skærmen bokse. Keramik beholdere eller ler mel vokse beholdere er praktiske såvel som øko-venlige påske tid giver. Du kan gave sætteplante pakker indeholdende foråret blomsterfrø for eksempel hyacint, dahlia eller tulipan sammen med smukke plante indikatorer. Farming leverancer vil også være forsigtig krystal dekorationer giver. Landbrug handsker sammen ved at have en dragt vil blive værdsat af familiemedlemmer med en grøn USB. Springtime aktiviteter hjælper dig med at slippe af vinteren og få energisk. Hvis dine venner tendens til at være aktiviteter elskere, vil de helt sikkert elske krystal dekorationer gaver for eksempel tennis poser eller golf jerseys.With disse ideer, kan du sikkert vælge krystal gaver, der vil blive holdt, og valued.Very dekoration har undergået talrige ændringer, nogle ideer og ideer er ekstremt nyt og et par af ejendommen design ideer samt ideer er krystal. Nogle nyere elementer og attraktive artikler er vokset til at være mode i den nuværende æra og nøjagtig samme tid nogle er aftagende sin glød, da den anbefalede idé eller et koncept af boligen stil. Et par af de begreber og faktorer i forbindelse med huset stil og afgørende faktorer fra bopælen design som møbler, har møbler som gardiner, hjem parfumer og belysning som modeller og belysning i tillæg til de lysekroner allerede altid de awesome begreber fra boligen stil og individuelle design.crystal dekorationer handler om at skabe den beboelig plads, der er levende, storslåede, elegant samt komfortable. Hus design er også en metode til at vise identifikation af mennesker om, hvem de har tendens til at være, og hvad de sætter pris på i deres liv. Dekorative ting og et par af ejendommen udsmykning idéer virkelig hjælpe enkeltpersoner til at forbedre og styrke foruden at løfte hele huset design tema. Et par af den livsstil og bolig udsmykning butik tilfældigvis konstant opgradering personer, der bruger de nyere ideer, ideer og meget mere om ejendommen stil. Gardiner, inde sofaer, spise sæder, og plader udføre en stor del i at øge hjem design. Forhæng som hjem design element er yderst vigtige. Gardiner er meget mere end bare effektiv nødvendighed. Derudover de udfører en stor del i den samlede design forbundet med hjem. Gardiner er væsentlige aspekter af hjem design som effektive værktøj ved at forhindre den faktiske dagslys og synlige, da det er en del af ejendommen leverer samt design.For mere information om krystal gaver besøge denne hjemmeside.

commmenttor
celine phantom bag new york

Demand for a status bag has become so hot that several Web site have emerged where people can borrow luxury handbags.

commmenttor
buy jordans

You need to take part in a contest for one of the preferred blogs on the internet. I will suggest this internet site!
buy jordans http://www.thefactoryentertainment.com/blogs/item/if-you-include-marketing

commmenttor
China nba jerseys outlet

Aux municipales,China nba jerseys outlet Freddy Vasseu se présentait en tant que membre du Parti pirate sur une liste de centre droit à Port-Lès-Valence. Résultat des élections, China nba jerseys outletil est élu au conseil municipal de la ville dès le premier tour. Le Parti pirate compte également 2 autres élus municipaux.

commmenttor
cheap hermes luggage

cheap hermes luggage exact replica hermes handbags mens obmijhayu

commmenttor
jordan retro 9 kilroy

This internet site is actually a walk-through for all the information you wanted about this and didn't know who to ask. Glimpse here, and you'll certainly discover it.
jordan retro 9 kilroy http://coachstresszero.com/Images/_notes/sys.asp?key=jordan-retro-9-kilroy-93

commmenttor
wholesale jerseys usa

I'm often to blogging and i genuinely appreciate your content. The write-up has truly peaks my interest. I'm going to bookmark your web-site and maintain checking for new data.
wholesale jerseys usa http://gjslgds.2335656.n4.nabble.com/Cheap-Nfl-Jerseys-China-Nike-The-Most-Thrilling-Professions-To-Essay-Buying-And-Commerce-trading-td2.html

commmenttor
Sale Bottega Veneta For Cheap

Fast shipping and product as described. Great service.

commmenttor
Post a comment Name :

Email :

Your site url:

Comment :

© Agiliq, 2009-2012