Database Migration in Django using South

When I started using Django, if I came across a situation where I’ve to change the max_length of a charfield(for example if I want to change the length from 3 to 4) what I normally did was delete the entire database and start from the beginning using syncdb. At that time I was thinking what will happen if the data in the db are important, I continued to use the same method as I was not handling db populated with client data. But the requirement soon came. I’ve to write sql query inorder to change the field structure. It was a hard process(as I was less familiar with writing sql queries). At that time I came to know about an app built in Python to do the process automatically(basic usage), South.

South brings migrations to our Django applications.

You can install South either by pip or easy_install

  • pip install south

South have two types of migrations:

  • schemamigration(migration for schema change)
  • datamigration(migration for data)

Using South with a New App

Create a new Django project and add a new app(let’s name it web). Add a class to models.py
class Poll(models.Model):
    name = models.CharField(max_length=100)
    published = models.BooleanField()

For automatic migration we have two options initial and auto. auto will look for previous migration and create a new migration which applies the difference. But if we are using South for first time, we don’t have any previous migration. Here comes the role of initial. It’ll create the first migration. It works almost similar to syncdb. Afterwards we use auto.

Instead of running syncdb, execute the following command:
shinu@odol:~/login$ python manage.py schemamigration web --initial
Creating migrations directory at '/home/users/shinu/venvs/login/login/web/migrations'...
Creating __init__.py in '/home/users/shinu/venvs/login/login/web/migrations'...
+ Added model web.Poll
Created 0001_initial.py. You can now apply this migration with: ./manage.py migrate web

On executing the above code, a new folder(migrations) will be created inside your app folder(web). It will contain the migration files.

And inorder to apply the changes to your db you’ve execute:
shinu@odol:~/login$ python manage.py migrate web
Running migrations for web:
 - Migrating forwards to 0001_initial.
 > web:0001_initial
 - Loading initial data for web.
No fixtures found.

What we have done is similar to what syncdb does. Now comes the real play of South. Now if we want to add a new field to the class Poll.
class Poll(models.Model):
    name = models.CharField(max_length=100)
    published = models.BooleanField()
    latest = models.BooleanField()

As we are using South we need only two steps to make db know we’ve added a new field. First create a schemamigration with auto feature(because we’ve already created a migration with initial feature) and then migrate the db using the file created by schemamigration.
shinu@odol:~/login$ python manage.py schemamigration web --auto
 + Added field latest on web.Poll
Created 0002_auto__add_field_poll_latest.py. You can now apply this migration with: ./manage.py migrate web

You can now apply the migration
shinu@odol:~/login$ python manage.py migrate web
Running migrations for web:
 - Migrating forwards to 0002_auto__add_field_poll_latest.
 > web:0002_auto__add_field_poll_latest
 - Loading initial data for web.
No fixtures found.

A new column latest will be added to your table.

Using South in an Existing App

In many case you may need to install South after working on an app a bit. In that case you have to first convert you existing app to a app using South. You have to perform the following steps:

  • Install South and add south to your INSTALLED_APPS in settings.py
  • Run python manage.py syncdb
  • Run ./manage.py convert_to_south web

That’s it.

If you are working in a team and using any version control system, you have to commit this migration. And every other member have to perform a syncdb and then run
./manage.py migrate myapp 0001 --fake
on their machine after pulling code.

But if you are using a fresh installation of the same codebase, you just have to run syncdb and a normal migration.

Advanced Uses

Upto now we just saw the basic usage of South. Let’s find something advance.

Add a column length to the model class Poll:
class Poll(models.Model):
    name = models.CharField(max_length=100)
    published = models.BooleanField()
    latest = models.BooleanField()
    length = models.IntegerField(null=False)

The length field is not nullable and doesn’t have a default given. So the problem here is South doesn’t know whether our database have any value stored or not. It just assumes that the db contains some data. And now we are adding a new column which is not having a default value or can be made nullable. If that field is nullable South will save null to all the existing data. In this case it will prompt for the user to do the needful.

python manage.py schemamigration web --auto
 ? The field 'Poll.length' does not have a default specified, yet is NOT NULL.
 ? Since you are adding this field, you MUST specify a default
 ? value to use for existing rows. Would you like to:
 ? 1. Quit now, and add a default to the field in models.py
 ? 2. Specify a one-off value to use for existing columns now
 ? Please select a choice: 2
 ? Please enter Python code for your one-off default value.
 ? The datetime module is available, so you can do e.g. datetime.date.today()
 >>> 0
 + Added field length on web.Poll
Created 0003_auto__add_field_poll_length.py. You can now apply this migration with: ./manage.py migrate web

In this case what I’ve done is simply given a default value(0) to existing db data. If I have selected option 1, the command will quit automatically. We have to manually goto models.py and add a default value and run the schemamigration again.

Apply the migration normally
./manage.py migrate web
Running migrations for web:
 - Migrating forwards to 0003_auto__add_field_poll_length.
 > web:0003_auto__add_field_poll_length
 - Loading initial data for web.
No fixtures found.

Data Migration

Some case may come in which you want to modify a field(for example from IntegerField to ForeignKey), but you need the values stored in that field to compute the new value as you cannot give a default value or make it null. In that case South comes with data migration. We can view data migration with an example(it’s the best way to explain it).

In the existing Poll class let’s add a new field.
comment_id = models.IntegerField()
Let it store the id of Comment class. Apply the necessary schemamigration and migrate the app.

Now I want to change the comment_id field to a field which is a ForeignKey to the comment. That means I’ve to make use of all comment_id‘s and have to find corresponding comment. This can be done with 3 migrations: first a schemamigration, then a data migration and finally again a achemamigration.

Let me explain it in the following steps:

Step1:

Add the new field with null=True to Poll class:
comment = models.ForeignKey(Comment, null=True)

Do a schemamigration for this change:
python manage.py schemamigration web --auto
If you’ve defined the class Comment after the class Poll you’ll get a name error:
NameError: name 'Comment' is not defined

You can get rid of this error in two ways:

  • move the class Comment before Poll
  • modify the field comment as following
    comment = models.ForeignKey('Comment', null=True)

python manage.py schemamigration web --auto
 + Added field comment on web.Poll
Created 0005_auto__add_field_poll_comment.py. You can now apply this migration with: ./manage.py migrate web

Step2:

Now we’ve to do a datamigration. Use the field comment_id to populate the field comment, which is null now. Usage of datamigration is a little bit different. We have to create a new migration file and manually edit it. After that apply the migration as normal.
python manage.py datamigration web poll_comment_added
Created 0006_poll_comment_added.py.

A new file with name 0006_poll_comment_added.py is created in migration folder. The last parameter is the name of the migration, which you can name at your choice. Now open the file to edit. You will have to edit two methods: forward and backward.

def forwards(self, orm):
    "Write your forwards methods here."
    for poll in orm.Polls.objects.all():
        poll.comment = orm.Comment.objects.get(id=poll.comment_id)
        poll.save()

forwards will populate the comment field using comment_id field.
def backwards(self, orm):
    "Write your backwards methods here."
    raise RuntimeError("Cannot reverse this migration.")

backwards is set to raise an exception as datamigration is an irreversible process.

Step3:

Now do a schemamigration after removing the field comment_id
python manage.py schemamigration web --auto
 ? The field 'Poll.comment_id' does not have a default specified, yet is NOT NULL.
 ? Since you are removing this field, you MUST specify a default
 ? value to use for existing rows. Would you like to:
 ? 1. Quit now, and add a default to the field in models.py
 ? 2. Specify a one-off value to use for existing columns now
 ? 3. Disable the backwards migration by raising an exception.
 ? Please select a choice: 3
 - Deleted field comment_id on web.Poll
Created 0007_auto__del_field_poll_comment_id.py. You can now apply this migration with: ./manage.py migrate web

South is asking for a default value for comment_id because if you have to reverse this migration, South will try to add this default value when reversed. In this case I’ve selected choice 3, and if again we try to reverse this migration South will raise an exception.

Migrate the database:
./manage.py migrate web
Running migrations for web:
 - Migrating forwards to 0007_auto__del_field_poll_comment_id.
 > web:0006_poll_comment_added
 - Loading initial data for web.
No fixtures found.

Common Error

One error I came across while I tried to use datamigration is a KeyError, eg:
KeyError: "The model 'user' from the app 'web' is not available in this migration."
In this case I tried to use User class provided by Django in my forwards method. We have to modify the forwards method in such a way that wherever we are using orm.User change it to orm['auth.User'].

About Odol Shinu

I've completed my B Tech in Information Technology in 2010 from Government Engineering College Sreekrishnapuram Palakkad under Calicut University.

Posted on April 2, 2012, in Django and tagged , , , . Bookmark the permalink. 1 Comment.

  1. Hi. I noted that you wrote “We have to modify the forwards method in such a way that wherever we are using orm.User change it to orm[‘auth.User’].”

    Can you provide the original migration you were referring to? I came across what I believe to be a similar issue.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: