A Note on Squashing Django Migrations

Django support squashing migrations that compresses the migration history.

eventually you will want to move back from having several hundred migrations to just a few

Which beg the obvious question, why?

Motivation

speed up tests without disabling migration (e.g. when the success of migration is the very thing to test against)
speed up new machine setup (e.g. for ci/cd)

Limitation

The latest function added to squashing migration was the --squashed-name option, done in 2017, in version 2.0. The command hasn't received any major update since then. The command has quite a few limitations, or maybe bugs, that isn't obvious from the documentation

Dependency Resolution - The squashmigrations command does not resolve dependencies on your behalf, might result in CircularDependencyError
Equivalency - The squashed migration might not be equivalent to the migrations before squashing
Atomicity - For databases that wrap migration inside a transaction, the atomic=False flag is ignored by squashmigrations command
Custom Operation - RunPython and RunSQL (and SeparateDatabaseAndState with RunSQL/RunPython) are not automatically copied into the squashed migration
Reverting - When migrations messed up, reverting to pre-squashed migration messes up migration table

Dependency Resolution

Below I'm using the notation {app}_{N} to denote the N-th migration of app {app}. In a django project, it maps to file {app}/migrations/{N}_some_suffix.py For a django app with linear migration dependency, e.g.

where -> denotes the dependencies attribute in a migration file,

The command python manage.py squashmigrations app 001 005 squashes the five migrations nicely into one app_001_squashed_app_005. This form the basis of operation.

If one of the migrations has multiple dependencies to another app, e.g. app_003 depends also on foo_001

app_003 -> foo_001

The same operation python manage.py squashmigrations app 001 005 will leave the resulting squashed migration file app_001_squashed_app_005 having a dependency of foo_001

If foo_001 transitively, ultimately, depends on the app app, migration app_001

, running the migration by python manage.py migrate app app_001_squashed_app_005 will result in a CircularDependencyError

The squashmigrations command offers no programmatic checking to your dependency. User's discretion is required.

Equivalency

One might expect the squashed migration file app_001_squashed_app_005 is equivalent to its source migration files app_001 ... app_005. Put it differently, after running python manage.py squashmigrations app 001 005, one can verify equivalency by python manage.py makemigrations app. If they are the same, makemigrations should generate no further extra migrations.

It is not the case when an one-off default was concerned.

Suppose a model is create in app_001

# app/migrations/001.py
class Migration(migrations.Migration):
  initial = True
  dependencies = []
  operations = [
    migrations.CreateModel(
      name='Product',
      fields=[
        ('id', models.AutoField(auto_created=True, primary_key=True, ...),
        ('name', models.TextField())
      ]
    )
  ]

In later migrations, e.g. app_002, a new column with an one-off default were added

# app/migrations/002.py
class Migration(migrations.Migration):
  dependencies = [('app', '002')]
  operations = [
    migrations.AddField(
      model_name='product',
      name='description',
      fields=models.TextField(default='Some value'),
      preserve_default=False,  # one off default
    )
  ]

The squashed migration file will have something like

# app/migrations/001_squashed_005.py
class Migration(migrations.Migration):
  initial = True
  dependencies = []
  replaces = [('app', '001'), ('app', '002'), ('app', '003'), ...]
  operations = [
    migrations.CreateModel(
      name='Product',
      fields=[
        ('id', models.AutoField(auto_created=True, primary_key=True, ...),
        ('name', models.TextField()),
        ('description', models.TextField(default='Some value')),
      ]
    )
  ]

The squashing essentially turning a one-off default into a permanent default. Upon running makemigrations app, django will generate another migration file to alter the description field to take no default, breaking the equivalency assumption. In the above example, a manual removal of default value default='Some value' from the squashed migration suffice.

Atomicity

Squashed migration ignores (I could be wrong, but quick glance at MigrationWrite doesn't seem to process the atomic Flag) atomicity. While migration operations are wrapped in a transaction block in sqlite and postgres by default, there are migration operation that works only when atomic = False, e.g AddIndexConcurrently

Custom Operation

RunPython is not automatically copied into the squashed migration. A message will be displayed when running squashmigrations. Manual copying the function to the squashed migration file is needed.

RunSQL on the other hand, is copied. But depends on the actual SQL, the SQL might fail in the squashed migration for various reason. e.g.

operating on a no-longer-exist column
creating concurrent index which cannot operate under a transaction, as mentioned in previous section

SeparateDatabaseAndState is directly copied into the squashed migration file without validation. Since SeparateDatabaseAndState can wraps both RunPython and RunSQL, the operation might require manual copying code, or might not run at all.

Reverting a squashed migration

With limitations above, one might be motivated to revert a applied squashed migration to unsquashed state, exclude some migration history from squashing, and re-apply the squashing.

In pre-4.0 django, there was a bug that leaves the unapplied squashed migration still in the migration table. If one inherits a pre-4.0 django project, the fix is to manually remove the dangling migration record by MigrationRecorder.Migration.objects.filter(app=app, name=name).delete() after reversing the squashed migration.

kmcheung12/squashing.md