Skip to content

Instantly share code, notes, and snippets.

@kangfend

kangfend/remove_duplicates.py

Forked from victorono/remove_duplicates.py
Created January 7, 2020 05:32
Show Gist options
  • Save kangfend/351083ee29a1d2c53f8340a36a0b965f to your computer and use it in GitHub Desktop.
Save kangfend/351083ee29a1d2c53f8340a36a0b965f to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @victorono victorono revised this gist Dec 12, 2016. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion remove_duplicates.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,9 +1,11 @@
    from django.db.models import Count, Max

    unique_fields = ['field_1', 'field_2']

    duplicates = (
    MyModel.objects.values(*unique_fields)
    .order_by()
    .annotate(max_id=models.Max('id'), count_id=models.Count('id'))
    .annotate(max_id=Max('id'), count_id=Count('id'))
    .filter(count_id__gt=1)
    )

  2. @victorono victorono created this gist Dec 12, 2016.
    16 changes: 16 additions & 0 deletions remove_duplicates.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,16 @@
    unique_fields = ['field_1', 'field_2']

    duplicates = (
    MyModel.objects.values(*unique_fields)
    .order_by()
    .annotate(max_id=models.Max('id'), count_id=models.Count('id'))
    .filter(count_id__gt=1)
    )

    for duplicate in duplicates:
    (
    MyModel.objects
    .filter(**{x: duplicate[x] for x in unique_fields})
    .exclude(id=duplicate['max_id'])
    .delete()
    )