Last active
          September 27, 2019 19:57 
        
      - 
      
- 
        Save danielsousaio/f54efc1774af491e44292d102856d534 to your computer and use it in GitHub Desktop. 
Revisions
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 6 additions and 6 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -6,7 +6,7 @@ This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. Using the flag `--no-input` isn't enough as there is still 3 scenarios that raise the questioner. **Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`.** When we send the flag `--no-input` to the `makemigrations` command it calls the non-interactive questioner. @@ -18,7 +18,7 @@ else: questioner = NonInteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) ``` **`NonInteractiveMigrationQuestioner` itself will return `sys.exit(3)` in three methods.** [Source code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L227-L239): ```python @@ -37,9 +37,9 @@ class NonInteractiveMigrationQuestioner(MigrationQuestioner): sys.exit(3) ``` With this in mind, if we want to prevent a exit code other than 0 (successful exit) we have to avoid the call of the 3 methods. -**`ask_auto_now_add_addition` and `ask_not_null_addition`** This is from [where](https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py#L860-L871) they can be called: ```python @@ -57,7 +57,7 @@ if not preserve_default: field.default = self.questioner.ask_not_null_addition(field_name, model_name) ``` **`ask_not_null_alteration`** This is from [where](https://github.com/django/django/blob/5931d2e96ae94b204d146b7f751e0e804da74953/django/db/migrations/autodetector.py#L959-L962) it can be called: ```python @@ -96,7 +96,7 @@ def ask_not_null_alteration(self, field_name, model_name): return None ``` **TL;DR** The magic sauce: 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,8 +1,8 @@ #### Introduction **Issue**: sys.exit(3) on background job that runs makemigrations - infinite loop **PR**: https://github.com/crowdbotics/crowdbotics-slack-app/pull/1003 This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. Using the flag `--no-input` isn't enough as there is still 3 scenarios that raise the questioner. 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 3 additions and 2 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,7 +1,8 @@ #### Introduction *Issue*: sys.exit(3) on background job that runs makemigrations - infinite loop *PR*: https://github.com/crowdbotics/crowdbotics-slack-app/pull/1003 This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. Using the flag `--no-input` isn't enough as there is still 3 scenarios that raise the questioner. 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 3 additions and 0 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,5 +1,8 @@ #### Introduction Issue: sys.exit(3) on background job that runs makemigrations - infinite loop PR: https://github.com/crowdbotics/crowdbotics-slack-app/pull/1003 This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. Using the flag `--no-input` isn't enough as there is still 3 scenarios that raise the questioner. #### Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,6 @@ #### Introduction This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. Using the flag `--no-input` isn't enough as there is still 3 scenarios that raise the questioner. #### Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,4 +1,4 @@ #### Introduction This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. If we use the flag `--no-input` it isn't enough as 2 scenarios still raise the questioner no matter what. 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -2,7 +2,7 @@ This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. If we use the flag `--no-input` it isn't enough as 2 scenarios still raise the questioner no matter what. #### Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. When we send the flag `--no-input` to the `makemigrations` command it calls the non-interactive questioner. @@ -14,7 +14,7 @@ else: questioner = NonInteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) ``` #### `NonInteractiveMigrationQuestioner` itself will return `sys.exit(3)` in three methods. [Source code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L227-L239): ```python 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 7 additions and 5 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -2,7 +2,7 @@ This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. If we use the flag `--no-input` it isn't enough as 2 scenarios still raise the questioner no matter what. ### Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. When we send the flag `--no-input` to the `makemigrations` command it calls the non-interactive questioner. @@ -14,7 +14,7 @@ else: questioner = NonInteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) ``` ### `NonInteractiveMigrationQuestioner` itself will return `sys.exit(3)` in three methods. [Source code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L227-L239): ```python @@ -35,7 +35,7 @@ class NonInteractiveMigrationQuestioner(MigrationQuestioner): With thit in mind, if we want to prevent a exit code other than 0 (successful exit) we have to avoid the call of the 3 following methods. #### `ask_auto_now_add_addition` and `ask_not_null_addition` This is from [where](https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py#L860-L871) they can be called: ```python @@ -53,7 +53,7 @@ if not preserve_default: field.default = self.questioner.ask_not_null_addition(field_name, model_name) ``` #### `ask_not_null_alteration` This is from [where](https://github.com/django/django/blob/5931d2e96ae94b204d146b7f751e0e804da74953/django/db/migrations/autodetector.py#L959-L962) it can be called: ```python @@ -92,7 +92,9 @@ def ask_not_null_alteration(self, field_name, model_name): return None ``` #### TL;DR The magic sauce: ```python # When creating fields 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 32 additions and 16 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,10 @@ ## Introduction This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. If we use the flag `--no-input` it isn't enough as 2 scenarios still raise the questioner no matter what. ## Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. When we send the flag `--no-input` to the `makemigrations` command it calls the non-interactive questioner. [Source code](https://github.com/django/django/blob/290d8471bba35980f3e228f9c171afc40f2550fa/django/core/management/commands/makemigrations.py#L135-L138): ```python @@ -10,8 +14,7 @@ else: questioner = NonInteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) ``` ## `NonInteractiveMigrationQuestioner` itself will return `sys.exit(3)` in two methods. [Source code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L227-L239): ```python @@ -30,13 +33,11 @@ class NonInteractiveMigrationQuestioner(MigrationQuestioner): sys.exit(3) ``` With thit in mind, if we want to prevent a exit code other than 0 (successful exit) we have to avoid the call of the 3 following methods. ## `ask_auto_now_add_addition` and `ask_not_null_addition` This is from [where](https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py#L860-L871) they can be called: ```python time_fields = (models.DateField, models.DateTimeField, models.TimeField) preserve_default = ( @@ -52,17 +53,18 @@ if not preserve_default: field.default = self.questioner.ask_not_null_addition(field_name, model_name) ``` ## `ask_not_null_alteration` This is from [where](https://github.com/django/django/blob/5931d2e96ae94b204d146b7f751e0e804da74953/django/db/migrations/autodetector.py#L959-L962) it can be called: ```python if (old_field.null and not new_field.null and not new_field.has_default() and not new_field.many_to_many): field = new_field.clone() new_default = self.questioner.ask_not_null_alteration(field_name, model_name) ``` This one doesn't return `sys.exit(3)`, instead it defaults to the `NOT_PROVIDED` option just as we chose `Ignore for now..` option. This [Source Code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L166-L177) explains why its a good idea to avoid it too, check the message: ```python def ask_not_null_alteration(self, field_name, model_name): """Changing a NULL field to NOT NULL.""" @@ -88,4 +90,18 @@ def ask_not_null_alteration(self, field_name, model_name): else: return self._ask_default() return None ``` **TL;DR**: ```python # When creating fields field.null or field.has_default() or field.many_to_many or (field.blank and field.empty_strings_allowed) or (isinstance(field, time_fields) and field.auto_now) # When updating fields old_field.null and not new_field.null and not new_field.has_default() and not new_field.many_to_many ``` Above is all the logic Django uses to check if the questioner should display or not. We could do the same checks and rest assured that the non-interactive makemigrations never fails. I also suggested earlier that if the request to the model builder fails this very same checks that the API should reply with an error stating that something more is required (whatever is missing). 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -62,7 +62,7 @@ https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py if (old_field.null and not new_field.null and not new_field.has_default() and not new_field.many_to_many): ``` To avoid the call of `ask_not_null_alteration`. The [source code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L166-L177) explains why this is a good idea, because the non-interactive way always runs the `NOT_PROVIDED` option - `Ignore for now..`: ```python def ask_not_null_alteration(self, field_name, model_name): """Changing a NULL field to NOT NULL.""" 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 27 additions and 1 deletion.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -62,4 +62,30 @@ https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py if (old_field.null and not new_field.null and not new_field.has_default() and not new_field.many_to_many): ``` To avoid the call of `ask_not_null_alteration`. The [source code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L166-L177) explains why this is a good idea: ```python def ask_not_null_alteration(self, field_name, model_name): """Changing a NULL field to NOT NULL.""" if not self.dry_run: choice = self._choice_input( "You are trying to change the nullable field '%s' on %s to non-nullable " "without a default; we can't do that (the database needs something to " "populate existing rows).\n" "Please select a fix:" % (field_name, model_name), [ ("Provide a one-off default now (will be set on all existing " "rows with a null value for this column)"), ("Ignore for now, and let me handle existing rows with NULL myself " "(e.g. because you added a RunPython or RunSQL operation to handle " "NULL values in a previous data migration)"), "Quit, and let me add a default in models.py", ] ) if choice == 2: return NOT_PROVIDED elif choice == 3: sys.exit(3) else: return self._ask_default() return None ``` 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 5 additions and 2 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -56,7 +56,10 @@ if not preserve_default: **UPDATE:** This [check]( https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py#L957-L958) should also be made: ```python if (old_field.null and not new_field.null and not new_field.has_default() and not new_field.many_to_many): ``` To avoid the call of `ask_not_null_alteration`. 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 8 additions and 1 deletion.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -52,4 +52,11 @@ if not preserve_default: field.default = self.questioner.ask_not_null_addition(field_name, model_name) ``` **TL;DR**: Above is all the logic Django uses to check if the questioner should display or not. Checks such as is the field nullable? Does the field have a default? We could do the same checks and rest assured that the non-interactive makemigrations never fails. I also suggested earlier that if the request to the model builder fails this very same checks that the API should reply with an error stating that something more is required (whatever is missing). **UPDATE:** This check should also be made: https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py#L957-L958 To avoid the call of `ask_not_null_alteration`. 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 31 additions and 3 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,13 +1,19 @@ This gist proposes a solution to this issue by doing the same checks Django uses to either ask further questions during migrations creation or not. Currently our background job stays in a loop because we don't provide a valid answer to this questioner. If we use the flag `--no-input` it isn't enough as 2 scenarios still raise the questioner no matter what. Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. When we send the flag `--no-input` to the `makemigrations` command it calls the non-interactive questioner. [Source code](https://github.com/django/django/blob/290d8471bba35980f3e228f9c171afc40f2550fa/django/core/management/commands/makemigrations.py#L135-L138): ```python if self.interactive: questioner = InteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) else: questioner = NonInteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) ``` `NonInteractiveMigrationQuestioner` itself will return `sys.exit(3)` in two methods. [Source code](https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L227-L239): ```python class NonInteractiveMigrationQuestioner(MigrationQuestioner): @@ -24,4 +30,26 @@ class NonInteractiveMigrationQuestioner(MigrationQuestioner): sys.exit(3) ``` With that in mind, if we want to prevent a exit code other than 0 (successful exit) we have to avoid the call of: - `ask_auto_now_add_addition` - `ask_not_null_addition` `ask_not_null_alteration` doesn't fail but expects us to have a sql script to run after to set all records field value to NULL. [Source code](https://github.com/django/django/blob/2.2.5/django/db/migrations/autodetector.py#L860-L871): ```python time_fields = (models.DateField, models.DateTimeField, models.TimeField) preserve_default = ( field.null or field.has_default() or field.many_to_many or (field.blank and field.empty_strings_allowed) or (isinstance(field, time_fields) and field.auto_now) ) if not preserve_default: field = field.clone() if isinstance(field, time_fields) and field.auto_now_add: field.default = self.questioner.ask_auto_now_add_addition(field_name, model_name) else: field.default = self.questioner.ask_not_null_addition(field_name, model_name) ``` **TL;DR**: Above is all the logic Django uses to check if the questioner should display or not. Checks such as is the field nullable? Does the field have a default? We could do the same checks and rest assured that the non-interactive makemigrations never fails. I also suggested earlier that if the request to the model builder fails this very same checks that the API should reply with an error stating that something more is required (whatever is missing). 
- 
        danielsousaio revised this gist Sep 27, 2019 . 1 changed file with 2 additions and 3 deletions.There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,6 +1,5 @@ Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. When we send the flag `--no-input` to the `makemigrations` command it calls the non-interactive questioner. ```python if self.interactive: questioner = InteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) else: @@ -9,7 +8,7 @@ else: https://github.com/django/django/blob/290d8471bba35980f3e228f9c171afc40f2550fa/django/core/management/commands/makemigrations.py#L135-L138 `NonInteractiveMigrationQuestioner` itself will return `sys.exit(3)` in two methods: ```python class NonInteractiveMigrationQuestioner(MigrationQuestioner): def ask_not_null_addition(self, field_name, model_name): 
- 
        danielsousaio renamed this gist Sep 27, 2019 . 1 changed file with 0 additions and 0 deletions.There are no files selected for viewingFile renamed without changes.
- 
        danielsousaio created this gist Sep 27, 2019 .There are no files selected for viewingThis file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,28 @@ Calling `makemigrations` can call either `InteractiveMigrationQuestioner` or `NonInteractiveMigrationQuestioner`. When we send the flag `--no-input` to the `makemigrations` command it calls the non-interactive questioner. ``` if self.interactive: questioner = InteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) else: questioner = NonInteractiveMigrationQuestioner(specified_apps=app_labels, dry_run=self.dry_run) ``` https://github.com/django/django/blob/290d8471bba35980f3e228f9c171afc40f2550fa/django/core/management/commands/makemigrations.py#L135-L138 `NonInteractiveMigrationQuestioner` itself will return `sys.exit(3)` in two methods: ``` class NonInteractiveMigrationQuestioner(MigrationQuestioner): def ask_not_null_addition(self, field_name, model_name): # We can't ask the user, so act like the user aborted. sys.exit(3) def ask_not_null_alteration(self, field_name, model_name): # We can't ask the user, so set as not provided. return NOT_PROVIDED def ask_auto_now_add_addition(self, field_name, model_name): # We can't ask the user, so act like the user aborted. sys.exit(3) ``` https://github.com/django/django/blob/e90af8bad44341cf8ebd469dac57b61a95667c1d/django/db/migrations/questioner.py#L227-L239