Skip to content

Instantly share code, notes, and snippets.

@sandgate-dev

sandgate-dev/set_csv_field_limit.py

Created April 7, 2019 11:35
Show Gist options
  • Save sandgate-dev/385af65851ee2317285a3dd6ccbf999a to your computer and use it in GitHub Desktop.
Save sandgate-dev/385af65851ee2317285a3dd6ccbf999a to your computer and use it in GitHub Desktop.
Download ZIP
Increase field limit to address _csv.Error: field larger than field limit (131072)
Raw
set_csv_field_limit.py
# _csv.Error: field larger than field limit (131072)
# https://stackoverflow.com/a/15063941/5983691
def csv_field_limit():
maxInt = sys.maxsize
decrement = True
while decrement:
# decrease the maxInt value by factor 10
# as long as the OverflowError occurs.
decrement = False
try:
csv.field_size_limit(maxInt)
except OverflowError:
maxInt = int(maxInt/10)
decrement = True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment