Skip to content

Instantly share code, notes, and snippets.

@databill86

databill86/loading_wikipedia.py

Forked from thomwolf/loading_wikipedia.py
Created June 15, 2020 13:35
Show Gist options
  • Save databill86/f16a1b742f879d60e7ae466dde5d07c1 to your computer and use it in GitHub Desktop.
Save databill86/f16a1b742f879d60e7ae466dde5d07c1 to your computer and use it in GitHub Desktop.
Download ZIP
Load full English Wikipedia dataset in HuggingFace nlp library
Raw
loading_wikipedia.py
import os; import psutil; import timeit
from nlp import load_dataset
mem_before = psutil.Process(os.getpid()).memory_info().rss >> 20
wiki = load_dataset("wikipedia", "20200501.en", split='train')
mem_after = psutil.Process(os.getpid()).memory_info().rss >> 20
print(f"RAM memory used: {(mem_after - mem_before)} MB")
s = """batch_size = 1000
for i in range(0, len(wiki), batch_size):
batch = wiki[i:i + batch_size]
"""
time = timeit.timeit(stmt=s, number=1, globals=globals())
size = wiki.dataset_size / 2**30
print(f"Iterated over the {size:.1f} GB dataset in {time:.1f} s, i.e. {size * 8/time:.1f} Gbit/s")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment