Skip to content

Instantly share code, notes, and snippets.

View valiantone's full-sized avatar
🦋
learning to embrace chaos

Zubin J valiantone

🦋
learning to embrace chaos
View GitHub Profile
@valiantone
valiantone / pandas_s3_streaming.py
Created November 30, 2021 15:01
Streaming pandas DataFrame to/from S3 with on-the-fly processing and GZIP compression
def s3_to_pandas(client, bucket, key, header=None):
# get key using boto3 client
obj = client.get_object(Bucket=bucket, Key=key)
gz = gzip.GzipFile(fileobj=obj['Body'])
# load stream directly to DF
return pd.read_csv(gz, header=header, dtype=str)
def s3_to_pandas_with_processing(client, bucket, key, header=None):

Installing Cool-Retro-Term on Windows10

First of all, this document is just a recompilation of different resources that already existed on the web previously that I personally tested some ones did work and other not. I liked the idea to make a full guide from start to end so all of you could also enjoy playing with cool-retro-term on windows 10. Personally I installed it on a windows 10 pro version. Fingers crossed!

result

@valiantone
valiantone / mongo_to_csv.py
Created February 28, 2020 15:19 — forked from mieitza/mongo_to_csv.py
python mongo to csv use pandas.
# @Author: xiewenqian <int>
# @Date: 2016-11-28T20:35:09+08:00
# @Email: [email protected]
# @Last modified by: int
# @Last modified time: 2016-12-01T19:32:48+08:00
import pandas as pd
from pymongo import MongoClient
@valiantone
valiantone / pandas_labeled_csv_import_to_mongo.py
Created February 28, 2020 15:18 — forked from jxub/pandas_labeled_csv_import_to_mongo.py
A simple mongoimport for importing csv files with python and pymongo
import pandas as pd
from pymongo import MongoClient
import json
def mongoimport(csv_path, db_name, coll_name, db_url='localhost', db_port=27000)
""" Imports a csv file at path csv_name to a mongo colection
returns: count of the documants in the new collection
"""
client = MongoClient(db_url, db_port)
db = client[db_name]
@valiantone
valiantone / hash.py
Created October 4, 2019 13:30 — forked from scwood/hash.py
python hash table using linear probing
class HashTable(object):
def __init__(self):
self.max_length = 8
self.max_load_factor = 0.75
self.length = 0
self.table = [None] * self.max_length
def __len__(self):
return self.length
@valiantone
valiantone / mongodb_hook.py
Created April 17, 2018 12:23 — forked from gghez/mongodb_hook.py
MongoDB Hook for Airflow system.
from airflow.hooks.base_hook import BaseHook
from pymongo import MongoClient
class MongoDBHook(BaseHook):
def __init__(self, conn_id='mongodb_default'):
self.conn = self.get_connection(conn_id)
self.client = MongoClient(host=self.conn.host, port=self.conn.port)
def __getattr__(self, name):