First of all, this document is just a recompilation of different resources that already existed on the web previously that I personally tested some ones did work and other not. I liked the idea to make a full guide from start to end so all of you could also enjoy playing with cool-retro-term on windows 10. Personally I installed it on a windows 10 pro version. Fingers crossed!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def s3_to_pandas(client, bucket, key, header=None): | |
| # get key using boto3 client | |
| obj = client.get_object(Bucket=bucket, Key=key) | |
| gz = gzip.GzipFile(fileobj=obj['Body']) | |
| # load stream directly to DF | |
| return pd.read_csv(gz, header=header, dtype=str) | |
| def s3_to_pandas_with_processing(client, bucket, key, header=None): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # @Author: xiewenqian <int> | |
| # @Date: 2016-11-28T20:35:09+08:00 | |
| # @Email: [email protected] | |
| # @Last modified by: int | |
| # @Last modified time: 2016-12-01T19:32:48+08:00 | |
| import pandas as pd | |
| from pymongo import MongoClient |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pandas as pd | |
| from pymongo import MongoClient | |
| import json | |
| def mongoimport(csv_path, db_name, coll_name, db_url='localhost', db_port=27000) | |
| """ Imports a csv file at path csv_name to a mongo colection | |
| returns: count of the documants in the new collection | |
| """ | |
| client = MongoClient(db_url, db_port) | |
| db = client[db_name] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| class HashTable(object): | |
| def __init__(self): | |
| self.max_length = 8 | |
| self.max_load_factor = 0.75 | |
| self.length = 0 | |
| self.table = [None] * self.max_length | |
| def __len__(self): | |
| return self.length |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from airflow.hooks.base_hook import BaseHook | |
| from pymongo import MongoClient | |
| class MongoDBHook(BaseHook): | |
| def __init__(self, conn_id='mongodb_default'): | |
| self.conn = self.get_connection(conn_id) | |
| self.client = MongoClient(host=self.conn.host, port=self.conn.port) | |
| def __getattr__(self, name): |
