This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Hadoop Streaming step | |
| { | |
| "jobFlowId": "j-1JKGNMYXT59DP", | |
| "steps": [ | |
| { | |
| "name": "EMRHadoopLogPushingTest", | |
| "actionOnFailure": "CANCEL_AND_WAIT", | |
| "hadoopJarStep": { | |
| "jar": "command-runner.jar", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # submit a step to run a command on a cluster | |
| AWS_PROFILE=integ-preprod aws emr add-steps \ | |
| --cluster-id j-32M91WVU0V1OR \ | |
| --steps Name="Create table",ActionOnFailure=CONTINUE,Jar=command-runner.jar,Args=[beeline,-u,jdbc:hive2://ec2-52-91-119-206.compute-1.amazonaws.com:10000,-n,admin,-p,admin,-e,"create table d417f3333(id INT, value STRING);"] --region us-east-1 --endpoint https://emr-preprod-us-east-1.amazon.com --no-verify |
ranjitiyer
/ sqllite-example.py
Created
February 8, 2019 23:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| db = 'test.db' | |
| table = 'Table' | |
| releaseLabel = "releaseLabel" | |
| applicationString = "applicationString" | |
| account = "account" | |
| creationDate = "creationDate" | |
| def create_table(overwrite: bool): | |
| if overwrite and table_exists(table): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| time.asctime([t]) | |
| t is response of gmtime() or localtime() | |
| time.clock() | |
| time.clock_gettime(clk_id) > float | |
| time.clock_gettime_ns(clk_id) > float | |
| # Returns the response in this format |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "id": 56, | |
| "title": "Secret Garden", | |
| "description": "Mary Lennox is a spoiled, middle-class, self-centred child who has been recently orphaned. She is accepted into the quiet and remote country house of an uncle, who has almost completely withdrawn into himself after the death of his wife. Mary gradually becomes drawn into the hidden side of the house: why does she hear the crying of a unseen child? Why is there an overgrown, walled garden, its door long locked? (Summary by Peter)", | |
| "num_sections": 27, | |
| "sections": { | |
| "1": { | |
| "number": 1, | |
| "name": "01 - There Is No One Left", | |
| "url": "https://ia600204.us.archive.org/15/items/secret_garden_version2_librivox/secretgarden_01_burnett_64kb.mp3" |
ranjitiyer
/ gist:a46fdbf854a015c20ee37104c442322b
Created
June 21, 2018 01:28
Find important files in a project by looking at historic commits
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| git log --oneline | awk '{print $1}' | xargs git show | grep "diff" | awk '{print $3}' | sort | uniq -c | sort -nr | more |
ranjitiyer
/ gist:bcda4fb8f9e15ffea28f22b5083ec634
Created
June 1, 2018 23:56
Git show commit tree as graph
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| git log --oneline --graph --decorate --all |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| export PS1="\@ \w $(if [[ -e .gitignore ]]; then git branch | head -n 1 | awk '{print $2}'; fi) > " | |
| Prints time in am/pm, the current working directory and the checked out gitbranch if pwd is a git repo | |
| 01:39 PM ~/work/emr/sparkmagic jupyter > |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [[Title >>https://aws.amazon.com/blogs/big-data/running-jupyter-notebook-and-jupyterhub-on-amazon-emr]] | |
| Review Vanguard requirements | |
| ======== | |
| {{toc numbered="true" /}} | |
| **bold** | |
| * item | |
| ** subitem |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # Source https://www.linuxtrainingacademy.com/linux-commands-cheat-sheet/ | |
| # | |
| – SYSTEM INFORMATION | |
| # Display Linux system information | |
| uname -a | |
| # Display kernel release information | |
| uname -r |
NewerOlder