Last active
March 23, 2018 19:44
-
-
Save andrewrgoss/5121c78a3c98c7caac81146d2f8556a5 to your computer and use it in GitHub Desktop.
Console output from MovieSimilarities1M.scala - creating similar movie recommendations from one million ratings, run on AWS EMR cluster
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| mymachine:~ andgoss$ ssh -i ~/.credentials/ag-spark.pem [email protected] | |
| Last login: Thu Mar 8 16:53:31 2018 | |
| __| __|_ ) | |
| _| ( / Amazon Linux AMI | |
| ___|\___|___| | |
| https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/ | |
| 3 package(s) needed for security, out of 5 available | |
| Run "sudo yum update" to apply all updates. | |
| EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR | |
| E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R | |
| EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R | |
| E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R | |
| E::::E M::::::M:::M M:::M::::::M R:::R R::::R | |
| E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R | |
| E::::::::::::::E M:::::M M:::M:::M M:::::M R:::::::::::RR | |
| E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R | |
| E::::E M:::::M M:::M M:::::M R:::R R::::R | |
| E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R | |
| EE:::::EEEEEEEE::::E M:::::M M:::::M R:::R R::::R | |
| E::::::::::::::::::E M:::::M M:::::M RR::::R R::::R | |
| EEEEEEEEEEEEEEEEEEEE MMMMMMM MMMMMMM RRRRRRR RRRRRR | |
| [hadoop@ip-172-31-67-129 ~]$ pwd | |
| /home/hadoop | |
| [hadoop@ip-172-31-67-129 ~]$ aws s3 cp s3://ag-spark/ml-1m/movies.dat ./ | |
| download: s3://ag-spark/ml-1m/movies.dat to ./movies.dat | |
| [hadoop@ip-172-31-67-129 ~]$ aws s3 cp s3://ag-spark/MovieSimilarities1M.jar ./ | |
| download: s3://ag-spark/MovieSimilarities1M.jar to ./MovieSimilarities1M.jar | |
| [hadoop@ip-172-31-67-129 ~]$ ls -ltra | |
| total 260 | |
| -rw-r--r-- 1 hadoop hadoop 331 Feb 13 19:03 .bashrc | |
| -rw-r--r-- 1 hadoop hadoop 85 Feb 13 19:03 .bash_profile | |
| drwxr-xr-x 2 hadoop hadoop 4096 Feb 13 19:06 .aws | |
| -rw-rw-r-- 1 hadoop hadoop 171308 Mar 8 15:33 movies.dat | |
| drwxr-xr-x 4 root root 4096 Mar 8 16:35 .. | |
| drwx------ 2 hadoop hadoop 4096 Mar 8 16:36 .ssh | |
| -rw-rw-r-- 1 hadoop hadoop 68515 Mar 8 17:08 MovieSimilarities1M.jar | |
| drwxr-xr-x 4 hadoop hadoop 4096 Mar 8 17:08 . | |
| [hadoop@ip-172-31-67-129 ~]$ spark-submit --class com.andrewrgoss.spark.MovieSimilarities1M MovieSimilarities1M.jar 260 | |
| Loading movie names... | |
| 18/03/08 17:10:38 INFO GPLNativeCodeLoader: Loaded native gpl library | |
| 18/03/08 17:10:38 INFO LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev cfe28705e7dfdec92539cc7b24fc97936c259a05] | |
| [Stage 2:> (0 + 8) / 16]18/03/08 17:12:45 WARN ServletHandler: | |
| [Stage 2:=============================> (8 + 8) / 16]18/03/08 17:14:46 | |
| [Stage 7:========================================> (72 + 8) / 100]18/03/08 17:28:55 | |
| Top 50 similar movies for Star Wars: Episode IV - A New Hope (1977) | |
| Star Wars: Episode V - The Empire Strikes Back (1980) score: 0.9897917106566659 strength: 2355 | |
| Raiders of the Lost Ark (1981) score: 0.9855548278565054 strength: 1972 | |
| Star Wars: Episode VI - Return of the Jedi (1983) score: 0.9841248359926177 strength: 2113 | |
| Indiana Jones and the Last Crusade (1989) score: 0.9774440028650038 strength: 1397 | |
| Shawshank Redemption, The (1994) score: 0.9768332708746131 strength: 1412 | |
| Usual Suspects, The (1995) score: 0.9766875136831684 strength: 1194 | |
| Godfather, The (1972) score: 0.9759284503618028 strength: 1583 | |
| Sixth Sense, The (1999) score: 0.974688767430798 strength: 1480 | |
| Schindler's List (1993) score: 0.9746820121947888 strength: 1422 | |
| Terminator, The (1984) score: 0.9745821991816754 strength: 1746 | |
| Back to the Future (1985) score: 0.9743476892310179 strength: 1845 | |
| Fugitive, The (1993) score: 0.9740503810950097 strength: 1429 | |
| Princess Bride, The (1987) score: 0.9737384179609926 strength: 1657 | |
| Matrix, The (1999) score: 0.9732130645719457 strength: 1908 | |
| Butch Cassidy and the Sundance Kid (1969) score: 0.9731825975678353 strength: 1048 | |
| Hunt for Red October, The (1990) score: 0.9731286559518592 strength: 1229 | |
| Casablanca (1942) score: 0.9730078799612648 strength: 1113 | |
| Saving Private Ryan (1998) score: 0.9729484985516464 strength: 1709 | |
| Ghostbusters (1984) score: 0.9726721862046535 strength: 1447 | |
| Die Hard (1988) score: 0.9724843514829112 strength: 1369 | |
| L.A. Confidential (1997) score: 0.9722077641949141 strength: 1416 | |
| Toy Story (1995) score: 0.9721270419610062 strength: 1382 | |
| Stand by Me (1986) score: 0.9718025936506943 strength: 1212 | |
| Close Encounters of the Third Kind (1977) score: 0.9717491756795117 strength: 1242 | |
| Monty Python and the Holy Grail (1974) score: 0.9717238750026624 strength: 1248 | |
| Silence of the Lambs, The (1991) score: 0.9714472073187363 strength: 1587 | |
| Wizard of Oz, The (1939) score: 0.9713633100564869 strength: 1346 | |
| Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963) score: 0.9713269232938938 strength: 1149 | |
| One Flew Over the Cuckoos Nest (1975) score: 0.9708527915400245 strength: 1125 | |
| Ferris Bueller's Day Off (1986) score: 0.9705811698208009 strength: 1073 | |
| Godfather: Part II, The (1974) score: 0.9704073574007531 strength: 1246 | |
| Terminator 2: Judgment Day (1991) score: 0.9703674024729073 strength: 1889 | |
| E.T. the Extra-Terrestrial (1982) score: 0.9702456868065551 strength: 1714 | |
| [hadoop@ip-172-31-67-129 ~]$ exit | |
| logout | |
| Connection to ec2-34-236-254-148.compute-1.amazonaws.com closed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment