## Running nightmare on Amazon Linux You may have thought of running nightmare on AWS Lambda. But before we can run it on Lambda, we need first to make it run on Amazon Linux. ### Provision instance which replicates Lambda environment According to AWS Documentation on [Lambda Execution Environment and available Libraries](http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html) we would need this AMI image with this alias `amzn-ami-hvm-2016.03.3.x86_64-gp2`. Keep in mind that AMI-image-id for this instance would be different in different regions (eg): - In `eu-west-1` - `ami-f9dd458a` - In `us-east-1` - `ami-6869aa05 ` But you can find the right one in your "Community AMIs" section of "EC2 Launch Instance wizard". If in your region you find more than one image with this name, you need to pick one with with description `"Amazon Linux AMI 2016.03.3 x86_64 HVM GP2"`. ![image](https://cloud.githubusercontent.com/assets/3438798/23653998/a07cfad0-0326-11e7-9c46-6f9a97de1d06.png) So now you can launch the Amazon Linux AMI in your preferred way, I would [launch Amazon Linux Image via AWS CLI](http://google.com) Now let's connect to the instance via ``` ssh -i ~/.ssh/my-amazon-linux-keypair.pem ec2-user@instance.dns.name __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/ 22 package(s) needed for security, out of 80 available Run "sudo yum update" to apply all updates. Amazon Linux version 2016.09 is available. $ ``` ### Attempt to run nightmare Now let's prepare instance and install some basic tools we need to try to run nightmare. (Note that we will use node 4.3.2 because that's what [Lambda Environment docs say](http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html). ``` sudo yum update -y curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm nvm install v4.3.2 node -v npm -v ``` Now let's install nightmare and try running sample ``` mkdir nightmare-test && cd nightmare-test npm install nightmare # This will automatically install electron executable as dependency ``` So let's try running `example.js`: ``` [~/nightmare-test]$ node node_modules/nightmare/example.js # Nightmare will return without any output on the console. Let's add debug flag: [~/nightmare-test]$ DEBUG=nightmare node node_modules/nightmare/example.js nightmare queuing process start +0ms nightmare queueing action "goto" for http://yahoo.com +3ms nightmare queueing action "type" +2ms nightmare queueing action "click" +0ms nightmare queueing action "wait" +0ms nightmare queueing action "evaluate" +0ms nightmare running +1ms nightmare electron child process exited with code 127: command not found - you may not have electron installed correctly +19ms nightmare electron child process not started yet, skipping kill. +1ms ``` Now we get output like below, from which we can conclude that there's problem with running `electron` executable. But still we don't know what is the root cause of the problem. So let's try to run the `electron` executable manually: ``` [ec2-user@ip-172-31-5-1 nightmare-test]$ ./node_modules/nightmare/node_modules/electron/dist/electron electron: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory ``` ### We found a root cause Bingo! The dependency error was the **first informative error** and told us that `electron` can't run because of missing static library `libgtk` (which is installed by default on most of Desktop Ubuntu distros, but isn't always available on server distros, on CentOS or on Amazon Linux or Lambda). It is logical to assume that in case one library is missing, there will be more. Let's check ``` $ cd node_modules/nigthmare/node_modules/electron/dist [ec2-user@ip-172-31-5-1 dist]$ ldd electron | grep 'not found' libgtk-x11-2.0.so.0 => not found libgdk-x11-2.0.so.0 => not found libatk-1.0.so.0 => not found libpangocairo-1.0.so.0 => not found libgdk_pixbuf-2.0.so.0 => not found libcairo.so.2 => not found libpango-1.0.so.0 => not found libXcursor.so.1 => not found libXdamage.so.1 => not found libXrandr.so.2 => not found libXfixes.so.3 => not found libXss.so.1 => not found libgconf-2.so.4 => not found libcups.so.2 => not found ``` We can see that there's total of 14 static libraries missing. About a year ago Yuanyi wrote great article on [How to run electron on Amazon Linux](https://mockingbot.com/posts/173). By now it is a bit outdated and only helps tackle 11 dependencies and misses few important in 2017 parts, but has definitely has saved me a day or two and showed the right approach (and right versions of the libraries to build from source). However instead of simply rewriting a newer version of the article I have decided to put all of the commands and improvements into a script, so that the process can be automated. The tool is called [`eltool.sh` and is available as gist](https://gist.github.com/dimkir/52054dfca586cadbd0ecd3ccf55f8b98) So as the next step we can download the tool into the home directory: ``` curl -o- https://gist.github.com/dimkir/52054dfca586cadbd0ecd3ccf55f8b98/raw/2b5ebdf28f6a1aad760b5ab9cc581e8ad12a49f5/eltool.sh > ~/eltool.sh && chmod +x ~/eltool.sh ``` Now we can proceed with installing missing electron dependencies and compiling from source certain libraries. The syntax of the tool is `./eltool.sh task1 task2 task3` and task names are made to be self explanatory: ``` $ ./eltool.sh dev-tools # installs gcc compiler and some libs $ ./eltool.sh dist-deps # we install prebuilt dependencies from Amazon Linux repos by using yum $ ./eltool.sh centos-deps # we install some prebuil dependencies we can take from CentOS6 repo # There's still a number of libraries which need to compile from source $ ./eltool.sh gconf-compile gconf-install $ ./eltool.sh pixbuf-compile pixbuf-install $ ./eltool.sh gtk-compile # this will take 3 minutes on t2.small instance $ ./eltool.sh gtk-install ``` Now you have all dependencies available in the system directory /usr/local/lib, but some libraries need to be placed(hardlinked) into same directory as `electron` executable: ``` $ cd ~/nightmare-test/node_modules/nightmare/node_modules/electron/dist # Let's create hardlinks to the required libraries [dist]$ ln -PL /usr/local/lib/libgconf-2.so.4 [dist]$ ln -PL /usr/local/lib/libgtk-x11-2.0.so.0 [dist]$ ln -PL /usr/local/lib/libgdk-x11-2.0.so.0 [dist]$ ln -PL /usr/local/lib/libgdk_pixbuf-2.0.so.0 # or alternatively you can use shorthand [dist]$ ~/eltool.sh link ``` At this point in time you should not have any unresolved dependencies for `electron`, but let's double check: ``` [dist]$ ldd electron | grep 'not found' # Output should be empty ``` ## Let's run Xvfb and nightmare Now all missing electron dependencies are present, let's try to run our `example.js` again: ``` [nightmare-test]$ node example.js # Nothing happens, let's try to add DEBUG flag [nightmare-test]$ DEBUG=nightmare node example.js nightmare queuing process start +0ms nightmare queueing action "goto" for http://yahoo.com +3ms nightmare queueing action "type" +2ms nightmare queueing action "click" +0ms nightmare queueing action "wait" +0ms nightmare queueing action "evaluate" +1ms nightmare queueing action "screenshot" +0ms nightmare running +0ms nightmare electron child process exited with code 1: general error - you may need xvfb +42ms nightmare electron child process not started yet, skipping kill. +1ms ``` If you look carefully at this output, and compare it with the debug output we saw in the beginning of the article, you will notice two subtle differences : - exit code is `1` (and not `127` as was when we had problems with dependencies) - nightmare is suggesting actual error cause - missing Xvfb So let's follow suggestion and install X-server and Xvfb to the instance: ``` sudo yum -y install xorg-x11-server-Xorg xterm # x-server sudo yum -y install xorg-x11-drv-vesa xorg-x11-drv-evdev xorg-x11-drv-evdev-devel # x-drivers sudo yum -y install Xvfb # or alternatively you can use shortcut $ ~/eltool.sh xvfb-install ``` **And now let's finally run nightmare with Xvfb server running in the background**: ``` # Upon successful execution the output should be an url related to nightmare. Any url you see would be sign of success. [nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js https://github.com/segmentio/nightmare # Hurray! This seems to work! ``` ``` # If you're still not convinced we can run it with DEBUG flag [nigthmare-test]$ DEBUG=nightmare xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js nightmare queuing process start +0ms nightmare queueing action "goto" for http://yahoo.com +3ms nightmare queueing action "type" +2ms nightmare queueing action "click" +0ms nightmare queueing action "wait" +0ms nightmare queueing action "evaluate" +1ms nightmare running +0ms nightmare electron child process exited with code 0: success! +8s https://github.com/segmentio/nightmare ``` **CONGRATS! IT FINALLY WORKS!** **CONGRATS! IT FINALLY WORKS!** **CONGRATS! IT FINALLY WORKS!** --- ## Extras: Can we take a screenshot? ### Modify `example.js` to take screenshot To ensure everything really works, you probably do not want to limit yourself with single line of text returned by the `example.js`. Let's add screenshot functionality to `example.js`: ``` // ~/nightmare-test/example-screenshot.js var Nightmare = require('nightmare'); var nightmare = Nightmare({ show: true }) var dt = (new Date()).getTime(); var filename = `/tmp/image-${dt}.png`; nightmare .goto('http://yahoo.com') .type('form[action*="/search"] [name=p]', 'github nightmare') .click('form[action*="/search"] [type=submit]') .wait('#main') .evaluate(function () { return document.querySelector('#main .searchCenterMiddle li a').href }) .screenshot(filename) .end() .then(function (result) { console.log(`Screenshot was saved to filename ${filename}`); console.log('Result: ', result) }) .catch(function (error) { console.error('Search failed:', error); }); ``` ### Run nightmare and make this screenshot ``` [nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node example-screenshot.js Screenshot was saved to filename /tmp/image-1488912962584.png Result: undefined # Let's check size of the screenshot $ ls -lh /tmp/*.png -rw-rw-r-- 1 ec2-user ec2-user 87K Mar 7 18:56 /tmp/image-1488912962584.png ``` ### Send image to email If you still want to look at the actual image, you find many ways to get it to you: scp, upload to S3, send to email. I will show the simplest way to email attachment from linux box using `mutt` command: ``` sudo yum install -y mutt # let's install mutt mail client echo "Sending screenshot" | mutt -s "Nightmare screenshot" email@domain.name -a /tmp/image-1488912962584.png # Be sure to check your Spam folder for this message =) ``` And here's the reward: ![image](https://cloud.githubusercontent.com/assets/3438798/23673327/cc6c621e-0369-11e7-8b34-fb67f75c0db8.png)