While tools like cURL are useful, allowing easy methods to fetch  raw HTML from webpages, they do not execute JavaScript. In fact, cURL only grabs the file located directly at the specified URL and will not download or load any other external files either, such as CSS sheets. For requests that can also render JavaScript and CSS, as well as generally interact with webpages, there are various other tools and libraries that can be used.

PhantomJS vs. Headless Chrome/Chromium vs. Headless Firefox

Overall:

• PhantomJS was basically "the" library for headless browser automation until recently, but is no longer maintained and the realease of headless FireFox and Chrome suggest that it is unlikely to come back to life
• Running headless instances of Chrome/Chromium seems to be the "best" option both in terms of out-of-the-box features and long-term support.
• Headless Firefox seems adequate, but inferior to working with headless Chrome, unless you specifically need to use Firefox.

PhantomJS

PhantomJS has been around since at least 2011 and is, basically, the first popularized headless, scriptable web browser. While it is no longer being maintained (as of 2018), it is still a solid package with good documentation.

PhantomJS Example: Making "Not-Headless" Requests to Google Search Pages

Setup and Installation

1. Install PhantomJS. Running sudo apt-get install phantomjs to install PhantomJS worked fine on Ubuntu Linux. If you're using a different operating system, the installation may vary. See the PhantomJS installation page for more information.
2. Fuck around. If you want to familiarize yourself with PhantomJS further with basic examples, look at the quickstart guide, which is extremely straightforward.

Creat the Script

Create a new file called reacharound.js and paste the following code into it:

var page = require('webpage').create(),
        system = require('system'),
        t, address;

if (system.args.length === 1) {
        console.log('Usage: loadspeed.js [some URL]');
        phantom.exit();
}

address = system.args[1];
page.open(address, function(status) {
        if (status !== 'success') {
                console.log('FAIL to load the address');
        } else {
                console.log( page.content );
                phantom.exit();
        }
});

Then, save and exit the file.

Running the Script

To run the script, execute the following code:

phantomjs reacharound.js https://www.google.com/search?q=dog

Which will print out the page's code after loading and executing any JS.

Running the Script as a "Reacharound"

Since PhantomJS can easily be run from the terminal, the reacharound.js script above can also be easily piped between various BASH/shell scripts. For example, this code: 

phantomjs reacharound.js https://www.google.com/search?q=dog > reacharound-js-output.html

Will load the same page, but save the result into a file. This could incorporated into other scripts, such as a search engine result monitoring script or local directory scraper.

Other Notes and Observations

Command-Line Flags

There are various command-line flags listed in this documentation file, which can simplify some use cases without the need to write code solutions directly to a script.

User-Agent

The default user-agent is:

• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1

And can be changed as specified in this settings documentation file.

Detecting PhantomJS

There is a quality article on how to--at least in theory--detect whether requests are from PhantomJS, as well as an associated presentation embedded below.

Whether there are any sites that actually use these methods is unclear.

Headless Chrome/Chromium

With PhantomJS no longer being maintained, headless Chrome is one of the most common solutions in Current Year, being released initially in 2017.

Installation and Setup

1. Install Chrome and/or Chromium. Running sudo apt-get install chromium-browser or sudo apt-get install google-chrome should work on Ubuntu Linux.
2. [Troubleshooting]. Open Chromium/Chrome at least once before attempting to launch it headless. When writing this article, I ran into some error with Chromium after installing it was first installed to this machine, which then fixed itself after launching Chromium once normally to trigger the first-launch "setup" page. 
3. More documentation. See the "getting started" guide from Google linked above.

Basic Headless Requests

Get Page Source

Here's an extremely basic example that grabs a page's source code, runs the JS, and then saves it to an HTML file. The "getting started" guide linked under "installation and setup" suggests that this command should be "google --headless..." instead of "google-chrome --headless..." or "chromium-browser --headless..."; it may depend on your system. 

google-chrome --headless --dump-dom https://www.google.com/search?q=dog > google-search-headless.html

Or:

chromium-browser --headless --dump-dom https://www.google.com/search?q=dog > google-search-headless.html

Get page as a PDF

google-chrome --headless --print-to-pdf https://www.google.com/search?q=dog

Other Notes and Observations

Puppeteer

For more advanced scripting and automation, note that there is a Node.JS library called Puppeteer that can be used to control headless Chrome programmatically. It can also/alternatively be combined with libraries tools like Selenium webdriver.

Headless Firefox

Headless Firefox is basically the same as headless Chrome, but for Firefox. It was also released in 2017.

Installation and Setup

• Install Firefox

Basic Headless Requests

firefox -headless --screenshot https://duckduckgo.com/?q=rare+clowns

Also see the list of command line options for other flags that can be used.

Other Observations

There are no other observations about headless Firefox. It works correctly. It's CLI has fewer "out of the box" tricks for common tasks than does headless Chrome. If you're using it for more complex testing or automation, you're probably combining it with something like Selenium.

Other Shit

• SlimerJS
• CasperJS
• Selenium WebDriver, which was used in this reCaptcha solver.

In Conclusion

Hell yeah. It's time to use some or all of these various libraries for purposes that are relevant to your use case.

---