Use wget and elfinder together to search for a CSS selector in all pages of a remote site

Earlier this month I introduced Element Finder, a command line tool which lets you search through multiple HTML files and find matches for a CSS selector. This is fine if there is an HTML template for every page in your site. But what if the pages are generated dynamically, for example with WordPress? Or what if you want to search in a remote site?

In these cases you can use `wget` to crawl a website and download all of the available HTML files. When they have been saved, you can search through them with Element Finder.

If you’re using Linux, `wget` should be pre-installed. If you’re on a Mac, the easiest way to install it is via a package manager like Homebrew: `brew install wget`. There are Windows options available too.

For example, lets say we want to search through this WordPress website, http://keegan.st, and see which pages contain elements matching the following selector, which comes from the default Twenty Eleven WordPress theme stylesheet:

.entry-content h1, .entry-content h2, .comment-content h1, .comment-content h2

First, we would use `wget` to download all HTML files from the website:

wget --mirror --adjust-extension --convert-links --accept html http://keegan.st

Or, for short:

wget -m -E -k -A html http://keegan.st

Following is an explanation of the wget options used in the command above:
–mirror, -m: Recursively download all available files.
–adjust-extension, -E: Add a `.html` filename extension when saving the files.
–convert-links, -k: Change the links in the downloaded files so that the stylesheets and images will be loaded from the remote server when the files are opened off the local file system.
–accept html, -A html: Only download HTML files. Don’t download images, videos, stylesheets, etc.

Next, we CD into the directory where the files where downloaded, and run Element Finder with the CSS selector we want to search for:

cd ~/Downloads/keegan.st
elfinder -s ".entry-content h1, .entry-content h2, .comment-content h1, .comment-content h2"

The following screenshot shows the results:

We can use this information to view the pages that would be effected if we were to modify that CSS rule.

Find in Files with CSS Selectors

Most code editing software has a Find in Files tool, where you can search through all of the files in your project for a string or regular expression. But that isn’t much help if you want to find the files which contain HTML elements matching a CSS selector. That’s why I have built Element Finder.

Element Finder searches through your project for any given CSS selector, for example `.layout-fluid #sidebar`, and provides a list of the files which contain elements matching that selector. In this example that would be any element with an ID of `sidebar` which is a descendant of an element with a class of `layout-fluid`.

This is useful for front-end developers because sometimes when we have to modify a CSS rule, it is not clear if the change will have implications on other pages. This is particularly the case for large projects involving multiple front-end developers.

It is easy to use Element Finder from the command line. First you need to `cd` into the directory you would like to search. Then run `elfinder` and pass in the selector you would like to search for as the `-s` argument.
elfinder -s ".layout-fluid #sidebar"

When you press `Enter`, Element Finder will run and display the results:
Searching for ".layout-fluid #sidebar" in 3 files in "/Users/kestreet/Projects/MyProject". Found 2 matches in 2 files.

Element Finder ignores `.git` and `.svn` files by default. You can ignore other files too with the `-i` argument. If you wanted to search all files except for files in the `partials` folder, you would run:

elfinder -s ".layout-fluid #sidebar" -i ".git, .svn, partials"

Element Finder only searches files with an extension of `html` by default. If you would like to search other files, set the `-x` argument. For example:

elfinder -s ".layout-fluid #sidebar" -x "html, shtml, htm"

If you edit in Vim you can call Element Finder directly from your code editor. Just run:

:!elfinder -s ".layout-fluid #sidebar"

How it works

Element Finder is built with Node JS. JavaScript was designed for interacting with web pages so it is a natural fit for this app. Another benefit is that front-end developers know JavaScript, so it should be easy for other people to fork Element Finder and change it to suit their needs.

Element Finder utilises a number of useful Node JS modules for building command line apps, such as TJ Holowaychuk’s commander.js and node-progress. Elijah Insua’s jsdom module is used to construct a W3C conforming DOM for each HTML file we need to search. Sizzle JS is used to find matches for the search input in the DOM. Sizzle JS is jQuery’s selector engine, which means Element Finder will consistently return the same elements as the jQuery selectors in your JavaScript.

Looking forward

It would be useful to have Element Finder implemented as a plugin in Vim, Sublime Text and other code editors. But in the meantime I hope you will find it useful as a command line app. Technically the app should work in Mac, Windows and Linux, but so far I have only tested it on a Mac. For installation instructions, visit Element Finder on GitHub.