tailwindui-crawler

This script will crawl the tailwindui.com website and download all the component HTML to the ./output folder.

🛠 How to use

To install, clone this repo and run yarn or npm install to pull down the dependencies.

Then create a .env file with your email, password, and optional output folder.

EMAIL=youremail
PASSWORD=yourpassword
OUTPUT=/path/to/output # optional, defaults to ./output
HTMLMODE=alpine|comments # save HTML with alpine (default) or comments
TRANSFORMERS=... # comma-delimited list of transformers (see below)
BUILDINDEX=(0 | 1)  # generate index file to view components offline

✨ New in v2.3

The Tailwind UI components have removed the Alpine.js code from the HTML. The crawler now includes a HTMLMODE setting to include either the Alpine.js code (alpine default) or download with HTML comments (comments).

There are also new transformers to change the color and logo of the HTML components.

NOTE: The tool uses dotenv-expand to support variable expansion like $HOME/path/to/output so if your password or any other value includes a $, make sure you add a \ (backslash) to escape the $. For example, [email protected]\$\$w0rd

Then finally, run yarn start or npm start

The script will login to tailwindui.com with your credentials, and download all the components as individual HTML files in the ./output folder.

🤔 What's it for?

The benefit of pulling down all the components is that you can commit them to a local or private repo, and by running this periodically, you can see exactly which files were added or changed. Hopefully, some time in the future, they will open up a private repo for those that have purchased the library.

🚀 New v2.0

The crawler has been re-written to make it easier to update the processing pipeline by simply adding a new transformer function.

It also adds the ability to generate an index page that emulates the tailwindui.com website so you can browse components offline.

🔄 Transformers

Each transformer is simply a JavaScript file (in ./transformers folder) that exports a function to be called from the processing pipeline. Each transformer will take a cheerio instance (basically a jQuery-like interface) which enables the transformer to update the generated HTML. The crawler will call each transformer in turn, then writes the final HTML file.

To add a new transformer, update the TRANSFORMERS key in the .env file. This is a comma-delimited list of transformers. The crawler will call each transformer in the specified order.

The following transformers are availble:

🗂 Preview page

You can set the .env key BUILDINDEX=1 to have the crawler generate an index file similar to the components page on tailwindui.com. Install and run the serve package to view the index.

NOTE: The HTML Preview does not apply transformers. It's a copy of the component site on tailwindui.com. However, the CODE button will show the transformed code. It now includes the ability to resize the iframe. All existing interactivity should work. The COPY function is not available.

yarn global add serve
cd $OUTPUT # change to your OUTPUT folder
serve

⚙️ Example .env file

EMAIL=******
PASSWORD=******
OUTPUT=$HOME/Projects/oss/tailwindui
HTMLMODE=alpine # apline | comments
BUILDINDEX=1    # 0 | 1

TRANSFORMERS=addTailwindCss,prefixSrc,useInter,changeColor,changeLogo,prefixClasses,convertReact,stripAlpine

# addTailwindCss
ADDTAILWINDCSS_URL=http://localhost/path/to/css # defaults to twui CDN

# convertVue
VUE_OUTPUT=$OUTPUT/vue  # path to save Vue files (defaults to $OUTPUT)

# convertReact
CONVERTREACT_OUTPUT=$OUTPUT/react # path to save React files (default to $OUTPUT)

# stripAlpine
STRIPALPINE_OUTPUT=$OUTPUT/no-alpine # path to save stripped HTML files (REQUIRED)

# changeColor
CHANGECOLOR_TO=red # name of color to change from indigo

# changeLogo
CHANGELOGO_URL=http://localhost/path/to/logo # URL of logo (defaults to generic tailwind logo)

# prefixClasses
PREFIXCLASSES_PREFIX=tw- # adds prefix to all tailwind classes

Automatically keep a private GitHub Repository up-to-date

You can automatically keep a private GitHub repository up-to-date with component changes from TailwindUI by using this tool with GitHub Actions.

1. Create a private GitHub repository.

2. Add TAILWINDUI_EMAIL and TAILWINDUI_PASSWORD secrets to the GitHub repository.

3. Optionally create a .env file with additional settings for the crawler.

4. Create a new file .github/workflows/default.yml:

   name: Update
   on:
     schedule:
       - cron: '0 0 * * *' # Every day at midnight
   
   jobs:
     update:
       name: Update
       runs-on: ubuntu-latest
       steps:
         - name: Checkout
           uses: actions/[email protected]
         - name: Run crawler
           uses: gregbrimble/[email protected]
           with:
             email: ${{ secrets.TAILWINDUI_EMAIL }}
             password: ${{ secrets.TAILWINDUI_PASSWORD }}
   

   Read more about the schedule cron syntax in the official GitHub Actions documentation.

Email Notifications

To be emailed whenever there is a change to a component, simply setup GitHub Notifications on your repository.

🚦 Upgrading to v2.

Since the transformers can make a lot of changes to the files, I would recommend you run the current crawler first to generate the diffs and commit those. Then upgrade and run with transformers enabled. This way you're not mixing up changes.

NOTE: Since this script is essentially screen scraping, there's the potential of it breaking if the HTML structure changes. I will do my best to keep it in sync with the website.

😍 Thank you

Thanks to Adam and Steve for making an amazing library. This has definitely made creating a UI for my applications a pleasant experience.

Enjoy and let me know if you have any questions.

Kiliman

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Kiliman 💻

Simon Waloschek 💻

Pavel Fomchenkov 💻

Robin Malfait 💻

Miguel Piedrafita 💻 📖 🤔

Vlad Dumitrescu 📖

C-Bass 💻

This project follows the all-contributors specification. Contributions of any kind welcome!