Pix2Code

Within three years deep learning will change front-end development. It will increase prototyping speed and lower the barrier for building software. The field took off last year when Tony Beltramelli introduced the pix2code paper and Airbnb launched sketch2code. Currently, the largest barrier to automating front-end development is computing power. However, we can use current deep learning algorithms, along with synthesized training data, to start exploring artificial front-end automation right now!

Try it now

Click this button to open a Workspace on FloydHub that will train this model.

Turning Design Mockups Into Code With Deep Learning

In this notebook, we’ll build a neural network to code a basic a HTML and CSS website based on a picture of a design mockup.

Image from the Blog

We’ll use a dataset of generated bootstrap websites from the pix2code paper. By using Twitter’s bootstrap, we can combine HTML and CSS and decrease the size of the vocabulary.

Instead of training it on the bootstrap markup, we’ll use 17 simplified tokens that we then translate into HTML and CSS. The dataset includes 1500 test screenshots and 250 validation images. For each screenshot there are on average 65 tokens, resulting in 96925 training examples.

By tweaking the model in the pix2code paper, the model can predict the web components with 97% accuracy (BLEU 4-ngram greedy search).

We will:

• Preprocess webpage images and the code related HTML for this mixed NLP-CV task
• Build and train the pix2code model using Keras and Tensorflow
• Evaluate our model on the test set