pix2code: Generating Code from a Graphical User Interface Screenshot

Published in

Engineering Interactive Computing System(2017)

External Links:

Generate Graph

TL;DR

It is shown that deep learning methods can be leveraged to train a model end-to-end to automatically reverse engineer user interfaces and generate code from a single input image with over 77% of accuracy for three different platforms.

Abstract

Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically reverse engineer user interfaces and generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).

Authors

Tony Beltramelli

1 papers

References29 items

StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images

Towards Diverse and Natural Image Descriptions via a Conditional GAN

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

DeepCoder: Learning to Write Programs

pix2code: Generating Code from a Graphical User Interface Screenshot

Published in

Engineering Interactive Computing System(2017)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Tony Beltramelli

1 papers

References29 items

StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images

Towards Diverse and Natural Image Descriptions via a Conditional GAN

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

DeepCoder: Learning to Write Programs

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

TerpreT: A Probabilistic Programming Language for Program Induction

Generative Adversarial Text to Image Synthesis

Latent Predictor Networks for Code Generation

Reverse Engineering Mobile Application User Interfaces with REMAUI (T)

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Deep visual-semantic alignments for generating image descriptions

Show and tell: A neural image caption generator

Long-term recurrent convolutional networks for visual recognition and description

Recurrent Neural Network Regularization

Very Deep Convolutional Networks for Large-Scale Image Recognition

Neural Machine Translation by Jointly Learning to Align and Translate

Generative Adversarial Nets

OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks

Distributed Representations of Words and Phrases and their Compositionality

Generating Sequences With Recurrent Neural Networks

Re-engineering graphical user interfaces from their resource files with UsiResourcer

ImageNet classification with deep convolutional neural networks

Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure

Learning to Forget: Continual Prediction with LSTM

Long Short-Term Memory

Dropout: a simple way to prevent neural networks from overfitting

Lecture 6 . 5 - rmsprop : Divide the gradient by a running average of its recent magnitude

COURSERA: Neural networks for machine learning

Field of Study

Computer ScienceMathematics

Journal Information

Name

Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems

Venue Information

Name

Engineering Interactive Computing System

Type

conference

URL

N/A

Alternate Names

EICS
Eng Interact Comput Syst