Technology Stack

Why?

As a team of people using technology to make digital products,
it's essential to be unambiguous about the technology/tools we use,
so that everyone is clear what we all need to learn/use
to make product(s) that are functional, fast, beautiful, useable and reliable!

If anything is unclear or you have any questions please ask.

What?

This document + diagram describes the full "PETE" Technology Stack we use for dwyl products and services (clients).

Each element in our stack was carefully selected based on its individual merits.
When assembled into a seamless "machine", the stack is unrivaled for developer productivity and world-class quality!

The "PETE" Stack

"PETE" is an acronym¹ for the following elements:

• Phoenix is a Web Application Framework that does not compromise on speed, reliability or maintainability! Phoenix is the "successor" to the incredibly popular "Ruby-on-Rails" framework. Built from scratch by highly experienced engineers who worked on/with Rails. It solves all of the speed/socket/scaling/concurrency, issues people felt when building/using Rails apps. The list of benefits Phoenix has over (virtually every) other Web Frameworks is extensive.
  Please see: github.com/dwyl/learn-phoenix-framework#our-top-10-reasons-why-phoenix
  

• Elixir is the functional programming language used by the Phoenix framework. Elixir is a beautiful language written from scratch to be friendly, concise and efficient. Yes, Elixir not as "popular" as JavaScript, Java, C# or PHP, but the adoption is growing rapidly and most importantly many experienced developers are gravitating towards and describing it as their "most wanted" Also a language's popularity has more to do with the intellectual inertia people/companies have because they allow existing (legacy) codebases to dictate future development; i.e. sunk cost bias. see: github.com/dwyl/learn-elixir#key-advantages

• Tachyons is the most sane way of creating a beautiful web app UI that can easily be extended by a team of people without fear of one person's change "breaking" another feature. Unlike "traditional" CSS which - as it's name implies - encourages "cascading" of styles, Tachyons makes the style of each component specific and local to that component. see: https://github.com/dwyl/learn-tachyons

• Elm is a front-end developer's dream made reality by a community of unbelievably brilliant and dedicated developers.
  While Elm is incredibly fast the reason we love elm has nothing to do with "benchmarks" and everything to do with the experience of writing/reading/maintaining an app built with elm; that experience is like a reassuring hug from a loved one.
  When you discover elm, things that used to be difficult or "error-prone" become easy and reliable. See: github.com/dwyl/learn-elm#why

Elm has considerably better performance than similar/alternative front-end frameworks (e.g: Angular, Ember, React, Vue.js etc.) so hopefully the "performance" discussion can "go away"!

¹Thanks to Jimmy Ruts for coining the "PETE" acronym. He's the guy we go to for naming and other "hard things!" ;-)

Database?

The reason we do not specify our Database in the "PETE" Acronym is that Phoenix allows us to use any Relational Database.

By abstracting the data layer using "Ecto" the application is "decoupled" from the database.
This means that if a client asks us to deploy to MySQL or Microsoft SQL Server
(e.g. because they already have in-house capability for maintaining one of these databases)
we can easily accommodate that without re-writing any of the Phoenix app!

We Prefer PostgreSQL (Postgres)

Our "standard" (preference) @dwyl is for Postgres. see: github.com/dwyl/learn-postgresql
Postgres is the most "mature" Open Source Relational Database. It's 100% Free (including all "advanced" features)
and has been deployed and battle-tested in every environment from AWS to "Bare Metal" and Google Cloud to Microsoft Azure!

Many well-known/successful apps rely on Postgres as their main database.
NOT that you should adopt a particular technology based on who else is using it,
but it's good to know that plenty of teams are getting excellent results with PostgreSQL!

See: List of Organizations Using PostgreSQL

We have used most of the "popular" Relational Databases. e.g: MySQL, MS SQL Server, Oracle and Aurora; all RDBMS have their pros/cons.
The reason we like/use Postgres is because the community is superb. There is a great "bank" of answered questions on StackOverflow and new questions get answered fast.

Operating System?

A "traditional" LAMP stack includes the Linux Operating System in the name.
The "PETE" stack runs on any (desktop/server) Operating System
and can be deployed to any "cloud" infrastructure provider.

While we have a strong preference for Unix (e.g. FreeBSD) or Linux (e.g. Ubuntu or CentOS) we know that
both Phoenix and Postgres run on almost any environment including Microsoft Windows Desktop & Server.

Continuous Integration

We use and recommend Travis-CI for Continuous Integration (CI). If you or anyone on your team are new to Travis-CI, please checkout our complete beginner's guide: github.com/dwyl/learn-travis

Deployment

We make a point of deploying our work as soon as there is something worth showing to the target audience of "end users" so that we can get feedback as early as possible.

"Development" and Testing

We use Heroku for all our development deployment. It's by far the most time-effective way to "ship"
the project and get it tried & tested by real people. Using heroku is like having a "Dedicated DevOps" person on the team.
Once the deployment is configured we can automatically deploy a new version of app for each feature and test incrementally.
This alone is worth the modest fee Heroku charges for their service!

If you are completely new to Heroku you can get started in 10 minutes with our
complete beginner's tutorial: https://github.com/dwyl/learn-heroku

For a step-by-step guide to deploying a Phoenix ("PETE") application to Heroku,
please see: github.com/dwyl/learn-phoenix-framework/heroku-deployment
(special thanks to Tony & Jimmy for doing a superb job putting the guide together!)

"Under the Hood" Heroku is using AWS for their infrastructure.
So when we need to "graduate" an app from Heroku to AWS it's easy!

"Production"

Our preference is to deploy to Amazon Web Services (AWS) because we have good experience/knowledge of the platform.

Microsoft Azure

One of our projects has a requirement to be deployed to Microsoft Azure Cloud. We achieve this with the following configuration:

Application Server

The Phoenix Application Server is hosted on (a minimum of) Two Linux Servers
(often many more which "message" one another to distribute load as a cluster).
The "cluster" is managed by Erlang's "Supervisor". The Erlang Supervisor is the "Gold Standard" in infrastructure management, having been used by Telecoms companies for over 20 years in production with some Telcos reporting 99.99999% ("seven nines") of "up-time".

It's far more likely that the infrastructure provider (e.g. AWS/Azure) will have a fault in their network/datacenter than an Erlang server "crashing".

Database Cluster

Thanks to Esmaeil Sarabadani of Haufe.com there is a "Template" for deploying a Highly-Available PostgreSQL Cluster on Azure this can be setup with only a few clicks and scaled automatically to as little or large as required! See: http://dev.haufe.com/PostgreSQL-Cluster-Azure/

SSL/TLS Encryption

All communication is over secure/encrypted channel (by default at all times)
to protect the data/privacy of people using the applications we make.

We recommend using the "Let's Encrypt" service for SSL Certificates it's 100% Free (and provided by a Non-Profit foundation)
to help you get started, we wrote a step-by-step setup guide for apps deployed to Heroku: SSL-certificate-step-by-step-setup-instructions.md

Beginner Tutorials?

We have crafted a "Complete Beginner's Guide" for each element in the stack, so that we:

• Document our collective learning while we are building projects. (because as humans we forget fast unless we write it!)
• Share our knowledge with other people so we can
  • Help to train (potential) new team members as quickly/effectively as possible.
  • Collectively iterate on our knowledge and "level-up" as a team!
  • "Onboard" the client team (who may want/need) to support/maintain the codebase/project if/when we seamlessly "hand over".
  • Inform the wider community of both technical and non-technical people ("stake holders") who are generally interested in understanding the project.
  • Enlighten other teams/organisations/agencies/etc. we aren't in direct contact with that there is a "more fun" way of building software!
• Make everyone's life easier/better by having a "launch pad" for rapid learning!

For an example project where this "Stack" is implemented see: https://github.com/healthlocker/healthlocker

Node.js Stack

We have deployed our Node.js stack for many clients and internal apps and achieved great results!
It works really well and we have not had any issues with "performance" or "scaling" deploying to AWS.

For examples of apps built using our Node.js Stack
see: https://github.com/TheScienceMuseum/collectionsonline
and https://github.com/emfoundation/ce100-app

There's no "reason" to "rewrite" any of our existing projects to any other "stack".
Node.js works perfectly well and will continue to be supported for the lifetime of the project(s)

See: "tl;dr" section below if you are interested in why we decided to "evolve" to "PETE" for new projects...

Overview

The following diagram is an overview of our Node.js stack:

Note: To edit/improve this diagram: https://github.com/dwyl/technology-stack/issues/1

We have produced a complete beginners guide for each of the components in our stack. (see below)

Open Source Projects We Use

For Us By Us

We craft code to scratch our own itch and everything we do is always Open Source

Project	Used For	Build Status	Test Coverage	Dependency Status	Tutorial
env2	Loading Environment Variables				learn-environment-variables
esta	ElasticSearch CRUD				learn-elasticsearch
goodparts	Consistent Code (Linting & Style)				goodparts#why
hapi-auth-jwt2	Authentication & Sessions				learn-json-web-tokens
hapi-error	Human-Friendly Error Messages				hapi-error#why
hapi-login	User Login				learn-hapi
hapi-postgres-connection	Postgres Connection Pooling				learn-postgresql
hapi-redis-connection	Simplify Redis Connection				learn-redis
hapi-register	User Registration				learn-hapi
hapi-riot	Server-side (Fast) Rendering of Riot Tags				learn-riot

Note: to update this table, use the script: generate_dependency_table.js

Dependencies (projects built by people we trust)

• Node.js - the most popular JavaScript runtime for easily building fast, scalable simple network applications. Lightweight and efficient, perfect for data-intensive real-time apps. http://nodejs.org/
• Hapi.js - A rich web framework for building applications and services. https://github.com/dwyl/learn-hapi
• Socket.io - a JavaScript library for realtime web applications. It enables realtime, bi-directional communication between web clients and server. Socket.io lets us send data to/from everyone connected to our app(s) without having to refresh the web page. http://socket.io/
• Riot.js - is the most light-weight user-interface (UI) framework available which is compatible with IE 8/9 and has good server-side rendering (which means pages load faster for slow devices like budget smart phones). see: https://github.com/dwyl/learn-riot
• Redis - the most popular in-memory data store which is essential
  for building the fastest possible apps. read more: https://github.com/dwyl/learn-redis
• ElasticSearch - the most feature-rich search engine. we use it to find things fast. Learn more: https://github.com/dwyl/learn-elasticsearch

Development Dependencies

We carefully select and only use well-maintained "pure" JavaScript modules in our development toolchain:

• Tape for testing: https://github.com/dwyl/learn-tape
• Istanbul for Code Coverage: https://github.com/dwyl/learn-istanbul
• Pre-commit for ensuring all commits pass strict quality checks before being pushed to GitHub. see: https://github.com/dwyl/learn-pre-commit
• GoodParts "linting" and checking code style is consistent: https://github.com/dwyl/goodparts
• CodeCov for detailed test/code coverage stats https://github.com/dwyl/learn-istanbul#tracking-coverage-as-a-service
• CodeClimate for tracking code quality: https://github.com/dwyl/learn-codeclimate

tl;dr

There is no shortage of options available for Technology Stack!
See: https://www.google.com/search?q=technology+stack&tbm=isch
So, how did we arrive at the conclusion that "PETE" was "the one" for us...? We already had a really good Node.js Stack which worked well for us and our clients. so . . .

Why Try a "New Stack"?

Why Try Something New When We're Already Good with the "Old"...?

Our reasoning for considering an alternative approach/stack for building web apps was fueled by our curiousity and "shoshin".

"The important thing is not to stop questioning. Curiosity has its own reason for existence." ~ Albert Einstein
~ "Old Man's Advice to Youth: 'Never Lose a Holy Curiosity.'" LIFE Magazine (2 May 1955) p. 64”

In November 2016 we (once again) questioned our assumptions, re-examined and surveyed the "landscape" of "emerging trends" in web app development. We were pleasantly surprised delighted to see the amazing progress made by the people in the Elixir, Phoenix and Elm communities! These technologies are set to "take off" in 2017 and we are excited to be sharing the technical/competitive advantage with our clients!

Please see: https://github.com/dwyl/learn-phoenix-framework#questions

Making Difficult Decisions

One of the most "difficult decisions" you will make in your "career" is which technologies and tools you will use to deliver the desired solution/benefit to the "end users".

Most people have the Tech/Tools decision made for them by the company/organisation/boss they work for (e.g: Java -> Spring, Ruby -> Rails or PHP -> WordPress or Symphony, etc.) This is because most companies already have an existing app in "production", which you have been hired to extend.

Occasionally you will get the chance to build an app from "scratch"
however most of the time someone else (the "Architect") will make the decision for what "stack" to use on your behalf, so you will still be stuck with someone else's choices. If you are incredibly lucky the "Architect(s)" will have done their homework: surveyed the latest industry trends and investigated the "new and promising" technologies e.g: Stack Overflow "Most Wanted" list.

Most "Application Architects" will pick one of these 3 options:

1. Go with what you (already) know, use existing stack with a minor variation because it's "easy to deploy" with the existing infrastructure and will not get questioned by the "Executives", DevOps team or "Compliance" department. This is the easy choice and nobody ever got "fired" for sticking with what they know "works".

2. Buy the whizz-bang all-in-one solution sold to them by the "Consultant" from "Big Vendor XYZ" (outsource the thinking to a sales person who last wrote code in the 90's ... seems like a great idea ... NOT!)

3. Be "Bold" and try "Popular Framework XYZ" and hire an external team to build the new magic app. Then attempt to "up-skill" the internal team to maintain the code written by the consultants.

None of these choices is optimal, all have different levels of risk/reward. The "hardest" choice to make is the one where you try something totally different The reality is that very few people have the time/resources/mindset/inclination to take a step back and open their minds to the idea that there might be a "better tool" for the job than the one they are currently using.

Toast Knife Analogy

Imagine Want to Make Yourself Some Toast. Let's write a quick "user story" for this:

As a peckish person
I want a slice of tasty toast
So that I can have some crunch for my brunch!

(let's assume that you bought a loaf of bread from a baker to reduce the options for solutions for simplicity (i.e. not baking from scratch!)

The "traditional" way to "solve" the challenge of making toast:

• Cut bread with bread knife
• Put sliced bread in toaster
• Turn on toaster for pre-determined amount of time
• Wait patiently for toaster to make toast

But ... what if instead the "old" way we just described, someone invented a way to toast the bread while slicing it...?!

Simply by using the "New Tool" - in this case the "Toast Knife" - you can simplify the process to a single step!
This is the power of being open to considering "New" Tools/Technologies! Get the same result in fewer than half the "steps"!

Focussing on Long-term Benefits

The short-term cost of learning a new stack (programming language or framework) is time, We contend that the 1-3 week learning time (depending on the focus of learners) will pay for itself within 3 months (often sooner if the team is large/distributed because the structure offered by Phoenix means everyone working on the project has a disciplined approach to adding new features)

Further Reading on Long-term Thinking

• https://hbr.org/2012/08/thinking-long-term-in-a-short
• https://hbr.org/2011/03/capitalism-for-the-long-term

Contextualising Technology Adoption (Mini History Lesson)

In 1996 Nokia introduced the "Communicator" and was a incredible revolutionary innovation! Internet, Email and Fax in your Pocket!!

Nokia continued to dominate the mobile phone industry/market for the next decade producing the best-selling 5110 and 3310 we all remember!
But by being "ahead" Nokia was unable to see the "contender" coming "out of nowhere" to challenge their position.

In 2006 nobody was making/buying "smart" mobile phones with glass touch screens that ran "apps" ...
in January 2007 Steve Jobs introduced the iPhone and literally changed the industry!

The dominant/incumbent mobile phone maker Nokia had 49% market share in 2007 mocked Apple's lack of features, poor battery life and high price.
By 2013 Nokia had 3% Market Share (for new device sales) and was sold off "for parts" to Microsoft while Apple was the most valuable company in history!

Many people still buy "feature phones" (the polite name for a device that does not have any "smart" functionality!)
but few people can convincingly argue that the reason they don't have a smart phone is because they don't want one;
over 90% 16-24 year olds own a smart phone and that number is expected to reach 100% by 2020 ...
ask someone in their 20's if they would go "back" to using a non-smart phone and see what they say ... "No Way!!"
they laugh uncomfortably and admit that: "My Smartphone is my Life!"
We feel exactly the same about "old technology". Sure the "old way still works", but if you can inexpensively switch
to something demonstrably better in every aspect, why would you stick with the "feature phone" of web frameworks...?
It's like taking the Bus to work when there's a perfectly good teleporter right next to the bus stop!! Madness.

We are not suggesting that everyone is going to suddenly flock to the "PETE" stack the way people adopted smart phones. This is merely an illustration that when a technology has a specific/measurable advantage to it's users the adoption can be fast.

In the case of programming languages or application frameworks, moving one framework to another is a much more difficult decision.

But one thing is for sure we are going to use the "smart phone" even if other people insist on using the "brick".

But Phoenix Uses Node.js for Static Asset Compilation ...

Yes, if you are using the (http://brunch.io/) "static asset compilation". We aren't using it for our project(s) because our only "static assets" are the Elm files which we compile using elm-compiler. The elm compiler is written in Haskell so technically we aren't writing any JS in our PETE projects.

Does it Scale?!?

If you are new to web development, please focus on UX and forget about "scale"!

Unless you work somewhere that already has "millions of users" and
your team cannot consider anything that does not support a million concurrent connections...!

But let's face it, most people have imaginary scaling issues not real ones.
discussing "scalability" before you have 10,000 paying customers is a waste of time!!

Stop worrying about "scalability" and instead focus on building something useful
focus on User Experience not "backend" scalability!

The good news is that Phoenix "scales" really well!
see: http://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections

Forget about "scaling" until you have made something people want and are paying for!
Then use the pile of cash you got from your product to hire "engineers" to make it available to more people!!

No JavaScript in "PETE"...?

This not the place to "diss" JavaScript; plenty of people have written blog posts/tweets "ranting" about the "State of Web Development".
e.g: Douglas Crockford (the authority on JS) recently gave a presentation on The Post JavaScript Apocalypse: https://youtu.be/6Fg3Aj9GzNw
(in which he describes all the "features" of JS that are "unnecessary"...)
Or Max Ogden (a prominent JS developer) who created a website dedicated to the "Callback Hell" issue: http://callbackhell.com
why do we even need to have an expression for that? why can't elegant asynchronous control flow be the only way to write code?

The fact is: we only use JavaScript because it is the "Lingua Franca" that all web browsers "understand".
It's definitely not because it's a "better" language than Python, Julia or Lisp; we write JS out of necessity not choice.

With Elm we no longer need to write our Client-side (progressive enhancement) code in JavaScript! We can write in a beautiful/functional language
and "compile" it to JS for running in Browsers. The JS that is produced by the Elm compiler is almost always more efficient/faster
than "hand-written" JS, so why would we waste our time with writing JavaScript...?!

If you aren't .ready() to try something (way) more productive than JavaScript, .then please just ignore this!
We have written this "Technology Stack" description for ourselves and not because we want to "convince" anyone.
But ... we wanted to put down our thoughts in case anyone is "on the fence"! If you're in any doubt, Just Do it.

What About Full Stack JavaScript?

We still think that "Full Stack JavaScript" is a compelling proposition especially for people who are just starting out! If you are learning programming and want a "quick start" try Meteor.js!!

Alternative Databases?

If we were to consider an alternative to SQL, we would use RethinkDB: https://rethinkdb.com
But we are relieved that the Phoenix team is focussed on PostgreSQL because that eliminates
the "ambiguity" or "discussion" of "which database" to use! Postgres is a fantastic "general purpose"
store that has a rich ("structured") query language that lets you JOIN data!!
Also, now that Citus DB is Open Source we know that Postgres can easily handle billions of writes per day!