This is an implementation of https://github.com/gothinkster/realworld

Got it running on Heroku as of April 2021 at https://cirosantilli-realworld-next.herokuapp.com/. On August 2022, heroku announced the end of free tiers as of November 28, 2022: https://blog.heroku.com/next-chapter, so it might go down, I don’t think I’ll pay 10 dollars/month to keep it going. Here’s an archive: https://web.archive.org/web/20220826111459/https://cirosantilli-realworld-next.herokuapp.com/. More discussion at:

• #23

• gothinkster/realworld#974

This implementaion includes Database migrations, DB and API unit testing, DB seeding, and Linting setups.

This is a branch of https://github.com/cirosantilli/node-express-sequelize-realworld-example-app rebased on top of the master branch of that repository, we show it in a separate GitHub repository just to give it more visibility.

Ideally we would like to have both demos on the same tree as they share all backend code, but Next.js and NPM impose too many restrictions which we don’t have the time to work around.

The frontend part of this repository is a fork of https://github.com/reck1ess/next-realworld-example-app/tree/66003772a42bf71c7b2771119121d4c9cf4b07d4 which was SSR-only via API, not pure SSR/SSG/ISG.

We decided to merge the frontend it in-tree with the API rather than keep it a separate submodule because the fullstack implementation means that the barrier between front-end and back-end becomes is quite blurred, e.g. "front-end" files under /pages also contain functions with direct backend database access without going through the API.

Both Next.js and the backend API run on the same Express.js server via a Next.js custom server, which is basically what you always want for a sane deployment.

The app is by default ISR. We started doing a bit of SSR as well under SSR version but didn’t to very far. Ideally, we would want to implement both for every page, and then benchmark them.

In ISR, user-specific information such as "do I follow this article or not" is fetched by the client after it receives the user-agnostic statically generated page. The client then patches that page with this new user-specific information. This is what you would want do on a real world app, so that things that are common to all users can be cached across all users, and user-specifics are patced on later via a potentially slower API call. See also: What works with JavaScript turned off.

The design goals of this repository are similar to https://dev.to/givehug/next-js-apollo-client-and-server-on-a-single-express-app-55l6 except that we are trying to implement the Realworld App :-) GraphQL would be ideal, but its need is greatly diminished by SSR, which essentially leads to a manual ad-hoc implementation of GraphQL for each page.

This repository is a baseline for the more realworld/advanced/specialized:

• https://github.com/cirosantilli/ourbigbook/tree/master/web

• https://ourbigbook.com

• https://cirosantilli.com/ourbigbook-com

We are trying to port back any problems that are noticed in that repository here, but sometimes we got lazy, and that repo has been getting more code cleanups in general. Also, some features will only be present there because of our constraint of not diverging from the Realworld spec here. Notably that repository contains:

• signup email verification: https://docs.ourbigbook.com/#ourbigbook-web-email-sending

• reCAPTCHA signup protection: https://docs.ourbigbook.com/#ourbigbook-web-recaptcha-setup

• 1. Local development with SQLite
• 2. Local optimized frontend
  • 2.1. Local run as identical to deployment as possible
  • 2.2. Development run in PostgreSQL
• 3. Heroku deployment
  • 3.1. devDependencies vs dependencies
  • 3.2. Demo mode
  • 3.3. Heroku instance management
• 4. ISR vs SSR
  • 4.1. SSR version
  • 4.2. No flickering and automatic updates
• 5. What works with JavaScript turned off
• 6. TODO
  • 6.1. Single URL with multiple pages
  • 6.2. Personal user data in /user/settings
  • 6.3. TODO security
• 7. Known divergences
• 8. Database migrations
  • 8.1. Debugging migrations
• 9. Bugs
• 10. Alternatives
• 11. Keyboard shortcuts
• 12. Debugging
  • 12.1. Step debugging
  • 12.2. VERBOSE environment variable
  • 12.3. Log database queries done
  • 12.4. Generate demo data
  • 12.5. Prevent the browser from opening automatically
  • 12.6. Sequelize sometimes does not show the full stack trace
• 13. Linting
• 14. Testing
  • 14.1. test.js
    • 14.1.1. Next.js tests
  • 14.2. Realworld API tests
• 15. Authentication
• 16. Benchmarks
• 17. Anti-spam
• 18. Optimization
• 19. Directory structure
• 20. Backlinks

You can now visit http://localhost:3000 to view the website. Both API and pages are served from that single server.

You might also want to generate some test data as mentioned at: Generate demo data.

The SQLite database is located at db.sqlite3.

The Node.js and NPM versions this was tested with are specified at package.json engines: entry, which is what the Heroku deployment uses as mentioned here. You should of course try to use those exact versions for reproducibility. The best way to install a custom node and NPM version is to use NVM as mentioned here: https://stackoverflow.com/questions/16898001/how-to-install-a-specific-version-of-node-on-ubuntu/47376491#47376491.

If you make any changes to the code, at least code under /pages for sure, you have to rebuild before they take effect in this mode, as Next.js appears to also run server-only code such as getStaticPaths from one of the webpack bundles.

Running next build is one very important test of the code, as it builds many of the most important pages of the website, and runs checks such as TypeScript type checking. You should basically run it after every single commit that touches the frontend.

If you look into what npm run start-dev does in the package.json, you will see the following environment variables, which are custom to this project and not defined by Next.js itself:

• NEXT_PUBLIC_NODE_ENV=development: sets everything to be development by default.

  If this variable not given, NODE_ENV is used instead.

  Just like NODE_ENV, this variable affects the following aspects of the application:

  • if the Next.js server will run in development or production mode. From the Next.js CLI, this determination is done with next dev vs next start. But we use a custom server where both dev and prod are run from ./app, and so we determine that from environment variables.

  • if the database will be SQLite (default development DB) or PostgreSQL (default production DB)

  • in browser effects, e.g. turns off Google Analytics

    We cannot use NODE_ENV here directly as we would like because and Next.js forces process.env.NODE_ENV to match the server’s dev vs production mode. But we want a production mode server, and no Google analytics in this case.

• NODE_ENV_NEXT_SERVER_ONLY=production: determines is the Next.js server will run in development or production mode.

  This variable only affects the Next.js server dev vs prod aspect of the application, and not any other aspects such as the database used and in browser effects such as having Google Analytics or not.

  If given, this variable overrides all others in making that determination, including NEXT_PUBLIC_NODE_ENV. If not given, NODE_ENV is used as usual.

  If this variable is not given, NEXT_PUBLIC_NODE_ENV is given instead.

2.1. Local run as identical to deployment as possible

sudo apt install postgresql

# Become able to run psql command without sudo.
sudo -u postgres createuser -s "$(whoami)"
createdb "$(whoami)"

createdb realworld_next
psql -c "CREATE ROLE realworld_next_user with login password 'a'"
psql -c 'GRANT ALL PRIVILEGES ON DATABASE realworld_next TO realworld_next_user'
echo "SECRET=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 256)" >> .env

npm run build-prod
npm run start-prod

npm run seed-prod

2.2. Development run in PostgreSQL

npm run dev-pg

REALWORLD_PG=true ./bin/sync-db.js
REALWORLD_PG=true ./bin/generate-demo-data.js

psql realworld_next

The setup is analogous to: https://github.com/cirosantilli/node-express-sequelize-realworld-example-app#heroku-deployment but instead of heroku git:remote -a cirosantilli-realworld-express you should use:

This is done because this repository is normally developed as a branch of that one, which would lead to a conflicting name for the branch heroku.

Additionally, before deploying, you must also make sure that you can run the PostgreSQL tests by setting PG up as shown at: Local run as identical to deployment as possible. This is because before deployment we always run the tests to ensure that nothing broke. And running the tests in PostgreSQL is particularly crucial, because since Sequelize is not stellar, sometimes things work in SQLite but fail in PostgreSQL.

Then, to deploy the latest commit run:

This also pushes the code to GitHub on success, and markes the deployed commit with the deploy branch on GitHub, to mark clearly the deployed commit clearly in case local development moves ahead of the deployed commit a bit.

You then have to add --app cirosantilli-realworld-next to any raw heroku commands to allow Heroku to differentiate between them, e.g.:

for which we have the helper:

e.g. to delete, recreate and reseed the database:

We are not sure if Next.js ISR can be deployed reliably due to the ephemeral filesystem such as those in Heroku…​: https://stackoverflow.com/questions/67684780/how-to-set-where-the-prerendered-html-of-new-pages-generated-at-runtime-is-store but it has worked so far.

3.1. devDependencies vs dependencies

3.2. Demo mode

npm run dev-demo

3.3. Heroku instance management

./heroku.sh psql

./heroku.sh psql -c 'SELECT * FROM "User"'

./heroku.sh run bash

bin/generate-demo-data.js --force-production

./heroku.sh run bin/generate-demo-data.js --force-production

./heroku.sh ps:exec
termit app.js

ISR is an optimization that aims to:

• reduce load times

• reduce server load

Like all optimizations, it makes things more complex, so you really have to benchmark things to see if you need them.

As mentioned at: SSR version, this is one of the main goals of this website.

The main complexity increase of ISR is that you have to worry about React usEffect chains of events after the initial page load, which can be very hard to debug.

With ISR, we want article contents and user pages to load instantly from a prerendered cache, as if the user were logged out.

Only after that will login-specific details be filled in by client JavaScript requests to the backend API, e.g. "have I starred/favorited this article or not".

This could lead to amazing article text loading performance, since this is the same for all users and can be efficiently cached.

The downside of that is that the user could see a two stage approach which is annoying, especially if there is no clear indication (first download, then wait, then if updates with personal details). This could be made even better by caching things client side, and userSWR which we already using likely makes that transparent, so there is hope. Even more amazing would be if it could cache across requests, e.g. from index page to an article! One day, one day, maybe with GraphQL.

Another big ISR limitation is that you can’t force a one-off immediate page update after the user edits a post, a manual refresh is generally needed: vercel/next.js#25677. However, this is not noticeable in this website, because in order to show login-specific information, we are already re-fetching the data from the API after every single request, so after a moment it gets updated to the latest version.

Our organizational principle is that all logged-in API data will be fetched from the toplevel element of each page. It will have the exact same form as the static rendering props, which come directly from the database during build rather than indirectly the API.

This data will correspond exactly to the static prerendered data, but with the user logged in. It will then simply replace the static rendered logged out version, and trigger a re-render.

This approach feels simple enough that it could even be automated in a framework manner. One day, one day.

It is true that the pass-down approach goes a bit against the philosophy of useSWR, but there isn’t much we can do, e.g. / fetches all articles with /api/articles, and determines favorite states of multiple posts. Therefore, we if hardcoded useSWR for the article under FavoriteArticleButton, that would fetch the states from each article separately /api/articles/[slug]. We want that to happen on single article inspection however.

4.1. SSR version

4.2. No flickering and automatic updates

Due to ISR/SSR, all pages of the website that have distinct URLs, which includes e.g. articles and profiles but not "Your Feed" vs "Global Feed, look exactly the same with and without JavaScript for a logged out user.

For the pages without distinct URLs, we don’t know how to do this, the only way we can do it is by fetching the API with JavaScript.

SSR would require <a href elements to send custom headers, so that URLs won’t be changed, which is impossible:

• https://stackoverflow.com/questions/15835783/adding-http-request-header-to-a-a-href-link

• https://stackoverflow.com/questions/374885/can-i-change-the-headers-of-the-http-request-sent-by-the-browser

• https://softwareengineering.stackexchange.com/questions/250602/why-doesnt-the-html-dom-specification-allow-hyperlinks-to-set-an-accept-header

SSG would, in addition to the previous, require specific Next.js support for the above.

You can turn JavaScript off easily on Chromium with this extension: https://github.com/maximelebreton/quick-javascript-switcher which adds the shortcut Alt + Shift + Q to toggle JavaScript.

6.1. Single URL with multiple pages

6.2. Personal user data in /user/settings

Error: No router instance found.
You should only use "next/router" inside the client side of your app.

6.3. TODO security

We aim to make this website look exactly like https://github.com/gothinkster/angular-realworld-example-app/tree/9e8c49514ee874e5e0bbfe53ffdba7d2fd0af36f pixel by pixel which we call "our reference implementation, and have the exact same DOM tree as much as possible, although that is hard because Angular adds a gazillion of fake nodes to the DOM it seems.

We test this by running this front/backend, and then also running angular in another browser tab. We then switch between browser tabs quickly back and forth which allows us to see even very small divergences on the UI.

Some known divergences:

• reference shows "Your Feed" for logged out user, click leads to login. This just feels wrong, not done anywhere else.

• gothinkster/angular-realworld-example-app#202 "No articles are here…​ yet" clearly broken on Angular

• reck1ess had implmented pagination really well with limits and previous/first/next/last, it would be a shame to revert that: gothinkster/realworld#684

Error messages due to API failures are too inconsistent across implementations to determine what is the correct behaviour, e.g. if you patch:

• reference: shows "Loading articles…​" forever, does not inform user about error

• https://github.com/gothinkster/react-redux-realworld-example-app just throws an exception

Database migrations are illustrated under migrations.

Any pending migrations are done automatically during deployment as part of npm run build, more precisely they are run from bin/sync-db.js.

We also have a custom setup where, if the database is not initialized, we first:

• just creates the database from the latest model descriptions

• manually fill in the SequelizeMeta migration tracking table with all available migrations to tell sequelize that all migrations have been done up to this point

This is something that should be merged into sequelize itself, or at least asked on Stack Overflow, but lazy now.

8.1. Debugging migrations

git add migrations && git commit -an && git checkout HEAD~ && bin/generate-demo-data.js && git checkout - && ./bin/sync-db.js

https://github.com/reck1ess/next-realworld-example-app has several UI bugs/missing functionality, some notable ones:

• reck1ess/next-realworld-example-app#22 Your Feed not working. We fixed it at d98637bb10af2bb111f0f2a6ccc72c1de6c8f351.

The implementation of reck1ess/next-realworld-example-app felt a bit quirky in a few senses:

• usage of useSWR even for data that can be already pre-rendered by Next.js such are articles. Presumably this is to give some kind of pool based realtime support? Or maybe it is just part of a workaround for the problem described at Single URL with multiple pages. But that is not what other implementations do, and neither should we. We don’t want data to update by surprise under a user’s feet.

• uses custom emotion-js CSS in addition to the global http://demo.productionready.io/main.css, which is also required since not everything was migrated to emotion.

  We later completely removed motion from this repository.

  And also has a global style.css.

  While this is good to illustrate that library, it also means that a lot of reimplementation is needed, and it is hard to be accurate at times.

  And if it were to use emotion, it should be emotion only, without the global CSS. Instead, that repo uses both, sometimes specifying the same CSS multiple times in two ways.

  It is also very annoying that they used separated defined components rather than in-tree emotion CSS which can be done as:

  <div css={css`
    font-weight: 300;
  `}>

  which leads to a much easier to read DOM tree, and less identifiers flying everywhere.

  It must be said that the port to emotion was made in a way that closely mimicks the original class/CSS structure. But still, it is just too much work, and mistakes popped up inevitably.

These are all points that we have or would like to address in this fork.

• https://github.com/lifeiscontent/realworld: Rails backend is a downside, as it adds another language to the mix besides JavaScript. But it has graphql, which is really really awesome technology.

Ctrl + Enter submits articles.

12.1. Step debugging

npm run back-inspect

npm run front

12.2. VERBOSE environment variable

VERBOSE=1 npm run dev

12.3. Log database queries done

DEBUG='sequelize:sql:*' npm run start-prod

12.4. Generate demo data

./bin/generate-demo-data.js

./bin/generate-demo-data.js --n-users 5 --n-articles-per-user 8 --n-follows-per-user 3

./bin/generate-demo-data.js --empty

12.5. Prevent the browser from opening automatically

BROWSER=none npm run dev

12.6. Sequelize sometimes does not show the full stack trace

The following lint checks are run automatically as part of:

from Local optimized frontend, but it can be good to isolate the command to speed up the development loop.

Run typescript type checks:

Run eslint checks:

These lint checks include both:

• prettier checks, which do style checking. Since it is just style checks, any problems with those can be fixed automatically by prettier’s auto-refactoring functionality with:

  npm run format

• more functional checks, including important checks such as those provided by eslint-config-react-hooks as opposed to more functional thing

Rationale for some rules we’ve disabled:

• @next/next/no-img-element: Next insist that you whitelist servers, which is only possible if we implement profile picture upload: #16 We will actually do this at some point.

• import/no-anonymous-default-export: what’s the point??? It just duplicates module names as pointed in comments. https://stackoverflow.com/questions/64729264/how-can-i-get-rid-of-the-warning-import-no-anonymous-default-export-in-react

When running:

the server runs on a temporary in-memory database when using the default SQLite database.

It has no effect on PostreSQL, as we don’t know of any reasonable alternatives unfortunately. We could grant a create database privilege to our PostgreSQL test user…​ but Sequelize does not seem to support database creation there: https://stackoverflow.com/questions/31294562/sequelize-create-database/32212001.

One implication of this is that it is not currently possible to run test.js in parallel mode for PostgreSQL.

14.1. test.js

npm test

npm test -- -g 'substring of test title'

DEBUG='sequelize:sql:*' npm run test

npm run test-pg

DATABASE_URL_TEST=postgres://realworld_next_user:a@localhost:5432/realworld_next_test npm run test-pg

npm run test-next

npm run test-pg-next

14.2. Realworld API tests

npm run back-test

npm run test-api

npm run test-api -- --folder Register

npmr test-api -- --folder 'Register' --folder 'Login and Remember Token' --folder 'Create Article'

As specified by Realworld, we use JWT authentication.

This can happen in two ways:

• Authentication header ("standard JWT"): sent to the API routes. Immune to XSS. Stored in window.localStorage. Requires JavaScript.

• a cookie that contains a copy of the JWT, used only on safe HTTP methods, notably GET.

  The goal of the cookie auth is allow true SSR, and reduce complexity (useEffect debugging hell). This way, on every page load Next.js immediately sees that the user is logged in, and getServerSideProps is able to return the appropriate page rendered for that specific user by reading the cookie in the request headers. Note that is not possible in ISR.

Because cookies are used exclusively for safe methods, we don’t need to worry about implementing the synchronizer token pattern.

Currently the login page requires JavaScript, so you can only login with JavaScript. But at some point we could enable a non-JavaScript method for that login, which would allow users to view logged-in-only pages too without JavaScript. They just won’t be able to use any non-safe methods. But meh, non-JavaScript is for bots.

Methodology:

• time after click event https://stackoverflow.com/questions/67750849/how-to-filter-by-event-type-in-chrome-devtools-profile-tab-e-g-to-see-mouse-cli/67750850#67750850 up until new page renders, not considering any images on the new page, just text

• caches warmed by clicking all pages involved just before the experiment

• hardware: Lenovo ThinkPad P51

• browser: Chromium 91

That website has no signup verification mechanism, users can just spam it at will via API.

However, until someone decides to spam nonstop 24/7 to the point of actually preventing other users from viewing their own posts, it doesn’t matter that much. Remember that in Demo mode we limit the ammount of articles and comments, so unless we implement further restrictions, a spammer could easily replace all data with their own.

Some things we could do include:

• log IPs. Started doing that at https://github.com/cirosantilli/node-express-sequelize-nextjs-realworld-example-app/commit/f2ee0bea8c081fbd6bb42052a15ed55f3909ab3f on account creation. The only way to check IPs currently is through direct database access on:

  ./heroku.sh psql

  e.g.:

  ./heroku.sh psql -c 'SELECT username, email, ip, "createdAt" FROM "User" WHERE ip IS NOT NULL ORDER BY ip ASC, "createdAt" ASC'

  or maybe to find potential bots by IPs with most accounts:

  ./heroku.sh psql -c 'SELECT ip, COUNT(ip) FROM "User" WHERE ip IS NOT NULL GROUP BY ip ORDER BY COUNT(ip) DESC, ip ASC'

  Of course, IP checks can be overcome with Tor, and blocking IPs is really messy because you can take down entire institutions.

• captcha for signup. Captcha for post creation would be too annoying. This would immediatly block any bots, but not manual spammers.

  hcapcha looks decent: https://docs.hcaptcha.com/. We have to make a request from our server to theirs to verify user login.

• limit number of articles and comments per user. So spammers would need to create new accounts, and therefore redo captchas. 25 posts per account feels like enough.

Some spam events:

• 2021-12-22 https://archive.ph/EVcUw spammed a bunch of pro Chinese government messages, see also: https://github.com/cirosantilli/china-dictatorship and https://cirosantilli.com/china-dictatorship/backlinks

  Did a bit of updating with ./heroku.sh psql:

  ./heroku.sh psql -c "UPDATE \"Tag\" SET name = REPLACE(name, '', '六四事件法轮功新疆再教育營')"

Implementing something something without any efficiency considerations is one thing.

Implementing it with efficiency is another.

We tried a bit to achieve the following, TODOing where we know we failed:

• minimize API calls to the minimum. It can be easy to make multiple unecessary API calls with React if don’t have a clue what you are doing, especially while waiting to decide if we are logged in or not, which must be done from useEffect: https://stackoverflow.com/questions/54819721/next-js-access-localstorage-before-rendering-page/68136224#68136224. We often have to differentiate between: "we are logged off" and "we don’t know our logged in status yet".

• minimize database calls, notably use single JOINs wherever possible, especially on the index page where lots of articles are brought in. This is hard in part due to the inflexibility of sequelize, some notes at: #5 *

• Database migrations related:

  • bin/sync-db.js

  • migrations

• Testing related:

  • test.js: test.js

• front/: files that are safe to import from the frontend, all requires from inside front/ should also be inside front/ see: https://stackoverflow.com/questions/64926174/module-not-found-cant-resolve-fs-in-next-js-application/70363153#70363153 As mentioned there, everything that is usable in the frontend is also usable in the backend since we are an SSR setup. But not the other way around. pages/ can contain a mixture of frontend and backend in each file however when HoCs are not being used and things don’t blow up.

  • back/: this directory contains the backend corresponding to pages in front, e.g. back/ArticlePage contains the getStaticPaths and getStaticProps that correspond to front/ArticlePage. Other backend-only files are in general placed anywhere outside of the front/ directory.

A very large source of traffic to the website has been from Ukraine-based Mate Academy: https://uk.mate.academy/ | https://github.com/mate-academy/conduit-node-express-sequelize-nextjs which seems to use it as one of their webdev tutorial examples. Glad it has been of use to someone, hope their war ends soon…​