Real-Time Scalable GraphQL and JAMstack with Gatsby, Hasura, and YugabyteDB

JAMstack is a new way of building websites and apps. It’s not a technology but rather an architectural pattern that is growing in popularity. In JAMstack, the JAM acronym stands for JavaScript, API, and Markup, and the main idea behind the technology is that web applications don’t have to rely on the application server to be fully functional and robust.

There are four primary benefits of adopting a JAMstack architecture.

• Higher performance driven by efficient use of static assets and Content Delivery Networks (CDNs).
• Stronger security resulting from the delegation of server-side and database operations to other expert teams.
• Cost-effective scaling that is easier to manage than running dedicated servers.
• Better developer experience that results from reduction of bloated application logic.

The popularity of JAMstack has risen along with the wide adoption of static site generators, one of which is GatsbyJS. In this post, we will show you how easy it is to get started with JAMstack, Gatsby, and a distributed SQL database on the backend, in order to quickly build and deploy a modern site that easily scales. To get started, we will use a Gatsby starter template to create a basic blog site. To do so run

npx gatsby new gatsby-starter-blog https://github.com/gatsbyjs/gatsby-starter-blog

There is a lot of content out there on various aspects of Gatsby, but in this blog I will focus on three main things:

• Adding dynamic functionality to a Gatsby site using GraphQL subscriptions
• Creating a GraphQL CRUD backend using Hasura engine that will support these subscriptions
• Scaling the database tier powered by Hasura using YugabyteDB, a PostgreSQL-compatible distributed SQL database

Let’s start from the third bullet point and work towards the first.

Setting up YugabyteDB as our database

So what is YugabyteDB and why should we definitely consider using it? I wrote a blog post describing why YugabyteDB is a great fit to run Hasura on top of it so you are welcome to check it out.

In a nutshell, with YugabyteDB we have a distributed SQL database with built-in sharding/rebalancing and fault-tolerance mechanisms. We have multiple nodes in a database cluster and if one of these nodes is down, data is still intact because it is always automatically replicated across multiple nodes.

Let’s see it in action. We will start by setting up YugabyteDB on macOS following the official tutorial, or you can follow my blog post about YugabyteDB as well.

When we have everything configured, we can create our YugabyteDB cluster by running

./bin/yb-ctl --rf 3 start

In our example, we will use the fact YugabyteDB is compatible with the SQL dialect and wire protocol of PostgreSQL, which means Hasura can run on top of it pretty easily. If you are not familiar with hasura.io, you can check out a blog post I wrote about Hasura.

Now it’s time to set up Hasura and run it on top of our freshly created YugabyteDB cluster. Instead of using Heroku as a hosting platform with the PostgreSQL addon, as advised in the official Hasura docs quick start, I will start with the Docker compose example and alter it to run with YugabyteDB. The alteration will basically be removing the PostgreSQL container section from the docker-compose.yaml file and changing the database to be:

postgres://postgres:@host.docker.internal:5433/yugabyte

Alternatively, you can just create a blank docker-compose.yaml file in the Gatsby project root folder and paste the following code:

version: '3.6'
services:
  graphql-engine:
    image: hasura/graphql-engine:v1.1.0
    ports:
    - "8080:8080"
    restart: always
    environment:
      HASURA_GRAPHQL_DATABASE_URL: postgres://postgres:@host.docker.internal:5433/yugabyte
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
volumes:
  db_data:

Now you can run docker-compose up -d in the project root folder to bring up your Hasura container. You will be able to access localhost:8080/console and see that it’s loading. Also, you can access Yugabyte admin UI on localhost:7000 and see that there are a bunch of Hasura specific tables that have been created, such as hdb-catalog and more.

Before jumping into data modeling, we need to make sure that when you model the data, you can track database changes, relationships, etc.

Using Hasura CLI and migrations

The Hasura CLI is a command line tool for managing Hasura projects and migrations. We will use this tool to create migration files whenever we make any changes to the console, so, later on, you will be able to just create your own cluster after forking the repo and running hasura migrate apply.

When you do so, Hasura will create all needed tables, apply migrations, and so on. It will look like this:

I will start by running hasura init inside the project root folder. The CLI will ask me what name I should call my Hasura project. For this example, I will stick with the defaults.

As a result, I will have a Hasura folder with the config.yaml file created inside.

Now it’s time to run the Hasura console through the CLI.

cd hasura
hasura console

That command will open the Hasura console on https://localhost:9695/ and will keep track of any changes that I will do through the console. Any changes will be written to disk as a migrations file that I can later commit to my GitHub repo.

For example, If I create a users table through the Hasura console, it will generate .up and .down migrations:

- args:
    cascade: false
    read_only: false
    sql: CREATE TABLE "public"."users"("id" text NOT NULL, "name" text NOT NULL, "email"
      text NOT NULL, PRIMARY KEY ("id") );
  type: run_sql
- args:
    name: users
    schema: public
  type: add_existing_table_or_view

- args:
    cascade: false
    read_only: false
    sql: DROP TABLE "public"."users";
  type: run_sql

Modeling our data

Now it’s time to create tables through the Hasura console.

We will create a users table in the Hasura console:

Next we will create the articles table. For this table we will set up a user_id field to be a foreign key and point to the users table.

Next we will create the authors table, which will have more author specific information and will be connected to the users table through the user_id foreign key.

The reason to differentiate between users and authors is that later we want to add authentication to our Gatsby site. All users will be able to see articles, but not everyone will be an author.

Rendering static data from our GraphQL API

So now that we have Hasura and our database set up, we need to connect Gatsby to render the static data that we have in the database.

For that, I will use the gatsby-source-graphql plugin. Let’s install it with

yarn add gatsby-source-graphql

After installing it, I need to add its configuration to my gatsby-config.js file:

{
      resolve: "gatsby-source-graphql",
      options: {
        typeName: "hasura",
        fieldName: "blog",
        createLink: () => {
          return createHttpLink({
            uri:
              "https://localhost:8080/v1/graphql"
            fetch,
          })
        },
      },
    },

Now, what will this do? This will expose a Hasura CRUD API as part of the Gatsby GraphQL schema.

During development, Gatsby exposes a GraphQL IDE in the browser on the https://localhost:8000/___graphql endpoint, and since we’ve added a Hasura GraphQL endpoint within the gatsby-source-graphql configuration, now we will be able to see a blog field with all available queries.

As you can see we can now get a list of articles from our database.

Getting the data on the page

Gatsby has a mechanism of getting data onto the page during the build process by executing so-called page queries. In order to execute a page query we need to export a GraphQL query from the page component. You can read more in detail about that here.

When we export a GraphQL query from a page component (in our case it will be pages/index.js), the result of the query will be injected into the page component through props. So if we export the following query:

export const pageQuery = graphql`
  query {
    site {
      siteMetadata {
        title
      }
    }
    allArticles: blog {
      posts: articles(order_by: { date: desc, title: desc }, limit: 1000) {
        id
        title
        body
        date
        excerpt
      }
    }
  }
`

then we can get all articles data inside the page component. To get the articles data I can use data.allArticles.

const BlogIndex = ({ data, location }) => {
  const siteTitle = data.site.siteMetadata.title
  let posts = data.allArticles.posts
  return (
    <Layout location={location} title={siteTitle}>
      <SEO title="All posts" />
      <SimpleGrid columns={[1, 1, 2, 3]} spacing="40px">
      {posts.map(post => {
        const title = post.title
        return (
          <Article
            id={post.id}
            title={post.title}
            key={post.id}
            date={post.date}
            excerpt={post.description || post.excerpt}
            isNew={!existingIds.includes(post.id)}
          />
        )
      })}
      </SimpleGrid>
    </Layout>
  )
}
export default BlogIndex

It’s important to understand that page queries are running in build time and as soon as a website is built, page queries won’t run until the next build process. In the next section we will explore how to add dynamic functionality on the client in case we want more interactivity from our website.

Adding dynamic data using GraphQL subscriptions

What if we want to get more real-time functionality and subscribe to article changes using GraphQL subscriptions?

For that, we will need to do several things:

• Install and configure the apollo-client.
• Configure both the client and the server side rendering part of Gatsby to deal with subscriptions. You can read more about Gatsby and server side rendering here.
• Run GraphQL subscriptions on the client to fetch real-time data.

In order to fetch data dynamically, using GraphQL we will need a GraphQL client library. In our example we will use the apollo-client library, as it’s the most widely adopted library.

Let’s install it along with all required dependencies needed for the subscriptions configuration.

yarn add @apollo/react-hooks apollo-cache-inmemory apollo-client apollo-link apollo-link-http apollo-link-ws apollo-utilities subscription-transport-ws ws isomorphic-fetch

After installing all dependencies, we will create a new file called apolloClient.js (though you can call it with any name that you want) where we will configure our apollo client.

First of all, let’s create httpLink. HttpLink is our network interface for all http based requests to the GraphQL endpoint:

const http = new HttpLink({
  uri: "https://localhost:8080/v1/graphql",
  headers: {
"x-hasura-admin-secret": process.env.GATSBY_HASURA_GRAPHQL_ADMIN_SECRET,
  },
  fetch,
})

Notice that this example is different from the usual apollo client example. We pass the fetch implementation to HttpLink. This happens because this code can run both during server-side rendering (SSR) and in the browser, and we need to use isomorphic-fetch so our code will work in both environments.

We need to do the same for subscriptions. We need to pass the websockets implementation, but this time only when we are in the node environment (SSR):

const wsForNode = typeof window === "undefined" ? ws : null

const wsClient = new SubscriptionClient(
  "ws://localhost:8080/v1/graphql",
  {
    reconnect: true,
  },
  wsForNode
)

const websocket = new WebSocketLink(wsClient)

So now we have two network interfaces configured (http and websockets). For GraphQL subscriptions we will use websockets and for GraphQL queries and mutations we will use the http interface. I will do so using the proven technique using apollo-utilities:

const link = split(
  ({ query }) => {
    const { kind, operation } = getMainDefinition(query)
  return kind === "OperationDefinition" && operation === "subscription"
  },
  websocket,
  http
)

The final apollo client will look like this:

export const client = new ApolloClient({ link, cache: new InMemoryCache() })

Now if you’ve worked with Apollo previously you are probably familiar with the fact that you need to wrap the root element in ApolloProvider and pass a newly created client.

In Gatsby the process will be a bit different. Gatsby has gatsby-browser.js and gatsby-ssr.js files. Both files are in charge of what’s happening in the browser or during server-side rendering.

In order to wrap our root element within ApolloProvider, we will need to export a function called wrapRootElement from each one of these files. We want to keep our code DRY so we will actually export this function from:

apolloClient.js
export const wrapRootElement = ({ element }) => (
  <apolloprovider client="{client}">{element}</apolloprovider>
)

Now in both gatsby-browser.js and gatsby-ssr.js, we can do the following trick:

export { wrapRootElement } from "./src/apolloClient"

GraphQL subscriptions in action

Now let’s check how GraphQL subscriptions work in action. In our pages/index.js page we can run a GraphQL subscription:

import gql from "graphql-tag"
import { useSubscription } from "@apollo/react-hooks"

const SUBSCRIBE_TO_BLOG_POST_UPDATES = gql`
  subscription subscribeToBlogPostById($id: uuid) {
    articles {
      id
      title
      body
      date
      excerpt
    }
  }
`

To run this subscription, we use useSubscription hook like so:

const { data: subData } = useSubscription(SUBSCRIBE_TO_BLOG_POST_UPDATES, {
    suspend: false,
  })

During build time, Gatsby will call the Hasura engine, which will retrieve data from YugabyteDB and serve it to Gatsby during the build process. From that point on, a static page will be served to the client.

When a static page will be served, ApolloClient will connect to the Hasura engine via WebSockets and retrieve updated data from YugabyteDB using GraphQL subscriptions. Whenever a new article will be added or changed, the subscription will push real-time data from YugabyteDB to the client and our UI component will be rerendered.

Summary

As shown in this post, we can simultaneously have SEO-friendly static markup and real-time dynamic updates in our implementation of GraphQL and JAMstack. With Hasura we get GraphQL auto-generated for us and with YugabyteDB we get the resilience, scale, and geo-distribution of a distributed SQL database.

You can check out the code for the sample app here: https://github.com/vnovick/gatsby-realtime-blog