The Great API Masher
Proof-of-Concept (PoC) for Remixing REST APIs with GraphQL
Agile Architecture
The aim is to make UI development more agile by leveraging the declarative paradigm of GraphQL, an open source spec with NodeJS, Java and other implementations that allows us to specify the data flow and business logic outside the UI in a flexible, easy to change way. By keeping the data flow and business logic outside the UI we can keep the UI as a thin I/O layer, with only state being managed in client being the local state needed for visual behaviors and input validation.
The other great benefit of using a declarative data flow paradigm is remiving the blocking dependency the frontend team often has on the backend team (the endless requests to tweak existing APIs to work better for a particular client, e.g. mobile, or build new APIs on top of existing ones only to aggregate data, simply go away with GraphQL.
As we can see in the example below, instead of having many requests between UI and backend, we have just one request. This increases page responsiveness greatly and can be done for the whole page or on per-component basis. It makes it possible to get all the data we need for a given page (or component) in a directly consumable manner, including derived state, with just one request, which dramatically reduces page load time and increases rendering performance, not to mention providing a much lighter, decoupled UI architecture that is much easier to evolve.
.
Application Correctness
To ensure correct application behavior and guarantee consistent reads and updates for a given set of related entities, the backend APIs, under the GraphQL layer, must leverage transaction isolation at the database level. In addition, we must use an API design pattern that guarantees consistent reads and updates. The main reason for this is that Eventually Consistent backends put the burden on the developer to prevent incorrect application behavior that results from conccurent writes and reads on shared data. One good way we found to solve the consistency problem is to use the domain Aggregates pattern to build an API layer under GraphQL (see: Developing Microservices with Aggregates)
Accomplished Goals (so far)
-
Convert REST APIs into GraphQL data sources that can be shared amongst internal and/or external teams.
-
Enable automatic merging of such sources into one GraphQL Schema that can be accessed by internal and/or external teams to build apps in agile manner by using GraphQL’s declarative data-flow capabilities.
-
Enable remixing of the GraphQL types (including queries and mutations) from the merged data source schemas into new GraphQL types to produce client-specific schema. This includes the ability to compose higher-order types to query data from various sources with one request and the ability to derive state based on some field in the query/mutation result, and represent the derived state in a sibling field, using declarative syntax. This removes the need for imperatively hardcoding common data-flow processes. It means the UI becomes be a pure projection of persisted/derived state on the server (aside from client-specific logic for UI component visual behavior and validation), and a thin I/O layer.
GraphQL & Application Correctness
While the API and the persistence layer should be designed in such a way as to guarantee consistent reads and uodates for each set of related data, e.g. by using domain Aggregates when the transaction executes in one database (so that distributed transactions can be avoided and all reads and updates from/to a set of related data can happen within a single database transaction with the appropriate isolation level), having a client asynchronously call the same API endpoint more than once, in rapid sequence and with different input, is not handled in anyway by GraphQL when it comes to assuring correct application behavior. Moreover, if different clients, e.g mobile vs desktop vs xbox, infer state from the API in different ways some of them may break following changes in the API.
In general the following are good rules to follow:
-
There should be no attempt to perform distributed transactions via GraphQL (instead use Aggregates on the backend to avoid distributed transactions when dealing with one database and perform related mutations/queries within a single database transaction boundary, using the appropriate transaction isolation level, e.g. strict serializable for writes and snapshot isolation for reads) If a distributed transaction involves multiple systems, or is long running, a transaction management layer should be created that manages such distributed transactions.
-
When multiple queries to the same API (or APIs) need to be processed in sync with UI state, e.g. multiple queries from an autocomplete text box where the query results could come back out-of-order with respect to the HTTP requests, GraphQL doesn't have a built-in way to handle that. Therefore, we would need to rely on the presence of request-response mapping, e.g. add a 'uuid(val: String) : String' field in each query so if the client receives multiple results from the same API that are out of order it can use the uuid field in the query result (which reflects the input val) to filter for the response that matches the current state of the autocomplete.
-
If the API response can be interpreted differently by different clients that's a problem. Inferring a definite state in extra fields in the query output eliminates that problem. In other words if state needs to be "inferred" from API response, it should be done using extra fields in query's return type, where normally the client would have to infer state (based on presence/absence of certain fields or other types of inference) This feature of GraphQL allows us to augment our API's response to eliminate the need to infer state in the client.
Maintaining Declarative Composition
- Single Responsibility Principle (SRP) must be preserved in Type Resolvers (aka Controllers) by limiting interactions with the backend to a single API call per resolver invocation and letting GraphQL perform the composition of the returned types by following the resolver dependency chain. This way we can keep the composition declarative.
Query Performance
- To avoid N+1 query proliferation when resolving sub-types in a list type, e.g. get the user's Friends and the name of each of the Friends, we may use batched resolution, via e.g. graphql-resolve-batch. In this case, the API for Friends must support batched input.
Examples of Existing REST APIs (and Mock APIs) and their GraphQL Schema
XKCD API
// KXCD API
// Request made to https://xkcd.com/info.0.json
{ month: '1',
num: 1943,
link: '',
year: '2018',
news: '',
safe_title: 'Universal Dreams',
transcript: '',
alt: '"That\'s ... unsettling." "Yeah, those definitely don\'t sound like the normal dream" ',
img: 'https://imgs.xkcd.com/comics/universal_dreams.png',
title: 'Universal Dreams',
day: '17' }XKCD Schema
// Internal GraphQL Schema for XKCD API
type Query {
latestComic: XKCD_Comic
comic(
id: ID!
): XKCD_Comic
}
type XKCD_Comic {
num: ID!
title: String!
safe_title: String!
# Direct link to the comic image.
img: String!
alt: String!
transcript: String
year: String
month: String
day: String
link: String
news: String
}
// Internal GraphQL Resolvers for XKCD Schema
{
Query: {
latestComic: (parent, args, context) => model.getLatestComic(parent, args, context),
comic: (parent, { id }, context) => model.getComicById(parent, {id}, context),
},
XKCD_Comic: {
// The link is often empty, so build one if it’s not returned.
link: data => data.link || `https://xkcd.com/${data.num}/`,
},
}Numbers Trivia API
// Numbers Trivia API
// Request made to http://numbersapi.com/98/trivia
{ text: '98 is the highest jersey number allowed in the National Hockey League (as 99 was retired by the entire league to honor Wayne Gretzky).',
number: 98,
found: true,
type: 'trivia' }
// Request made to http://numbersapi.com/1/17/date
{ text: 'January 17th is the day in 1912 that Captain Robert Falcon Scott reaches the South Pole, one month after Roald Amundsen.',
year: 1912,
number: 17,
found: true,
type: 'date' }Numbers Trivia Schema
// Internal GraphQL Schema for Numbers Trivia API
type Query {
trivia(number: Int): Numbers_Trivia
date(date: String): Numbers_Trivia
math(number: Int): Numbers_Trivia
year(number: Int): Numbers_Trivia
}
type Numbers_Trivia {
text: String
found: Boolean
number: Int
type: String
date: String
year: String
}
// Internal Resolvers for Numbers Trivia Schema
{
Query: {
trivia: (parent, { number }, context) => model.getNumbers(parent, {number, type: 'trivia'}, context),
date: (parent, { date }, context) => model.getNumbers(parent, {number: date, type: 'date'}, context),
math: (parent, { number }, context) => model.getNumbers(parent, {number, type: 'math'}, context),
year: (parent, { number }, context) => model.getNumbers(parent, {number, type: 'year'}, context),
},
Numbers_Trivia: {
date: data => data.date || null, /* have to be explicit if it might be missing */
year: data => data.year || null, /* have to be explicit if it might be missing */
},
}Mock Schema
// Using mock data source that returns (based on query type):
// an array of GreenApple,
// an array of Cherry
// an an array of Union type of GreenApple and Cherry
//
// Internal GraphQL Schema for Mock API
type Query {
greenApple: [GreenApple]
cherry: [Cherry]
fruit: [MixedFruit]
someQuery: SomeType
}
type SomeType {
abc (someInput: String!): String
uuid(val: String): String
xyz: SomeOtherType
}
type SomeOtherType {
test: String
anotherTest: YetAnotherType
}
type YetAnotherType {
test: String
}
type Cherry {
cherry: String
}
type GreenApple {
apple: String
}
union MixedFruit = Cherry | GreenApple
// Internal Resolvers for Mock API Schema
{
Query: {
greenApple: (parent, args, context) => model.getFruit({type: "GreenApple"}),
cherry: (parent, args, context) => model.getFruit({type: "Cherry"}),
fruit: (parent, args, context) => model.getFruit({}), // returns Union of both fruit types
someQuery: (parent, args, context) => {
return {}
}
},
SomeType: {
abc: (parent, args, context) => Promise.resolve(`some autocompletion of ${args.someInput}`),
uuid: (parent, args, context) => Promise.resolve(args.val),
xyz: (parent, args, context) => model.getSomeOtherData(parent, args, context)
},
SomeOtherType: {
anotherTest: (parent, args, context) => model.getYetAnotherData(parent, args, context)
},
// GraphQL must be able to distinguish GreenApple from Cherry in MixedFruit
// which is a Union of different types (i.e. the actual type is fixed at design
// time)
// We do this with __resolveType
MixedFruit: {
__resolveType(obj) {
if (obj.cherry) {
return "Cherry"
} else {
return "GreenApple"
}
}
}
}Example of Automatically Generated Internal MERGED Schema
type GreenApple {
apple: String
}
union MixedFruit = Cherry | GreenApple
type Numbers_Trivia {
text: String
found: Boolean
number: Int
type: String
date: String
year: String
}
type Query {
"""Returns the current version of GrAMPS."""
grampsVersion: String!
latestComic: XKCD_Comic
comic(id: ID!): XKCD_Comic
trivia(number: Int): Numbers_Trivia
date(date: String): Numbers_Trivia
math(number: Int): Numbers_Trivia
year(number: Int): Numbers_Trivia
greenApple: [GreenApple]
cherry: [Cherry]
fruit: [MixedFruit]
someQuery: SomeType
}
type SomeOtherType {
test: String
anotherTest: YetAnotherType
}
type SomeType {
abc(someInput: String!): String
uuid (val: String): String
xyz: SomeOtherType
}
type XKCD_Comic {
num: ID!
title: String!
safe_title: String!
"""Direct link to the comic image."""
img: String!
alt: String!
transcript: String
year: String
month: String
day: String
link: String
news: String
}
type YetAnotherType {
test: String
}Example of Public GraphQL Schema that Remixes the Merged Internal Schema
# the following is not a comment; see graphql-import
# import XKCD_Comic, Numbers_Trivia, GreenApple, Cherry, MixedFruit, SomeType from "./generated/gramps.graphql"
type Query {
comicAndTrivia: ComicAndTrivia
triviaAndFruit: TriviaAndFruit
someQuery: SomeType
debug: String
}
type ComicAndTrivia {
comic: XKCD_Comic # exposing type from XKCD source
trivia: Numbers_Trivia # exposing type from Numbers source
}
type TriviaAndFruit {
triviaContent: String # resolved by Numbers source
aBasketOfGreenApples: [GreenApple] # resolved by Mock source
aBasketOfCherries: [Cherry] # resolved by Mock source
aBasketOfMixedFruit: [MixedFruit] # resolved by Mock source
legend: Legend
}
type Legend {
greenApple: String
cherry: String
}Example of Public Resolvers for the Remixed Public Schema
{
Query: {
async comicAndTrivia(parent, args, ctx: Context, info) {
const comic = await XKCDResolvers.Query.latestComic(parent, args, ctx)
return { comic }
},
async triviaAndFruit(parent, args, ctx: Context, info) {
const trivia = await NumbersResolvers.Query.trivia(parent, { number: Math.round(Math.random()*100) }, ctx)
return {triviaContent: trivia.text}
},
someQuery (parent, args, ctx: Context, info) {
const mockData = MockResolvers.Query.someQuery(parent, args, ctx)
return mockData
},
debug(parent, args, ctx, info) {
console.log(info);
console.log(info.fieldNodes)
return 'Hello'
}
},
ComicAndTrivia: {
trivia: {
/* reselect a field and its descendants from parent that this field depends on,
using fragment. This is also how we can implement #derived state. */
fragment: `fragment ComicFragment on ComicAndTrivia { comic { day month } }`,
resolve: async (parent, args, ctx: Context, info) => {
const {day, month} = parent.comic
const trivia = await NumbersResolvers.Query.date(parent, { date: `${month}/${day}` }, ctx)
return trivia
}
}
},
TriviaAndFruit: {
aBasketOfGreenApples (parent, args, ctx: Context, info) {
const mockData = MockResolvers.Query.greenApple(parent, args, ctx)
return mockData
},
aBasketOfCherries (parent, args, ctx: Context, info) {
const mockData = MockResolvers.Query.cherry(parent, args, ctx)
return mockData
},
aBasketOfMixedFruit (parent, args, ctx: Context, info) {
const mockData = MockResolvers.Query.fruit(parent, args, ctx)
return mockData
},
legend (parent, args, ctx: Context, info) {
return {greenApple: "🍏", cherry: "🍒"}
}
},
}Example Public Query and its Output Using the Remixed Public Schema
{
someQuery {
abc (someInput:"some string")
uuid(val: "6eghwudf7iy3idhgs8o9s89ds89f9gghgh")
xyz {
test
anotherTest {
test
}
}
}
comicAndTrivia {
comic {
title
}
trivia {
text
}
}
triviaAndFruit {
triviaContent
aBasketOfCherries {
cherry
}
aBasketOfGreenApples {
apple
}
aBasketOfMixedFruit {
... on Cherry {
cherry
}
... on GreenApple {
apple
}
}
legend {
greenApple
cherry
}
}
}{
"data": {
"someQuery": {
"abc": "some autocompletion of some string",
"uuid": "6eghwudf7iy3idhgs8o9s89ds89f9gghgh",
"xyz": {
"test": "some test xyz",
"anotherTest": {
"test": "yet another test"
}
}
},
"comicAndTrivia": {
"comic": {
"title": "The History of Unicode"
},
"trivia": {
"text": "February 9th is the day in 1975 that the Soyuz 17 Soviet spacecraft returns to Earth."
}
},
"triviaAndFruit": {
"triviaContent": "85 is the atomic number of astatine.",
"aBasketOfCherries": [
{
"cherry": "🍒"
},
{
"cherry": "🍒"
}
],
"aBasketOfGreenApples": [
{
"apple": "🍏"
},
{
"apple": "🍏"
},
{
"apple": "🍏"
},
{
"apple": "🍏"
},
{
"apple": "🍏"
},
{
"apple": "🍏"
},
{
"apple": "🍏"
},
{
"apple": "🍏"
},
{
"apple": "🍏"
}
],
"aBasketOfMixedFruit": [
{
"cherry": "🍒 "
},
{
"apple": "🍏"
},
{
"cherry": "🍒 "
},
{
"apple": "🍏"
},
{
"cherry": "🍒 "
},
{
"apple": "🍏"
}
],
"legend": {
"greenApple": "🍏",
"cherry": "🍒"
}
}
}
}