AWS Amplify GraphQL is Awesome!!! But Missing One Thing?

AWS Amplify GraphQL is Awesome!!! But Missing One Thing?

AWS Amplify with the built-in GraphQL code generators and convenience functions are awesome!!! Querying my GQL API is smooth like butter. However, none of this convenience is available in AWS Amplify Lambda function. This means that making GraphQL queries to your AppSync API is going to require some extra work.

Generators are Magic (but not for Lambdas)

In an AWS Amplify project, when you update your Schema, Amplify auto-generates convenience code for you, which can be used in your UI/React code. The problem, once again, is that it can only be used in your UI code.

When your run amplify push or amplify codegen Amplify takes your API schema definition and creates a set of fully specified query strings you can easily use to run queries, mutations, and subscriptions.

Super Semantic: a GraphQL Schema

The GraphQL definition in Amplify lives in amplify/backeng/api/schema.graphql. This is where you can define your GraphQL schema, with all kinds of decorators provided by Amplify.

type Knowledge
@model
@searchable
@auth(
    rules: [
        {
            allow: public,
            provider: apiKey,
            operations: [create, update, read, delete]
        }
    ]
) {
    id: ID!
    url: String
    text: String
    length: Int!
    embedding: [Float]!
    score: Float
    topics: [Topic] @hasMany
    organizationId: ID!
}

Amplify Generates Queries For You

When you amplify push or amplify codegen Amplify automatically creates a set of convenience variables for all the available queries, mutations, and subscriptions for you in these files: src/graphql/mutations.js, src/graphql/queries.js, and src/graphql/subscriptions.js.

These convenience variables provides things like this createKnowledge function. AWS Amplify maintains the mutations.js file for you, ensuring your queries match your schema. However, once again, they are only available by default in the UI code... not in Lambdas!

export const createKnowledge = /* GraphQL */ `
  mutation CreateKnowledge(
    $input: CreateKnowledgeInput!
    $condition: ModelKnowledgeConditionInput
  ) {
    createKnowledge(input: $input, condition: $condition) {
      id
      url
      text
      length
      embedding
      score
      topics {
        items {
          id
          tag
          description
          createdAt
          updatedAt
          knowledgeTopicsId
          __typename
        }
        nextToken
        __typename
      }
      organizationId
      createdAt
      updatedAt
      __typename
    }
  }
`;

React: Easily Use Convenience Functions and Queries

This is awesome, because this you can use this query variable throughout your UI/React code, and so you can avoid having to update your query everywhere you use it. It enables conveniently using the query in your React code, with some aws-amplify provided functions for making GraphQL calls to the AppSync endpoint.

Using Convenience Code in React / UI Code:

import {API, graphqlOperation} from "aws-amplify";
import {createKnowledge} from "@/graphql/mutations";

const response = await API.graphql(graphqlOperation(createKnowledge, {
   input: {
     text: "Some knowledge"
   }
  })
);

Lambdas: Out of Luck!!!

Unfortunately, AS I MENTIONED, Amplify does not provide a convenient, auto-magic way to do this inside your AWS Amplify Lambdas... yet. Instead, the code recommended by AWS looks like this:

const axios = require('axios');
const gql = require('graphql-tag');
const graphql = require('graphql');
const {print} = graphql;

const searchTrainingDatapointsQuery = gql`
    query SearchTrainingDatapoints(
        $filter: SearchableTrainingDatapointFilterInput
        $sort: [SearchableTrainingDatapointSortInput]
        $limit: Int
        $nextToken: String
    ) {
        searchTrainingDatapoints(
            filter: $filter
            sort: $sort
            limit: $limit
            nextToken: $nextToken
        ) {
            items {
                id
                manifestURL
                datasetARN
                createdAt
                updatedAt
                trainingDatapointDetectionId
            }
            nextToken
        }
    }
`


const searchTrainingDatapoints = async (nextToken) => {
  return axios({
    url: process.env.XXXX,
    method: 'post',
    headers: {
      'x-api-key': process.env.XXXX
    },
    data: {
      query: print(searchTrainingDatapointsQuery),
      variables: {
        nextToken,
        filter: {
          isApproved: {
            eq: true
          }
        }
      }
    }
  });
}

Yuck, This is Not Ideal

Yuck! You could put this code in every Lambda where you need it, but that approach comes with a lot of issues. Wouldn't it be so nice to have the same convenience, static-linking, and clean code in your Lambdas!

Why Should Lambdas be Second Class?

I found myself repeating this pattern of copying segments of the GQL queries and mutations I needed into custom functions in each of my Lambdas. This led to extra maintenance work whenever I changed the GQL Schema. I needed to find and update all instances of the queries in every Lambda. This borderline violates the DRY principle, because I'm maintaining multiple copies of nearly identical query code across lambdas and between Lambdas and my UI code. Let's fix that.

The Goal: Lambdas are First Class GQL Users

We want a solution that lets us access the convenience of the autogenerated GraphQL queries, and also some nice functions for hiding the boilerplate authentication, request, and marshalling code of AppSync. We want to be able to share and reuse that code between all my lambdas.

It would be nice if it were this easy to call a GraphQL AppSync endpoint in every Lambda:

const {updateData, updateKnowledge} = require('applyx-graphql')

const embeddings = []
const id = "xxxxx-xxxxx-xxxxx-xxxxx"
const updateResult = await updateData(updateKnowledge,
  "updateKnowledge", {
      input: {
        id: id,
        embedding: embeddings
      }
    })

Bridging the Gap: a Solution

Lambda Layers + Amplify Hooks for Automated Query Sharing

The approach is simple: copy the query generated queries to a lambda layer, alongside boilerplate code for making authenticated requests to AppSync more convenient. It's not perfect. It would be better if we could avoid copying the query files and instead share them between front and backend... but alas, the strong separation between the Lambda runtime and the UI/React/NextJS runtimes prevents that.

Lambda Layers for Code Sharing

AWS Lambda supports something called Lambda Layers. AWS Lambda Layers are a feature that allows you to package and manage common dependencies and code libraries separately from your Lambda function, making them reusable across multiple functions. PERFECT!. A Lambda Layer can hold and share common GraphQL code for use across all our lambdas.

Create a Lambda Layer

First we need to create a new Lambda layer, using amplify add function, and choose to add a Lambda layer.

This lambda layer will become the centralized serverless location for housing and sharing our GraphQL queries, and convenience code.

Add The Layer to Your Function

Now, just add the layer to the Lambda function where you wish to use the convenience queries and functions.

My example here shows a previous "applyxgraphql" layer. This is just an earlier layer I created using this same technique. You can ignore it. Focus on the "applyxgraphqllayer" layer.

Amplify Hooks: for Automating Queries Sharing

By using Amplify hooks, we can copy an updated version of the src/graphql/mutations.js, src/graphql/queries.js,src/graphql/subscriptions.js files to a Lambda layer prior to running amplify push. By adding amplify/hooks/Pre-Push.js, we can trigger an action before amplify push

This hook file copies a those files every time we run amplify push. It will copy the src/graphql/mutations. js, src/graphql/queries.js,src/graphql/subscriptions.js files to a directory inside this new layer.

/**
 * This is an Amplify hook file, wich runs before an ```amplify push```, and copies the autogenerated queries.js,
mutations.js, and subscriptions.js into an AWS Lambda Layer directory.
 *
 * learn more: https://docs.amplify.aws/cli/usage/command-hooks
 */
const fs = require("fs");

const copyFiles = (src, dest) => {
  if (!fs.existsSync(dest)) {
    fs.mkdirSync(dest);
  }
  const files = fs.readdirSync(src);
  for (const fileName of files) {
    const srcPath = `${src}/${fileName}`;
    const destPath = `${dest}/${fileName}`;
    if (fs.existsSync(destPath)) {
      fs.unlinkSync(destPath);
    }
    // also replace a given regex in the file with another string
    let fileContent = fs.readFileSync(srcPath, 'utf8');
    // Replace the text using regex
    // const newContent = fileContent.replace(/export const (.*) =/g, 'module.exports.$1 =');
    // ^ This is for a non-typescript Lambda, where export is not converted by tsc prior to deploying
    fs.writeFileSync(destPath, fileContent);
  }
}
/**
 * @param data { { amplify: { environment: { envName: string, projectPath: string, defaultEditor: string }, command: string, subCommand: string, argv: string[] } } }
 * @param error { { message: string, stack: string } }
 */
const hookHandler = async (data, error) => {
  const src = "src/graphql";
  const dest = "amplify/backend/function/applyxgraphql/lib/nodejs/src/my_graphql/graphql";
  copyFiles(src, dest);

};

const getParameters = async () => {
  const fs = require("fs");
  return JSON.parse(fs.readFileSync(0, { encoding: "utf8" }));
};

getParameters()
  .then((event) => hookHandler(event.data, event.error))
  .catch((err) => {
    console.error(err);
    process.exitCode = 1;
  });

Lambda Layer Files and Functions (An Overview of the Resulting Structure and Code)

Within this lambda layer amplify/backend/functions/applyxgraphql, we will add a set of helper functions alongside a directory of GraphQL queries, and export it all in an index file. Last, we will add a package.json file to make this "local module" requirable. Then we will import the local module into the layer with the layer's root package.json.

├── applyxgraphql-awscloudformation-template.json
├── layer-configuration.json
├── lib
│   └── nodejs
│       ├── package.json # layer dependencies go here.
│       ├── package-lock.json
│       ├── README.txt
│       └── src
│           └── applyx-graphql
│               ├── graphql # query files copied here
│               │   ├── mutations.ts
│               │   ├── queries.ts
│               │   ├── schema.json
│               │   └── subscriptions.ts
│               ├── index.ts
│               ├── package.json # local graphql dependencies go here
│               ├── package-lock.json
│               ├── storeHelpers.ts # convenience functions here
│               ├── storeHelper.test.ts
│               └── tsconfig.json
├── opt
└── parameters.json

NOTE: The 'applyx' is added by Amplify based on your Amplify stack name. My Amplify stack here is named 'applyx'

AppSync Helper Code

Inside storeHelpers.ts We will add some code for packaging up the query, adding the API_KEY for our AppSync endpoint, and packaging it all up as an axios request. The code below is what I'm currently using.

CAVEAT: It works, but could be improved in a few ways: (most notably by not requiring provision of the queryName.. . for another time...).

import axios from 'axios';

/**
 *  API_XXX_GRAPHQLAPIENDPOINTOUTPUT
 *  API_XXX_GRAPHQLAPIIDOUTPUT
 *  API_XXX_GRAPHQLAPIKEYOUTPUT
 */

interface PackageRequest2Result {
  items: Array<any>;
}

const packageRequest2 = async (query: string, queryName: string, variables: any): Promise<PackageRequest2Result> => {
  // console.log('packageRequest2', queryName, JSON.stringify(variables, null, 2));
  const headers = {
    'x-api-key': process.env.API_XXX_GRAPHQLAPIKEYOUTPUT,
    'Content-Type': 'application/json',
  };
  const response = await axios({
    url: process.env.API_XXX_GRAPHQLAPIENDPOINTOUTPUT as string,
    method: 'post',
    headers: headers,
    data: {
      query,
      variables
    },
  });
  const result = response.data;
  if (result.errors) {
    throw new Error(JSON.stringify(result.errors));
  }
  return result?.data[queryName];
};

const getData = async (query: string, queryName: string, id: string): Promise<any> => {
  return await packageRequest2(query, queryName, {id});
};

const searchData = async (query: string, queryName: string, variables: any): Promise<any> => {
  const result = await packageRequest2(query, queryName, variables);
  return result?.items;
};

const createData = async (query: string, queryName: string, variables: any): Promise<any> => {
  return await packageRequest2(query, queryName, variables);
};

const updateData = async (query: string, queryName: string, variables: any): Promise<any> => {
  return await packageRequest2(query, queryName, variables);
};

const deleteData = async (query: string, queryName: string, id: string): Promise<any> => {
  return await packageRequest2(query, queryName, {input: {id}})
}

export {getData, searchData, createData, updateData, deleteData};

Making the Local npm Module Available

A lambda layer can make any dependencies available to Lambda functions by adding those dependencies to the package.json. In this case, we want to make this local module available. That looks like this.

The amplify/backend/function/applyxgraphql/lib/nodejs/package.json file requires this local package as a dependency, making it available to layer users.

{
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "dependencies": {
    "applyx-graphql": "file:src/applyx-graphql"
  }
}

The Result: Convenience like in Amplify UI/React

As a result of this automation + convenience code, we can now more easily and DRY-ly use GraphQL queries, mutations, and subscriptions in our Lambda functions.

We can do something like this in any Lambda that uses this layer:

const {
  updateData,
  createData,
  updateKnowledge,
  createKnowledge
} = require('applyx-graphql')
const embeddings = []
const id = "xxxxx-xxxxx-xxxxx-xxxxx"
const updateResult = await updateData(
  updateKnowledge,
  "updateKnowledge", {
      input: {
        id: id,
        embedding: embeddings
      }
    })

const createResult = await createData(
  createKnowledge,
  "createKnowledge", {
      input: {
        embedding: embeddings
      }
    })

---

A Note on Typescript in Lambdas

For this example to work, we must compile Typescript in each Lambda/Layer into javascript. Here's a hint on how to automate that with amplify push as well.

amplify/backend/function/applyxgraphql/lib/nodejs/src/applyx-graphql/tsconfig.json

{
  "compilerOptions": {
    "target": "es2017",
    "noImplicitAny": false,
    "allowJs": true,
    "types": ["node"],
    "module": "NodeNext",
    "moduleResolution": "nodenext"
  },
  "include": ["."],
  "exclude": ["node_modules", "**/*.test.ts"]
}

{projectRoot}/package.json

{
  "scripts": {
   "amplify:applyxgraphql": "cd amplify/backend/function/applyxgraphql/lib/nodejs/src/applyx-graphql && tsc -p ./tsconfig.json && cd -",
  }
}