General Application Logic

This documentation outlines key libraries that the API uses as well as the logic flow to transform HTTP requests into a JSON response of database data.

Important Libraries

Koa

This is the server library. It routes HTTP requests to the correct handler functions to return the appropriate date. Koa provides a structure for defining the how to process requests, what to return, and who can access the data. Koa has good support for writing asynchronous code in a nice way by using the Async/Await es2015 functions that landed in Node v8.0.0. These functions mean that instead of writing code like this:

function logSomething() {
  const requestResultPromise = makeRequest();

  requestResultPromise.then(requestResponse => {
    console.log(requestResponse);
  });
}

You can now write it like this:

async function logSomething() {
  const requestResult = await makeRequest();

  console.log(requestResponse);
}

Much easier to read! All Koa middleware are async/await style functions. Since we are doing a lot of asynchronous data requests in our middleware functions, Koa makes writing our code much easier to read and maintain long term. The downside is more limited support than popular libraries like Express.js. Despite this, Koa is a well written and maintained library with growing support.

Knex

This is our database query builder. Knex.js establishes a connection pool to a database, and creates SQL queries using calls to chainable javascript functions. The order of the functions used to create SQL statements do no matter in Knex. This allows us to build queries incrementally in a script. e.g.

  knex
  .where({foo: "bar"})
  .select('name')
  .from('baz');

resolves to:

SELECT "name"
FROM baz
WHERE foo = "bar";

Knex seemlessly supports transforming data to and from JSON and (by default) always returns data as a promise. Because of the use of promises, Knex is easy to integrate with Koa since Koa can use async/await to resolve those promises. Optionally, Knex can also stream data as a Node ReadStream. This prevents us from having to buffer the data in API memory before responding to the HTTP request. Streams can also transform the data into a spreadsheet or zip file format on-the-fly without having to build those files on the server first.

JsonWebToken

This library generates, decodes, and validates all the tokens that the API uses for authorization. The tokens conform to the JWT spec and contain data about the hashing algorithm used, a base64 encoded version of JSON payload information, and a hash signature. When creating these tokens, a function uses a secret key to hash the JSON payload to create a unique value to "sign" the token. This signature allows us to verify that the JSON payload in the token is not altered in anyway when we use it to verify users have permission to access an API endpoint. This library is primarily used in the authorization routes and the auth middleware.

Database Query Construction

Middleware pipeline

The majority of the requests to the API are to database tables and views that to get, add or update data. A set of query string parameters in the API request contruct different parts of the SQL query that fetches the data. Querystring parameters map directly to SQL statements.

The system that converts the request URL into an SQL query is a chain of middleware functions that interpret the request as it passes through the chain. A middleware function is unique in that it can partially execute, wait for logic from the next middlewares in the chain, and then continue executing it's logic. All request pass down through the middleware chain like a yo-yo before continuing back up the chain and terminating in a response to the client.

Generic routes

Most of the routes that provide access to database resources use the same series of middlewares to handle them. For convenience, these chains are available as reusable functions located in utils/generic-middleware-stacks.

The middlewares incrementally add data to an "action" object that is ultimately handled by the action-handle.js middleware. The action object contains data like the request type, the database table to access, the "WHERE" clauses that were present in the URL querystring, etc. We store the action object in our database for any API call that alters data. The idea between this workflow is that all the modifying queries recorded can be and re-executed to provide undo/redo functionality in our database. This system is heavily modelled after redux.js and does not have the undo/redo funcitonality implemented yet.

Middleware summaries

For a summary of what each middleware does, consult the documentation and comments located in each middleware file.