api

module:api

Description:
  • HTTP API to the server from the clients, this module implements the basic HTTP(S) API functionality with some common features. The API module incorporates the Express server which is exposed as api.app object, the server spawns Web workers which perform actual operations and monitors the worker processes if they die and restart them automatically. How many processes to spawn can be configured via -server-max-workers config parameter.

    When an HTTP request arrives it goes over Express middleware, but before processing any registered routes there are several steps performed:

    • the req object which is by convention is a Request object, assigned with common backend properties to be used later:
      • user - an empty object which will be filled after by signature verification method, if successful, properties from the bk_user table will be set
      • options - an object with internal state and control parameters. Every request always has an options object attached very early with some properties always present:
        • ip - cached IP address
        • host - cached host header from the request
        • path - parsed request url path
        • apath - an array with the path split by /
        • secure - if the request is encrypted, like https
        • appTimezone - milliseconds offset from the UTC provided in the header by the app
    • access verification, can the request be satisfied without proper signature, i.e. is this a public request
    • autherization, check the signature and other global or user specific checks
    • when a API route found by the request url, it is called as any regular Express middlware
      • if there are registered pre processing callback they will be called during access or autherization phases
      • if inside the route a response was returned using api.sendJSON method, registered post process callbacks will be called for such response

    Every request has the trace property, either fake or X-Ray depending on the config, see metrics for usage

Properties:
Name Type Description
port int

HTTP port to listen to for Express app

bind string

listen on the specified local interfcae, 0.0.0.0 is default

Members

(inner) args :Array.<ConfigOptions>

Type:

Methods

(static) checkProxy() → {string}

Description:
  • Web proxy: checkProxy("web", req, res) WS proxy: checkProxy("ws", req, socket, head)

Returns:
Type Description
string
  • a host matched or undefined

(static) checkRateLimits()

Description:
  • Perform rate limiting by specified property, if not given no limiting is done.

    The following options properties can be used:

    • type - determines by which property to perform rate limiting, when using user properties the rate limiter should be called after the request signature has been parsed. Any other value is treated as custom type and used as is. If it is an array all items will be checked sequentially. This property is required.

      The predefined types checked for every request:

      • ip - check every IP address

      • path - limit by path and IP address, * can be used at the end to match only the beginning, method can be placed before the path to use different rates for the same path by request method

        -api-rlimits-rate-ip=100 -api-rlimits-rate-/api/path=2 -api-rlimits-rate-GET/api/path=10 -api-rlimits-rate-/api/path/=1 -api-rlimits-rate-/api/path/127.0.0.1=100 -api-rlimits-map-/api/=rate:100,interval:1000

    • ip - to use the specified IP address

    • max - max capacity to be used by default

    • rate - fill rate to be used by default

    • interval - interval in ms within which the rate is measured, default 1000 ms

    • message - more descriptive text to be used in the error message for the type, if not specified a generic error message is used

    • queue - which queue to use instead of the default, some limits is more useful with global queues like Redis instead of the default

    • delay - time in ms to delay the response, slowing down request rate

    • multiplier - multiply the interval after it consumed all tokens, subsequent checks use the increased interval, fractions supported, if the multiplier is positive then the interval will keep increasing indefinitely, if it is negative the interval will reset to the default value on first successful consumption

    The metrics are kept in the LRU cache in the process by default or in cluster mode in the server process.

Example
api.checkRateLimits(req, { type: "ip", rate: 100, interval: 60000 }, (err, info) => {
    if (err) return api.sendReply(err);
    ...
 });

(static) checkRedirectPlaceholders()

Description:
  • Replace redirect placeholders

(static) cleanupHeaders()

Description:
  • For now just convert cookies into an object so logger can process

(static) cleanupResult()

Description:
  • Process records and keep only public properties as defined in the table columns. This method is supposed to be used in the post process callbacks after all records have been processes and are ready to be returned to the client, the last step would be to cleanup all non public columns if necessary.

    table can be a single table name or a list of table names which combined public columns need to be kept in the rows. List of request tables is kept in the req.options.cleanup which by default is empty.

    By default primary keys are not kept and must be marked with pub property in the table definition to be returned.

    If any column is marked with priv property is never returned.

    The options.isInternal allows to return everything except 'priv' columns

    Columns with the pub_admin property will be returned only if options.isAdmin is truish, same with the pub_staff property, it requires options.isStaff.

    To return data based on the current user roles a special property in the format pub_types must be set as a string or an array with roles to be present in the current user type`` field. This is checked only if the column is allowed, this is an additional restriction, i.e. a column must be allowed by the pub` property or other way.

    To retstrict by role define priv_types in a column with a list of roles which should be denied access to the field.

    The req.options.pool property must match the actual rowset to be applied properly, in case the records have been retrieved for the different database pool.

    The req.options.cleanup_strict will enforce that all columns not present in the table definition will be skipped as well, by default all new columns or columns created on the fly are returned to the client. api.cleanupStrict=1 can be configured globally.

    The req.options.cleanup_rules can be an object with property names and the values 0, or 1 for pub or 2 for admin`` or 3forstaff``

    The req.options.cleanup_copy means to return a copy of every modified record, the original data is preserved

(static) createWebServer(port, bindopt, restartopt, sslopt, timeoutopt, keepAliveTimeoutopt, requestTimeoutopt, maxRequestsPerSocketopt, maxHeaderSizeopt, reusePortopt, nameopt)

Description:
  • Create a Web server with options and request handler, returns a server object.

    Options can have the following properties:

Parameters:
Name Type Attributes Description
port int

port number is required

bind string <optional>

address to bind

restart string <optional>

name of the processes to restart on address in use error, usually "web"

ssl objext <optional>

an object with SSL options for TLS createServer call

timeout int <optional>

number of milliseconds for the request timeout

keepAliveTimeout int <optional>

number of milliseconds to keep the HTTP connecton alive

requestTimeout int <optional>

number of milliseconds to receive the entire request from the client

maxRequestsPerSocket int <optional>

number of requests a socket can handle before closing keep alive connection

maxHeaderSize int <optional>

maximum length of request headers in bytes

reusePort boolean <optional>

allows multiple sockets on the same host to bind to the same port

name string <optional>

server name to be assigned

(static) getResultPage()

Description:
  • Return an object to be returned to the client as a page of result data with possibly next token if present in the info. This result object can be used for pagination responses.

(static) getTokenSecret()

Description:
  • Return a secret to be used for enrypting tokens, it uses the user property if configured or the global API token to be used to encrypt data and pass it to the clients. -api-query-token-secret can be configured and if a column in the bk_user with such name exists it is used as a secret, otherwise the api.queryTokenSecret is used as a secret.

(static) handleBody()

Description:
  • Parse incoming query parameters in the request body, this is default middleware called early before authenticatoion. Only methods in -api-body-methods processed, defaults are POST/PUT/PATCH. Store parsed parameters in the req.body.

(static) handleCleanup()

Description:
  • Call registered cleanup hooks and clear the request explicitly

(static) handleMetrics()

Description:
  • Finish metrics collection about the current rquest

(static) handleMultipart()

Description:
  • Parse multipart forms for uploaded files, this must be called explicitly by the endpoints that need uploads.

    Example

       api.app.post("/upload", api.handleMultipart, (req, res, next) => {
           if (req.files.file) ....
       })
    

    Another global way to handle uploads for many endpoints is to call it for all known paths at once before the actual upload handlers.

       api.app.post(/^\/upload\//, api.handleMultipart, (req, res, next) => (next("route")));
       ...
       api.app.post("/upload/icon", (req, res, next) => {
       ...
       api.app.post("/upload/icon", (req, res, next) => {
    

    The api module handles uploads automatically for configured paths via -api-allow-multipart config parameter.

(static) init()

Description:
  • Initialize API layer, this must be called before the api module can be used but it is called by the server module automatically so api.init is rearely need to called directly, only for new server implementation or if using in the shell for testing.

    During the init sequence, this function calls configureMiddleware and configureWeb methods of all modules.

    The api uses its own request parser that places query parameters into req.query or req.body depending on the method.

    For GET method, req.query contains all url-encoded parameters, for POST method req.body contains url-encoded parameters or parsed JSON payload or multipart payload.

    The reason not to do this by default is that this may not be the always wanted case and distinguishing data coming in the request or in the body may be desirable, also, this will needed only for Express handlers .all, when registering handler by method like .get or .post then the handler needs to deal with only either source of the request data.

(static) prepareOptions()

Description:
  • Parse or re-parse special headers about app version, language and timezone, it is called early to parse headers first and then right after the query parameters are available, query values have higher priority than headers.

(static) prepareRequest()

Description:
  • Prepare request options that the API routes will merge with, can be used by pre process hooks, initialize required properties for subsequent use

(static) registerPreHeaders()

Description:
  • Register a callback to be called just before HTTP headers are flushed, the callback may update response headers

    • callback is a function(req, res, statusCode)

(static) registerRateLimits()

Description:
  • Register access rate limit for a given name, all other rate limit properties will be applied as described in the checkRateLimits

(static) sendFile()

Description:
  • Send file back to the client, res is Express response object

(static) sendFormatted()

Description:
  • Send result back formatting according to the options properties:

    • format - json, csv, xml, JSON is default
    • separator - a separator to use for CSV and other formats

(static) sendJSON(req, erropt, dataopt)

Description:
  • Send result back with possibly executing post-process callback, this is used by all API handlers to allow custom post processing in the apps. If err is not null the error message is returned immediately with module:api.sendReply.

    if req.options.cleanup is defined it uses module:api.cleanupResult to remove not allowed properties according to the given table rules.

Parameters:
Name Type Attributes Description
req object

Express request object

err object | Error <optional>

error object

data object <optional>

data to send back as JSON

(static) sendReply()

Description:
  • Send formatted JSON reply to an API client, if status is an instance of Error then error message with status 500 is sent back.

    If the status is an object it is sent as is.

    All Error objects will return a generic error message without exposing the real error message, it will log all error exceptions in the logger subject to log throttling configuration.

(static) sendStatus()

Description:
  • Return reply to the client using the options object, it contains the following properties:

    • status - defines the respone status code
    • message - property to be sent as status line and in the body
    • type - defines Content-Type header, the message will be sent in the body
    • url - for redirects when status is 301, 302...

    i18n Note:

    The API server attaches fake i18n functions req.__ and res.__ which are used automatically for the message property before sending the response.

    With real i18n module these can/will be replaced performing actual translation without using i18n.__ method for messages explicitely in the application code for sendStatus or sendReply methods.

    Replies can be delayed per status via api.delays if configured, to override any daly set req.options.sendDelay to nonzero value, negative equals no delay

(static) shutdown()

Description:
  • Gracefully close all connections, call the callback after that

(static) startMetrics()

Description:
  • This is supposed to be called at the beginning of request processing to start metrics and install the handler which will be called at the end to finalize the metrics and call the cleanup handlers.

(static) toParams(req, schema, optionsopt) → {object|string}

Description:
  • Parse body/query parameters according to the schema by using lib.toParams, uses the req.body if present or req.query.

Parameters:
Name Type Attributes Description
req object

Express request

schema module:lib.ParamsOptions

schema object

options object <optional>
Properties
Name Type Attributes Description
defaults object <optional>

merged with global queryDefaults

query boolean <optional>

use only req.query, not req.body

Returns:
Type Description
object | string
  • a query object or an error message or null
Example
var query = api.toParams(req, { q: { required: 1 } }, { null: 1 });
 if (typeof query == "string") return api.sendReply(req, 400, query)