api

↓ Members.

↓ Methods.

module:api

Description:

  • HTTP API to the server from the clients, this module implements the basic HTTP(S) API functionality with some common features. The API module incorporates the Express server which is exposed as api.app object, the server spawns Web workers to perform actual operations, monitors the worker processes if they die and restart them automatically.

    How many processes to spawn can be configured via -server-max-workers config parameter.

    When an HTTP request arrives it goes over Express middleware, but before processing any registered routes there are several steps performed:

    • the req object which is by convention is a Request object, assigned with common backend properties to be used later:
      • user - an empty object which will be filled after by signature verification method, if successful, properties from the bk_user table will be set
      • options - an object with internal state and control parameters. Every request always has an options object attached very early with some properties always present:
        • ip - cached IP address
        • host - cached host header from the request
        • path - parsed request url path
        • apath - an array with the path split by /
        • secure - if the request is encrypted, like https
        • appTimezone - milliseconds offset from the UTC provided in the header by the app
    • access verification, can the request be satisfied without proper signature, i.e. is this a public request
    • autherization, check the signature and other global or user specific checks
    • when a API route found by the request url, it is called as any regular Express middlware
      • if there are registered pre processing callback they will be called during access or autherization phases
      • if inside the route a response was returned using api.sendJSON method, registered post process callbacks will be called for such response

    Every request has the trace property, either fake or X-Ray depending on the config, see metrics for usage

    Health enquiry

    When running with AWS load balancer there should be a url that a load balancer polls all the time and this must be very quick and lightweight request. For this purpose there is an API endpoint /ping that just responds with status 200. It is open by default in the default api-allow-path config parameter.

Source:

Members

(inner) args :Array.<ConfigOptions>

Source:

(inner) bind :string

Description:

  • listen on the specified local interfcae, 0.0.0.0 is default

Source:

(inner) port :int

Description:

  • HTTP port to listen to for Express app, 8000 default

Source:

Methods

(static) checkProxy() → {string}

Description:

  • Web proxy: checkProxy("web", req, res) WS proxy: checkProxy("ws", req, socket, head)

Source:
Returns:
Type Description
string
  • a host matched or undefined

(static) checkRateLimits(req, options, callback)

Description:

  • Perform rate limiting by specified property, if not given no limiting is done.

Source:
Parameters:
Name Type Description
req object

Express Request
options object
Properties
Name Type Attributes Description
type string | Array.<string>

determines by which property to perform rate limiting, when using user properties the rate limiter should be called after the request signature has been parsed. Any other value is treated as custom type and used as is. If it is an array all items will be checked sequentially. This property is required.

The predefined types checked for every request:
- ip - check every IP address
- path - limit by path and IP address, * can be used at the end to match only the beginning,
    method can be placed before the path to use different rates for the same path by request method

    -api-rlimits-rate-ip=100
    -api-rlimits-rate-/api/path=2
    -api-rlimits-rate-GET/api/path=10
    -api-rlimits-rate-/api/path/*=1
    -api-rlimits-rate-/api/path/127.0.0.1=100
    -api-rlimits-map-/api/*=rate:100,interval:1000
ip string <optional>

to use the specified IP address
max number

max capacity to be used by default
rate number <optional>

fill rate to be used by default
interval number <optional>

interval in ms within which the rate is measured, default 1000 ms
message string <optional>

more descriptive text to be used in the error message for the type, if not specified a generic error message is used
queue string <optional>

which queue to use instead of the default, some limits are more useful with global queues like Redis instead of the default in-process cache
delay number <optional>

time in ms to delay the response, slowing down request rate
multiplier number <optional>

multiply the interval after it consumed all tokens, subsequent checks use the increased interval, fractions supported, if the multiplier is positive then the interval will keep increasing indefinitely, if it is negative the interval will reset to the default value on first successful consumption
callback function

as function(err, info) where info is from module:cache.limiter
Examples
api.checkRateLimits(req, { type: "ip", rate: 100, interval: 60000 }, (err, info) => {
    if (err) return api.sendReply(err);
    ...
 });

More endpoint config examples

api-rlimits-map-/pub/settings=rate:10,interval:1000,delay:250
api-rlimits-map-GET/passkey/login=rate:3,interval:1000,delay:250
api-rlimits-map-/login=rate:3,interval:30000,delay:1000,multiplier:1.5,queue:unique
api-rlimits-map-/checkin*=rate:5,interval:30000

(static) checkRedirectPlaceholders(req, pathname) → {string}

Description:

  • Replace redirect placeholders

Source:
Parameters:
Name Type Description
req object

Express Request
pathname string

redirect path, it may contain placeholders in the form: @name@:

  • HOST - full host name from header
  • IP - remote IP address
  • DOMAIN - domain from the hostname
  • PATH([1-9])? - full path or a part
  • URL - full url
  • BASE - basename from the path no extention
  • FILE - base file name with extention
  • DIR - directory name only
  • SUBDIR - last part of the directory path
  • EXT - file extention
  • QUERY - stringified query
Returns:
Type Description
string

possibly new path

(static) cleanupHeaders()

Description:

  • For now just convert cookies into an object so logger can process

Source:

(static) cleanupResult(req, table, data) → {object|Array.<object>}

Description:

  • Process records and keep only public properties as defined in the table columns. This method is supposed to be used in the post process callbacks after all records have been processes and are ready to be returned to the client, the last step would be to cleanup all non public columns if necessary. See the api object in DbTableColumn for all supported conditions.

Source:
Parameters:
Name Type Description
req object

Express HTTP incoming request

Properties
Name Type Attributes Description
options.cleanup_strict boolean <optional>

will enforce that all columns not present in the table definition will be skipped as well, by default all new columns or columns created on the fly are returned to the client. api.cleanupStrict=1 can be configured globally.
options.cleanup_rules object <optional>

can be an object with property names and the values 0|1 for pub, 2 for admin, 3 for `staff``
options.cleanup_copy boolean <optional>

means to return a copy of every modified record, the original data is preserved
table string | Array.<string>

can be a single table name or a list of table names which combined public columns need to be kept in the rows.
data object | Array.<object>
Returns:
Type Description
object | Array.<object>

cleaned records

(static) createWebServer(port, bindopt, restartopt, sslopt, timeoutopt, keepAliveTimeoutopt, requestTimeoutopt, maxRequestsPerSocketopt, maxHeaderSizeopt, reusePortopt, nameopt)

Description:

  • Create a Web server with options and request handler, returns a server object.

    Options can have the following properties:

Source:
Parameters:
Name Type Attributes Description
port int

port number is required
bind string <optional>

address to bind
restart string <optional>

name of the processes to restart on address in use error, usually "web"
ssl objext <optional>

an object with SSL options for TLS createServer call
timeout int <optional>

number of milliseconds for the request timeout
keepAliveTimeout int <optional>

number of milliseconds to keep the HTTP connecton alive
requestTimeout int <optional>

number of milliseconds to receive the entire request from the client
maxRequestsPerSocket int <optional>

number of requests a socket can handle before closing keep alive connection
maxHeaderSize int <optional>

maximum length of request headers in bytes
reusePort boolean <optional>

allows multiple sockets on the same host to bind to the same port
name string <optional>

server name to be assigned

(static) getResultPage(req, rows, info) → {object}

Description:

  • Return an object to be returned to the client as a page of result data with possibly next token if present in the info. This result object can be used for pagination responses.

Source:
Parameters:
Name Type Attributes Description
req object

Express Request object

Properties
Name Type Attributes Description
options.total boolean <optional>

return count only from rows[0].count
rows object | Array.<object>
info object
Properties
Name Type Attributes Description
next_token any <optional>

if present returned in result, this is from DB pagination
req.options.total number <optional>

total results if available (Elasticsearch)
Returns:
Type Description
object

with properties { count, data, next_token, total }

(static) getTokenSecret(req) → {string}

Description:

  • Return a secret to be used for enrypting tokens, it uses the user property if configured or the global API token to be used to encrypt data and pass it to the clients. -api-query-token-secret can be configured and if a column in the bk_user with such name exists it is used as a secret, otherwise the api.queryTokenSecret is used as a secret.

Source:
Parameters:
Name Type Description
req object

Express Request object
Returns:
Type Description
string

(static) handleBody()

Description:

  • Parse incoming query parameters in the request body, this is default middleware called early before authenticatoion. Only methods in -api-body-methods processed, defaults are POST/PUT/PATCH. Store parsed parameters in the req.body.

Source:

(static) handleCleanup()

Description:

  • Call registered cleanup hooks and clear the request explicitly

Source:

(static) handleMetrics()

Description:

  • Finish metrics collection about the current rquest

Source:

(static) handleMultipart(req, res, next)

Description:

  • Parse multipart forms for uploaded files, this must be called explicitly by the endpoints that need uploads. The api module handles uploads automatically for configured paths via -api-body-multipart config parameter.

Source:
Parameters:
Name Type Description
req object

Express Request
res object

Express Response
next function
Example
api.app.post("/upload", api.handleMultipart, (req, res, next) => {
  if (req.files.file) ....
})

// Another global way to handle uploads for many endpoints is to call it for all known paths at once before the actual upload handlers.

api.app.post(/^\/upload\//, api.handleMultipart, (req, res, next) => (next("route")));
...
api.app.post("/upload/icon", (req, res, next) => {
...
api.app.post("/upload/icon", (req, res, next) => {

(static) init()

Description:

  • Initialize API layer, this must be called before the api module can be used but it is called by the server module automatically so api.init is rearely need to called directly, only for new server implementation or if using in the shell for testing.

    During the init sequence, this function calls configureMiddleware and configureWeb methods of all modules.

    The api uses its own request parser that places query parameters into req.query or req.body depending on the method.

    For GET method, req.query contains all url-encoded parameters, for POST method req.body contains url-encoded parameters or parsed JSON payload or multipart payload.

Source:

(static) prepareOptions()

Description:

  • Parse or re-parse special headers about app version, language and timezone, it is called early to parse headers first and then right after the query parameters are available, query values have higher priority than headers.

Source:

(static) prepareRequest()

Description:

  • Prepare request options that the API routes will merge with, can be used by pre process hooks, initialize required properties for subsequent use

Source:

(static) registerPreHeaders(req, callback)

Description:

  • Register a callback to be called just before HTTP headers are flushed, the callback may update response headers

Source:
Parameters:
Name Type Description
req object

Express Request
callback function

is a function(req, res, statusCode)

(static) registerRateLimits(name, options)

Description:

  • Register access rate limit for a given name, all other rate limit properties will be applied as described in the module:api.checkRateLimits

Source:
Parameters:
Name Type Description
name string

path or reserved rate type
options object
Properties
Name Type Description
rate number

base rate limit
max number

max rate limit
internal number

rate interval
queue number

which limiter queue to use

(static) replacePath(req, path)

Description:

  • Replace current request path including updating request options. It is used in routing and vhosting.

Source:
Parameters:
Name Type Description
req object

Express Request
path stream

new request path

(static) sendFile(req, file, redirectopt)

Description:

  • Send file back to the client or return 404 status

Source:
Parameters:
Name Type Attributes Description
req object

Express Request
file string

file path
redirect boolean <optional>

redirect url in case of error instead of returning 404

(static) sendFormatted()

Description:

  • Send result back formatting according to the options properties:

    • format - json, csv, xml, JSON is default
    • separator - a separator to use for CSV and other formats
Source:

(static) sendJSON(req, erropt, dataopt)

Description:

  • Send result back with possibly executing post-process callback, this is used by all API handlers to allow custom post processing in the apps. If err is not null the error message is returned immediately with module:api.sendReply.

    if req.options.cleanup is defined it uses module:api.cleanupResult to remove not allowed properties according to the given table rules.

Source:
Parameters:
Name Type Attributes Description
req object

Express Request object
options.cleanup string | Array.<string> <optional>

a table or list of tables to use for cleaning records before returning, see module:api.cleanupResult
err object | Error <optional>

error object
data object <optional>

data to send back as JSON

(static) sendReply(res, textopt)

Description:
Source:
Parameters:
Name Type Attributes Description
res object

Express Response
object | string | Error | number

different scenarios by type:

  • number: this is HTTP status code, text must be provided to return
  • string: return 500 error with status as text
  • object: status properties is set to 200 if not proided
  • Error: return a generic error message api.errInternalError without exposing the real error message, it will log all error exceptions in the logger subject to log throttling configuration.
text string <optional>

message to return
Example
api.sendReply(res, 400, "invalid input")
api.sendReply(res, "server is not available")

(static) sendStatus(res, options)

Description:

  • Return reply to the client using the options object, it contains the following properties: i18n Note:

    The API server attaches fake i18n functions req.__ and res.__ which are used automatically for the message property before sending the response.

    With real i18n module these can/will be replaced performing actual translation without using i18n.__ method for messages explicitely in the application code for sendStatus or sendReply methods.

    Replies can be delayed per status via api.delays if configured, to override any dalays set req.options.sendDelay to nonzero value, negative equals no delay

Source:
Parameters:
Name Type Description
res object

Express Response
options object
Properties
Name Type Attributes Default Description
status number <optional>
 200

the respone status code
message string <optional>

property to be sent as status line and in the body
contentType string <optional>

defines Content-Type header, the options.message will be sent in the body only
url string <optional>

for redirects when status is 301, 302...

(static) shutdown()

Description:

  • Gracefully close all connections, call the callback after that

Source:

(static) startMetrics()

Description:

  • This is supposed to be called at the beginning of request processing to start metrics and install the handler which will be called at the end to finalize the metrics and call the cleanup handlers.

Source:

(static) toParams(req, schema, optionsopt) → {object|string}

Description:

  • Parse body/query parameters according to the schema by using lib.toParams, uses the req.body if present or req.query.

Source:
Parameters:
Name Type Attributes Description
req object

Express request
schema module:lib.ParamsOptions

schema object
options object <optional>
Properties
Name Type Attributes Description
defaults object <optional>

merged with global queryDefaults
query boolean <optional>

use only req.query, not req.body
Returns:
Type Description
object | string
  • a query object or an error message or null
Example
var query = api.toParams(req, { q: { required: 1 } }, { null: 1 });
 if (typeof query == "string") return api.sendReply(req, 400, query)