module:db
- Description:
The Database API, a thin abstraction layer on top of SQLite, PostgreSQL, DynamoDB, Elasticsearch.
The idea is to not introduce a new abstraction layer on top of all databases but to make the API usable for common use cases.
On the source code level access to all databases will be possible using this API but any specific usage like SQL queries syntax or data types available only for some databases will not be unified or automatically converted. Instead it is passed to the database directly.
Only conversion between JavaScript types and database types is unified to some degree meaning JavaScript data types will be converted into the corresponding data types supported by any particular database and vice versa.
Basic CRUD operations are supported for all database and modelled after NoSQL usage, this means no SQL joins are supported by the API, only single table access. SQL joins can be passed as SQL statements directly to the database using low level module:db.query API call, all high level operations like add/put/del perform SQL generation for single table on the fly.
The common convention is to pass the options object with flags that are common for all drivers, this options object can be modified with new properties but all drivers should try not to modify or delete existing properties, so the same options object can be reused in subsequent operations.
All queries and update operations ignore properties that starts with underscore.
Before the DB functions can be used the app.init MUST be called first, the typical usage:
const { db } = require("backendjs"); app.init((err) => { db.add(... ... });All database methods can use default db pool or any other available db pool by using pool: name in the options. If not specified, then default db pool is used, none is default if no -db-pool config parameter specified in the command line or the config file.
Even if the specified pool does not exist, the default pool will be returned, this allows to pre-configure the app with different pools in the code and enable or disable any particular pool at any time.
To use PostgreSQL db pool to get a record and update the current pool:
db-pg-pool=pg://localhost/dbname
db.get("users", { login: "123" }, { pool: "pg" }, (err, row) => { if (row) { ...... db.update("users", row, { pool: "dynamodb" }); } }); const { data } = await db.aget("users", { login: "123" });Most database pools can be configured with options min and max for number of connections to be maintained, so no overload will happen and keep warm connection for faster responses.
db-pg-pool-max = 100 db-sqlite-pool-max = 100The following databases are supported with the basic db methods: Sqlite, PostgreSQL, DynamoDB, Elasticsearch
Multiple pools of the same type can be opened, just add N suffix to all database config parameters where N is a number, referer to such pools in the code as poolN or by an alias.
db-sqlite1-pool = /path/billing db-sqlite1-pool-max = 10 db-sqlite1-pool-options = journal_mode:OFF db-sqlite1-pool-alias = billingand in the Javascript using callback or await:
db.select("bills", { status: "ok" }, { pool: "billing" }, lib.log) await db.aselect("bills", { status: "ok" }, { pool: "billing" })
- Source:
Members
(static) args :Array.<ConfigOptions>
(static) config :string
(static) configMap :object
Properties:
| Name | Type | Description |
|---|---|---|
configMap.count |
int | max number of records to load from the DB |
configMap.interval |
int | how often to refresh config from DB, in mins |
(static) keys :object
(static) modules :object
(static) pool :string
(static) pools :object
(static) tables :object
Methods
(async, static) aadd(table, query, optionsopt)
- Description:
Async version of module:db.add
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | properties to add with primary keys |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data, info } = await db.aadd("users", { id: '123', login: 'admin', name: 'test' })
(async, static) abatch(list, optionsopt)
- Description:
Async version of module:db.batch
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
list |
Array.<object> | an array of objects to get/put/update/delete from the database in the format: |
|
options |
DbRequestOptions |
<optional> |
(async, static) abulk(list, optionsopt)
- Description:
Async version of module:db.bulk
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
list |
Array.<object> | an array of objects to get/put/update/delete from the database in the format: |
|
options |
DbRequestOptions |
<optional> |
(async, static) acopy(table, query, optionsopt)
- Description:
Async version of module:db.copy
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | name of the table to copy |
|
query |
object | a query condition for the table |
|
options |
DbRequestOptions |
<optional> |
(async, static) acreate(table, schema, optionsopt)
- Description:
Async version of module:db.create
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table name |
|
schema |
DbTable | an object with table schema |
|
options |
DbConfigOptions |
<optional> |
options for the database driver |
(static) add(table, query, optionsopt, callbackopt)
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to use |
|
query |
object | an actual record to be updated, primary key properties must be specified |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
db.add("users", { id: '123', login: 'admin', name: 'test' }, (err, rows, info) => {
});
(async, static) adel(table, query, optionsopt)
- Description:
Async version of module:db.del
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | condition |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, info } = db.adel("users", { login: '123' });
console.log('deleted:', info.affected_rows);
(async, static) adelAll(table, query, optionsopt)
- Description:
Async version of module:db.delAll
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | condition |
|
options |
DbRequestOptions |
<optional> |
(async, static) adrop(table, optionsopt)
- Description:
Async version of module:db.drop
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table name |
|
options |
DbRequestOptions |
<optional> |
(async, static) aget(table, query, optionsopt)
- Description:
Async version of module:db.get
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | an object with primary keys |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data } = await db.aget("users", { login: '12345' });
(async, static) aincr(table, query, optionsopt)
- Description:
Async version of module:db.incr
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | properties to update with primary keys |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data, info } = await db.aincr("bk_counter", { id: '123', like0: 1, invite0: 1 })
(async, static) alist(table, query, optionsopt)
- Description:
Async version of module:db.list
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to use |
|
query |
Array.<string> | Array.<object> | list of records |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data } = db.alist("users", ["id1", "id2"]);
console.log(data);
(static) applyPoolOptions()
- Description:
Apply a config parameter to live DB pool, used in config args
updatecallback to make a config value live
- Source:
(async, static) aput(table, query, optionsopt)
- Description:
Async version of module:db.put
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | properties to update with primary keys |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data } = await db.aput("users", { id: '123', login: 'test', name: 'test' })
(async, static) aquery(req) → {Promise}
- Description:
Async version of module:db.query with the same parameters, no exceptions are raised or reject, only resolve is called with an object
{ err, data, info }
- Source:
Parameters:
| Name | Type | Description |
|---|---|---|
req |
DbRequest |
Returns:
| Type | Description |
|---|---|
| Promise |
Example
const { err, data } = await db.aquery({ text: "SELECT ....." }, { pool: "sqlite" });
(async, static) ascan(table, query, optionsopt)
- Description:
Async version of module:db.scan
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table name |
|
query |
DbTable | an object with query condition |
|
options |
DbRequestOptions |
<optional> |
(async, static) asearch(table, query, optionsopt)
- Description:
Async version of module:db.search
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | condition |
|
options |
DbRequestOptions |
<optional> |
Example
await db.asearch("users", "john*", { pool: "elasticsearch" });
(async, static) aselect(table, query, optionsopt)
- Description:
Async version of module:db.select
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | condition |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data } = await db.aselect("users", { roles: 'admin' });
(async, static) asql(test, optionsopt)
- Description:
Async version of module:db.sql
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
test |
string | SQL statement |
|
|
Array.<any> | values for position placeholders |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data, info } = await db.asql("SELECT * FROM users WHERE name=$1 LIMIT $2", ["admin", 1])
(async, static) atransaction(list, optionsopt)
- Description:
Async version of module:db.transaction
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
list |
Array.<object> | an array of objects to get/put/update/delete from the database in the format: |
|
options |
DbRequestOptions |
<optional> |
(async, static) aupdate(table, query, optionsopt)
- Description:
Async version of module:db.update
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | properties to update with primary keys |
|
options |
DbRequestOptions |
<optional> |
Example
const { err, data, info } = await db.aupdate("users", { login: 'test', name: 'Test' })
if (!err && !info?.affected_rows) logger.error("no updates")
(async, static) aupdateAll(table, data, query, optionsopt)
- Description:
Async version of module:db.updateAll
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
data |
object | properties to update |
|
query |
object | condition |
|
options |
DbRequestOptions |
<optional> |
(async, static) aupgrade(table, schema, optionsopt)
- Description:
Async version of module:db.upgrade
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table name |
|
schema |
DbTable | an object with table schema |
|
options |
DbConfigOptions |
<optional> |
options for the database driver |
(static) batch(list, optionsopt, callbackopt)
- Description:
Perform a batch of operations at the same time, all operations for the same table will be run together one by one but different tables will be updated in parallel.
NOTE: it may use different DB client(session) even for the same table due to each op running independently.
On return the second arg to the callback is a list of records with errors as the input record with added property
errstatusanderrmsg, for get ops the result will be retrieved record if exists.One advantage using this with
op: getover module:db.list is caching, module:db.get may use both local and remote caches but db.list always hits the database.
- Source:
Parameters:
| Name | Type | Attributes | Description | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
list |
Array.<object> | an array of objects to get/put/update/delete from the database in the format: Properties
|
||||||||||||||||||||||||||
options |
DbRequestOptions |
<optional> |
default options with additional specific below Properties
|
|||||||||||||||||||||||||
callback |
DbResultCallback |
<optional> |
Example
var ops = [ { table: "bk_counter", query: { id:1, like:1 } },
{ op: "put", table: "users", query: { login: "test", id:1, name:"test" }]
db.batch(ops, { factorCapacity: 0.5 }, lib.log);
(static) bulk(list, optionsopt, callbackopt)
- Description:
Bulk operations, it will be noop if the driver does not support it. The input format is the same as for the module:db.batch method.
The following databases support bulk:
- DynamoDB: only supports
add/put/deland 25 at a time, if more specified it will send multiple batches sequentially - ElasticSearch: supports all update ops
- SQL: supports all ops, runs sequentially or concurrently if
options.concurrencyis provided, uses the same client(session)
- DynamoDB: only supports
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
list |
Array.<object> | an array of objects to get/put/update/delete from the database in the format: |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
second arg is a list of records with errors, same input record with added property |
Example
var ops = [
{ table: "bk_counter", query: { id:1, like:1 } },
{ op: "del", table: "users", query: { login: "test1" } },
{ op: "incr", table: "bk_counter", query: { id:2, like:1 } },
{ table: "users", query: { login: "test2", id:2, name:"test2" } }
]
db.bulk(ops, { pool: "elasticsearch" }, lib.log);
(static) cacheColumns(options, callbackopt)
- Description:
Reload all columns into the cache for the pool,
- Source:
Parameters:
| Name | Type | Attributes | Description | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
options |
string | object | a pool name or an object Properties
|
|||||||||
callback |
function |
<optional> |
(static) checkCapacity()
- Description:
Check if number of requests exceeds the capacity per second, delay if necessary, for DynamoDB only but can be used for pacing requests with any database or can be used generically. The
capmust be initialized withdb.getCapacitycall.
- Source:
(static) configTypes()
- Description:
Build a list of all config types we want to retrieve, based on the
db-config-mapparameters that defines which fields to use for config types.- for each
topitem it create a mix of all main items below - for each
mainitem it creates a mix of all `other`` items
top... -> each of top-main... -> each of top-main-other...
Most common config parameters: runMode, role, roles, tag, region
- for each
- Source:
Example
top is runMode(prod), main is role(shell),tag(local), other is region(none)
- prod
- prod-shell
- prod-shell-none
- prod-local
- prod-local-none
(static) configureCollectStats()
- Description:
Collect stats for enabled pools, this is called by the
statsmoduleTo enable stats collection for a pool it must be explicitly set via config options, additionally collect stats individually for a list of tables
db-dynamodb-pool-options = metrics:1 db-dynamodb-pool-metrics-tables = bk_config, users
- Source:
(static) convertError(req, err) → {Error|object}
- Description:
Convert native database error in some generic human readable string
- Source:
Parameters:
| Name | Type | Description |
|---|---|---|
req |
DbRequest | |
err |
Error | object |
Returns:
| Type | Description |
|---|---|
| Error | object |
(static) convertRows(req, rows, optionsopt)
- Description:
Convert rows returned by the database into the Javascript format or into the format defined by the table columns, most use cases are json, lists, defaults
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
req |
DbRequest | ||
rows |
Array.<any> | ||
options |
DbRequestOptions |
<optional> |
Example
db.describeTables([ { user: { id: {}, name: {}, pair: { join: ["left","right"] } } ]);
db.put("test", { id: "1", type: "user", name: "Test", left: "123", right: "000" })
db.select("test", {}, lib.log)
(static) copy(table, query, optionsopt, callbackopt)
- Description:
Copy records from one table to another between different DB pools or regions Returns stats how many copied or errors
- Source:
Parameters:
| Name | Type | Attributes | Description | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
table |
string | name of the table to copy |
|||||||||||||||||||||||||||||||||||||||||||||||||||||
query |
object | a query condition for the table |
|||||||||||||||||||||||||||||||||||||||||||||||||||||
options |
DbRequestOptions |
<optional> |
default options with additional specific below Properties
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
callback |
DbResultCallback |
<optional> |
(static) create(table, schema, optionsopt, callbackopt)
- Description:
Create a table using column definitions represented as a list of objects.
Some properties may be defined multiple times with number suffixes like:
unique1, unique2, index1, index2to create more than one index for the table, same properties define a composite key in the order of definition or sorted by the property value, for example:{ a: { index:2 }, b: { index:1 } }will create index (b,a) because of theindex:property value being not the same. If all index properties are set to 1 then a composite index will use the order of the properties.*Each database pool also can support native options which are passed directly to the driver in the options, these properties are defined in the object with the same name as the db driver:
pg, sql, sqlite, dynamodb, elasticsearchDynmamoDB:
To create a DynamoDB secondary index, the first index property if not the same as primary key hash defines global index, if it is the same then local, or if the first property contains
dynamodb.globalproperty then it is a global index.DynamoDB projections are defined by the
dynamodb.projections[N]property in the first index property as an array with names of properties to include in projection or ALL for all properties.When using real DynamoDB creating a table may take some time, for such cases if
options.waitTimeoutis not specified it defaults to 1min, so the callback is called as soon as the table is active or after the timeout whichever comes first.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table name |
|
schema |
DbTable | an object with table schema |
|
options |
DbConfigOptions |
<optional> |
options for the database driver |
callback |
DbResultCallback |
<optional> |
response handler |
Example
// DynamoDB global index `name_descr` will include `type` property in the projection but local index `id_title` will include
// name and descr properties
db.create("test_table2", {
id: { primary: 1, type: "int", index1: 1, dynamodb: { projections1: ["name","descr"] } },
type: { primary: 2 },
title: { index1: 2 },
name: { index: 1, dynamodb: { projections: ["type"] } },
descr: { index: 2 },
});
(static) createTables(options, callbackopt)
- Description:
Create or upgrade the tables for the given pool
- Source:
Parameters:
| Name | Type | Attributes | Description | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
options |
objects | string |
Properties
|
|||||||||
callback |
function |
<optional> |
(static) del(table, query, optionsopt, callbackopt)
- Description:
Delete an object in the database, no error if the object does not exist
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to use |
|
query |
object | object with column values |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
db.del("users", { login: '123' }, (err, rows, info) => {
console.log('deleted:', info.affected_rows);
});
(static) delAll(table, query, optionsopt, callbackopt)
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | condition |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
db.delAll("users", { type: "tester" }, lib.log)
(static) describeTables()
- Description:
Define new tables or extend/customize existing tables. Table definitions are used with every database operation, on startup, the backend read all existing table columns from the database and cache them in the memory but some properties like public columns are only specific to the backend so to mark such columns the table with such properties must be described using this method. Only columns with changed properties need to be specified, other columns will be left as it is.
- Source:
Example
db.describeTables({
bk_user: { name: { pub: 1 },
test: { id: { primary: 1, type: "int" },
name: { pub: 1, index: 1 }
}});
(static) drop(table, optionsopt, callbackopt)
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table name |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
(static) existsPool()
- Description:
Returns true if a pool exists
- Source:
(static) get(table, query, optionsopt, callback)
- Description:
Retrieve one record from the database by primary key, returns found record or null if not found
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to read |
|
query |
object | an object with primary key(s) |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback | NOTE: On return the
|
Example
db.get("users", { login: '12345' }, function(err, row) {
if (row) console.log(row.name);
});
(static) getCached()
- Description:
Retrieve cached result or put a record into the cache prefixed with table:key[:key...] Options accept the same parameters as for the usual get action but it is very important that all the options be the same for every call, especially
selectparameters which tells which columns to retrieve and cache. Additional options:- prefix - prefix to be used for the key instead of table name
- Source:
Example
db.getCached("get", "bk_user", { login: req.query.login }, { select: "latitude,longitude" }, function(err, row) {
var distance = lib.geoDistance(req.query.latitude, req.query.longitude, row.latitude, row.longitudde);
});
(static) getCapacity(table, options[opt) → {object}
- Description:
Return an object with capacity property which is the max write capacity for the table, for DynamoDB only. By default it checks
writeCapacityproperty of all table columns and picks the max.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | ||
options[ |
object |
<optional> |
|
Returns:
| Type | Description |
|---|---|
| object |
(static) getConfig()
- Description:
Return all config records for the given instance, the result will be sorted most relevant at the top
- Source:
(static) getIndexes(table, optionsopt) → {Array.<string>}
- Description:
Return indexes for a table or empty object, each item in the object is an array with index columns
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | ||
options |
object |
<optional> |
Returns:
| Type | Description |
|---|---|
| Array.<string> |
(static) getKeys(table, optionsopt) → {Array.<string>}
- Description:
Return primary keys for a table or empty array
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | ||
options |
object |
<optional> |
Returns:
| Type | Description |
|---|---|
| Array.<string> |
(static) getPool(options)
- Description:
Return database pool by name or default pool, options can be a pool name or an object with { pool: name } to return the pool by given name. This call always returns a valid pool object, in case no requested pool found, it returns the default pool, in case of invalid pool name it returns
nonepool. A special poolnonealways returns empty result and no errors.
- Source:
Parameters:
| Name | Type | Description |
|---|---|---|
options |
string | object | a pool name or an object with pool property |
(static) getPoolTables()
- Description:
Return all tables know to the given pool, returned tables are in the object with column information merged from cached columns from the database with description columns given by the application. If
options.namesis 1 then return just table names as a list.
- Source:
(static) getPools()
- Description:
Return a list of all active database pools, returns list of objects with name: and type: properties
- Source:
(static) getProcessRows()
- Description:
Returns a list of hooks to be used for processing rows for the given table
- Source:
(static) getQueryForKeys(keys, query) → {object}
- Description:
Returns an object based on the list of keys, basically returns a subset of query properties.
- Source:
Parameters:
| Name | Type | Description |
|---|---|---|
keys |
Array.<string> | |
query |
object |
Returns:
| Type | Description |
|---|---|
| object |
(static) incr(table, query, optionsopt, callbackopt)
- Description:
Counter operation, increase or decrease column values, similar to update but all specified columns except primary key will be incremented, use negative value to decrease the value.
If no
options.opsobject specified or no 'incr' operations are provided then all columns with type 'counter' will use the update opincr, i.e. all counters will be incremented by the provided numberNote: The record must exist already for SQL databases without UPSERT support, for DynamoDB and Elasticsearch a new record will be created if it does not exist yet.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to use |
|
query |
object | an actual record to be updated, primary key properties must be specified |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
db.incr("bk_counter", { id: '123', like0: 1, invite0: 1 }, (err, rows, info) => {
});
(static) init(callback)
- Description:
Initialize all database pools. the options may containt the following properties:
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
options.createTables |
boolean |
<optional> |
if true then create new tables or upgrade tables with new columns |
callback |
function |
(static) initColumns()
- Description:
Merge cached columns into tables
- Source:
(static) initConfig()
- Description:
Load configuration from the config database, must be configured with
db-configpointing to the database pool where bk_config table contains configuration parameters.The priority of the parameters goes from the most broad to the most specific, most specific always wins, this allows for very flexible configuration policies defined by the app or place where instances running and separated by the run mode. See `db.getConfigTypes`` for more details.
The options takes the following properties:
- force - if true then force to refresh and reopen all db pools
- table - a table where to read the config parameters, default is bk_config
Do not create/upgrade tables and indexes when reloading the config, this is to avoid situations when maintenance is being done and any process reloading the config may create indexes/columns which are not missing but being provisioned or changed.
- Source:
(static) initTables()
- Description:
Merge all tables from all modules
- Source:
(static) list(table, query, optionsopt, callback)
- Description:
Convenient helper to retrieve all records by primary key, the obj must be a list with key property or a string with list of primary key columns
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to use |
|
query |
Array.<string> | Array.<object> | list of records |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
Example
db.list("users", ["id1", "id2"], (err, rows) => { console.log(err, rows) });
db.list("users", "id1,id2", (err, rows) => { console.log(err, rows) });
db.list("users", [{ login: "id1" },{ login: "id2" }], (err, rows) => { console.log(err, rows) });
(static) parseNameOp(name) → {Array.<string>}
- Description:
Split column name into pure name and possible op from the format: NAME[_$[OP]]
- Source:
Parameters:
| Name | Type | Description |
|---|---|---|
name |
string |
Returns:
| Type | Description |
|---|---|
| Array.<string> |
as [name, op] |
(static) prepare(options) → {DbRequest}
- Description:
Prepare for execution for the given operation: add, del, put, update,... Returns prepared object to be passed to the driver's .query method.
- Source:
Parameters:
| Name | Type | Description | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
options |
object | DbRequest |
Properties
|
Returns:
| Type | Description |
|---|---|
| DbRequest |
|
(static) prepareColumn(reqopt, name, value) → {DbRequestColumn}
- Description:
Prepare a column for a retrieval operation, convert aliases, types and ops according to the request and pool mapping
The op in the name after
_$optakes precedence over the options.opsNOTE:
opis lowercase and all underscores are converted into spaces
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
req |
DbRequest |
<optional> |
request object or null, it is not required to be strict DbRequest |
name |
string | column name |
|
value |
any | query or update value |
Returns:
| Type | Description |
|---|---|
| DbRequestColumn |
Example
{ name_$contains: "cat" } // means use the `contains` op for the column `name`
{ $or: { id: 1, id_$: 2, id_$$: 3 } } // means id=1 OR id=2 OR id=3
(static) prepareQuery(req)
- Description:
Preprocess a query object for a given operation, convert types, assign defaults...
- Source:
Parameters:
| Name | Type | Description |
|---|---|---|
req |
DbRequest |
(static) prepareValue(reqopt, options) → {object}
- Description:
Possibly convert the value according to the column op or type, upates the options.value in place
- Source:
Parameters:
| Name | Type | Attributes | Description | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
req |
DbRequest |
<optional> |
request object or null, it is not required to be strict DbRequest |
||||||||||||||||||||
options |
object |
Properties
|
Returns:
| Type | Description |
|---|---|
| object |
same options |
(static) put(table, query, optionsopt, callbackopt)
- Description:
Add/update an object in the database, if object already exists it will be replaced with all new properties from the obj
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to use |
|
query |
object | an actual record to be updated, primary key properties must be specified |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
db.put("users", { id: '123', login: 'test', name: 'test' }, function(err, rows, info) {
});
(static) query(req, callbackopt)
- Description:
Execute query using native database driver, the query is passed directly to the driver.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
req |
DbRequest | an object with request data prepared by module:db.prepareRequest or similar, missing required fileds will be filled in by calling db.prepareRequest on the object |
|
callback |
DbResultCallback |
<optional> |
an error or result |
Example
var req = new db.Request({
pool: "sqlite",
text: "SELECT a.id,c.type FROM users a,bk_icon c WHERE a.id=c.id and a.id=?",
values: ['123']
});
db.query(req, (err, rows, info) => { ... });
// same as
db.query({
pool: "sqlite",
text: "SELECT a.id,c.type FROM users a,bk_icon c WHERE a.id=c.id and a.id=?",
values: ['123']
}, (err, rows, info) => { ... });
(static) queryProcessSync()
- Description:
Post process hook to be used for replicating records to another pool, this is supposed to be used as this:
The conditions when to use it is up to the application logic.
It does not deal with the destination pool to be overloaded, all errors will be ignored, this is for simple and light load only
The destination poll must have tables to be synced configured:
- Source:
Example
db-elasticsearch-pool-tables=table1,table2
db.setProcessRow("post", "*", (req, row) => { db.queryProcessSync("elasticsearch", req, row) });
(static) refreshColumns()
- Description:
Refresh columns for all pools which need it
- Source:
(static) runProcessRows()
- Description:
Run registered pre- or post- process callbacks.
typeis one of thepreor 'post`table- the table to run the hooks for, usually the same as req.table but can be '*' for global hooksreqis the original db request object with the following required properties:op, table, obj, options, info,rowsis the result rows for post callbacks and the same request object for pre callbacks.
- Source:
(static) scan(table, query, optionsopt, rowCallback, endCallbackopt)
- Description:
Convenient helper for scanning a table for some processing, rows are retrieved in batches and passed to the callback until there are no more records matching given criteria. The
queryis the same as in module:db.select method which definedsa condition which records to get. TherowCallbackmust be present and is called for every row or batch retrieved and second parameter which is the function to be called once the processing is complete. At the end, the callback will be called just with 1 argument, err, this indicates end of scan operation. Basically, db.scan is the same as db.select but can be used to retrieve large number of records in batches and allows async processing of such records. To hint a driver that scanning is in progress theoptions.scanningwill be set to true.Parameters:
- Source:
Parameters:
| Name | Type | Attributes | Description | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
table |
string | table to scan |
|||||||||||||||||||||||||||||||||
query |
object | an object with query conditions, same as in module:db.select |
|||||||||||||||||||||||||||||||||
options |
DbRequestOptions |
<optional> |
default options with additional specific below Properties
|
||||||||||||||||||||||||||||||||
rowCallback |
function | process records when called like this `callback(rows, next, info) |
|||||||||||||||||||||||||||||||||
endCallback |
DbResultCallback |
<optional> |
end of scan when called like this: `callback(err) |
Example
// Copy all users from one db into another
db.scan("users", {}, { count: 10, pool: "dynamodb" }, (row, next) => {
db.add("users", row, { pool: "pg" }, next);
}, console.log;
(static) search(table, query, optionsopt, callbackopt)
- Description:
Perform full text search on the given table, the database implementation may ignore table name completely in case of global text index.
Query in general is a text string with the format that is supported by the underlying driver, the db module DOES NOT PARSE the query at all if the driver supports full text search, otherwise it behaves like
select.Options make take the same properties as in the
selectmethod.A special query property
qmay be used for generic search in all fields.Without full text search support in the driver this may return nothing or an error.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
query |
object | string | condition |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
db.search("users", { roles: "admin", q: "john*" }, { pool: "elasticsearch" }, lib.log);
db.search("users", "john*", { pool: "elasticsearch" }, lib.log);
(static) select(table, query, optionsopt, callback)
- Description:
Select objects from the database that match supplied conditions.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to read |
|
query |
object | an object with properties for the condition, all matching records will be returned.
Columns may contain deep property path can be used as well, like |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
Examples
get by primary key
db.select("users", { login: 'admin' }, (err, rows) => { });
get all users by roles, only 2 columns
db.select("users", { role: 'admin' }, { select: 'login,role' }, (err, rows) => { });
all Johns sorted by time, recent first
db.select("users", { name: 'John:' }, { sort: "mtime", desc: 1, ops: { name: "begins_with" } }, (err, rows) => { });
select users by type modified in last 24 hours
db.select("users", { role: 'admin', mtime: Date.now()-86400000 }, { ops: { mtime: "gt" } }, (err, rows) => { });
same as above with column alias
db.select("users", {
role_$in: ['admin', 'user'],
mtime_$gt: Date.now()-86400000
}, (err, rows) => { });
same as above OR created within 24hrs
db.select("users", {
role: 'admin',
$or: {
mtime_$gt: Date.now()-86400000
ctime_$gt: Date.now()-86400000
},
}, (err, rows) => { });
select users by JSON sub property using Elasticsearch
db.select("users", { "settings.status": "ok" }, { pool: "elasticsearch" }, (err, rows) => { });
select admin users with tags/roles using simple SQL join
const query = {
role_$in: ["admin", "user"],
$join: { tag: "tags.id" },
}
const select = "users.*, tags.name as tagname"
db.select("users", query, { select, pool: "pg" }, lib.log)
(static) setConfig()
- Description:
Update and create a DB config record for the given name/type, updates the last record only
- Source:
(static) setProcessColumns()
- Description:
Add a callback to be called after each cache columns event, it will be called for each pool separately. The callback to be called may take options argument and it is called in the context of the pool.
The primary goal for this hook is to allow management of the existing tables which are not own by the backendjs application. For such tables, because we have not created them, we need to define column properties after the fact and to keep column definitions in the app for such cases is not realistic. This callback will allow to handle such situations and can be used to set necessary propeties to the table columns.
- Source:
Example
a few public columns, allow an admin to see all the columns
db.setProcessColumns(function() {
var cols = db.getColumns("users", { pool: this.name });
for (var name in cols) {
if (["id","name"].includes(name)) cols[name].pub = 1; else cols[name].admin = 1;
}
})
(static) setProcessRow()
- Description:
Assign a processRow callback for a table, this callback will be called for every row on every result being retrieved from the specified table thus providing an opportunity to customize the result.
type defines at what time the callback will be called:
pre- making a request to the db on the query recordpost- after the request finished to be called on the result rows
All assigned callback to this table will be called in the order of the assignment.
The callback accepts 2 arguments: function(req, row) where:
req- the original request for a db operation with requiredrow- a row from the result
When producing complex properties by combining other properties it needs to be synchronized using both pre and post callbacks to keep the record consistent.
For queries returning rows, if the callback returns true for a row it will be filtered out and not included in the final result set.
- Source:
Example
db.setProcessRow("post", "bk_user", (req, row) => {
if (row.birthday) row.age = Math.floor((Date.now() - lib.toDate(row.birthday))/(86400000*365));
});
db.setProcessRow("post", "icons", (req, row) => {
if (row.type == "private" && row.id != req.options.user.id) return true;
});
(static) sql(test, optionsopt, callbackopt)
- Description:
Execute arbitrary SQL-like statement if the pool supports it, values must be an Array with query parameters or can be omitted.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
test |
string | SQL statement |
|
|
Array.<any> | values for position placeholders |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
db.sql("SELECT id,name FROM users WHERE name=$1 LIMIT $2", ["admin", 1], { pool: "sqlite" }, lib.log)
db.sql("SELECT * FROM users", { pool: "dynamodb" }, lib.log)
db.sql("SELECT * FROM users WHERE name=$1", ["John"], { pool: "pg", count: 10 }, lib.log)
db.sql("SELECT * FROM users", lib.log);
(static) transaction(list, optionsopt, callbackopt)
- Description:
Same as the module:db.bulk but in transaction mode, all operations must succeed or fail. Not every driver can support it
-
DynamoDB: only 25 operations can be done at the same time, if the list is larger then it will be sequentially run with batches of 25 records.
-
Elasticsearch: same as regular bulk
-
SQL: wraps all ops into standard SQL transaction using (BEGIN/COMMIT/ROLLBACK TRANSACTION).
NOTE: For specific SQL syntax the first statement can be
BEGIN ..., in this case only COMMIT/ROLLBACK will be added.
-
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
list |
Array.<object> | an array of objects to get/put/update/delete from the database in the format: |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
second arg is a list of records with errors, same input record with added property |
Examples
var ops = [
{ text: "BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE" },
{ op: "del", table: "users", query: { login: "test1" } },
{ op: "incr", table: "bk_counter", query: { id:2, like:1 } },
{ table: "users", query: { login: "test2", id:2, name:"test2" } }
]
db.transaction(ops, { pool: "pg" }, lib.log);
db.transaction([
{ op: "del", table: "users", query: { login: "test1" } },
{ op: "add", table: "users", query: { login: "test2", name: "tester 2" } },
], lib.log);
(static) update(table, query, optionsopt, callbackopt)
- Description:
Update existing object in the database. The primary use case is to update one row by the primary key, this way all columns and values are in the query object.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table to use |
|
query |
object | an actual record to be updated, primary key properties may be specified here or in the |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Examples
update users by primary key login
db.update("users", { login: 'test', id: '123' }, (err, rows, info) => {
console.log('updated:', info.affected_rows);
});
db.update("users", { login: 'test', id: '123', first_name: 'Mr' }, { pool: 'pg' }, (err, rows, info) => {
console.log('updated:', info.affected_rows);
});
update a row with primary key login = 'test' and first_name='Carl' to 'John'
db.update("users", { login: 'test', first_name: 'John' }, { query: { first_name: "Carl" } }, (err, rows, info) => {
console.log('updated:', info.affected_rows);
});
update all rows with first_name='Carl' or NULL to first_name='John'
db.update("users", { first_name: 'John' }, { query: { $or: { first_name: "Carl", first_name_$: null } }, { pool: "elasticsearch" }, (err, rows, info) => {
console.log('updated:', info.affected_rows);
});
remove "admin" role from the roles column by primary key
db.update("users", { login: "test", roles_$del: "admin" }, { pool: "dynamodb" }, (err, rows, info) => {
console.log('updated:', info.affected_rows);
});
(static) updateAll(table, data, query, optionsopt, callbackopt)
- Description:
Update all records that match given
queryusing properties from thedataobject
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | db table name |
|
data |
object | properties to update |
|
query |
object | condition for update |
|
options |
DbRequestOptions |
<optional> |
|
callback |
DbResultCallback |
<optional> |
Example
set status ok for all users with type = 'admin' or 'staff'
db.updateAll("users", { status: "ok" }, { type: ["admin", "staff"] }, lib.log)
(static) upgrade(table, schema, optionsopt, callbackopt)
- Description:
Upgrade a table with missing columns from the definition list, if after the upgrade new columns must be re-read from the database then
info.affected_rowsmust be non zero.
- Source:
Parameters:
| Name | Type | Attributes | Description |
|---|---|---|---|
table |
string | table name |
|
schema |
DbTable | an object with table schema |
|
options |
DbConfigOptions |
<optional> |
options for the database driver |
callback |
DbResultCallback |
<optional> |
response handler |