Example
{
"rootGroups": [
{
"name": "global",
"softMemoryLimit": "80%",
"hardConcurrencyLimit": 100,
"maxQueued": 1000,
"schedulingPolicy": "weighted",
"jmxExport": true,
"subGroups": [
{
"name": "data_definition",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 5,
"maxQueued": 100,
"schedulingWeight": 1
},
{
"name": "adhoc",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 50,
"maxQueued": 1,
"schedulingWeight": 10,
"subGroups": [
{
"name": "other",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 2,
"maxQueued": 1,
"schedulingWeight": 10,
"schedulingPolicy": "weighted_fair",
"subGroups": [
{
"name": "${USER}",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 1,
"maxQueued": 100
}
]
},
{
"name": "bi-${toolname}",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 10,
"maxQueued": 100,
"schedulingWeight": 10,
"schedulingPolicy": "weighted_fair",
"subGroups": [
{
"name": "${USER}",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 3,
"maxQueued": 10
}
]
}
]
},
{
"name": "pipeline",
"softMemoryLimit": "80%",
"hardConcurrencyLimit": 45,
"maxQueued": 100,
"schedulingWeight": 1,
"jmxExport": true,
"subGroups": [
{
"name": "pipeline_${USER}",
"softMemoryLimit": "50%",
"hardConcurrencyLimit": 5,
"maxQueued": 100
}
]
}
]
},
{
"name": "admin",
"softMemoryLimit": "100%",
"hardConcurrencyLimit": 50,
"maxQueued": 100,
"schedulingPolicy": "query_priority",
"jmxExport": true
}
],
"selectors": [
{
"user": "bob",
"group": "admin"
},
{
"userGroup": "admin",
"group": "admin"
},
{
"source": ".*pipeline.*",
"queryType": "DATA_DEFINITION",
"group": "global.data_definition"
},
{
"source": ".*pipeline.*",
"group": "global.pipeline.pipeline_${USER}"
},
{
"source": "jdbc#(?<toolname>.*)",
"clientTags": ["hipri"],
"group": "global.adhoc.bi-${toolname}.${USER}"
},
{
"group": "global.adhoc.other.${USER}"
}
],
"cpuQuotaPeriod": "1h"
}
Example Selectors
There are four selectors, that define which queries run in which resource group:
- The first selector matches queries from
bob
and places them in the admin group. - The second selector matches queries from
admin
user group and places them in the admin group. - The third selector matches all data definition (DDL) queries from a source name that includes
pipeline
and places them in theglobal.data_definition
group. This could help reduce queue times for this class of queries, since they are expected to be fast. - The fourth selector matches queries from a source name that includes
pipeline
, and places them in a dynamically-created per-user pipeline group under theglobal.pipeline
group. - The fifth selector matches queries that come from BI tools which have a source matching the regular expression
jdbc#(?<toolname>.*)
and have client provided tags that are a superset ofhipri
. These are placed in a dynamically-created sub-group under theglobal.adhoc
group. The dynamic sub-groups are created based on the values of named variablestoolname
anduser
. The values are derived from the source regular expression and the query user respectively. Consider a query with a sourcejdbc#powerfulbi
, userkayla
, and client tagshipri
andfast
. This query is routed to theglobal.adhoc.bi-powerfulbi.kayla
resource group. - The last selector is a catch-all, which places all queries that have not yet been matched into a per-user adhoc group.
Description
Together, these selectors implement the following policy:
- The user
bob
and any user belonging to user groupadmin
is an admin and can run up to 50 concurrent queries. Queries will be run based on user-provided priority.
For the remaining users:
- No more than 100 total queries may run concurrently.
- Up to 5 concurrent DDL queries with a source
pipeline
can run. Queries are run in FIFO order. - Non-DDL queries will run under the
global.pipeline
group, with a total concurrency of 45, and a per-user concurrency of 5. Queries are run in FIFO order. - For BI tools, each tool can run up to 10 concurrent queries, and each user can run up to 3. If the total demand exceeds the limit of 10, the user with the fewest running queries gets the next concurrency slot. This policy results in fairness when under contention.
- All remaining queries are placed into a per-user group under
global.adhoc.other
that behaves similarly.