Aurora guarantees SLA requirements for jobs. These requirements limit the impact of cluster-wide maintenance operations on the jobs. For instance, when an operator upgrades the OS on all the Mesos agent machines, the tasks scheduled on them needs to be drained. By specifying the SLA requirements a job can make sure that it has enough instances to continue operating safely without incurring downtime.
SLA is defined as minimum number of active tasks required for a job every duration window. A task is active if it was in
RUNNINGstate during the last duration window.
There is a default SLA guarantee for preferred tier jobs and it is also possible to specify custom SLA requirements.
Aurora guarantees a default SLA requirement for tasks in preferred tier.
95% of tasks in a job will be
activefor every 30 mins.
For jobs that require different SLA requirements, Aurora allows jobs to specify their own
SLA requirements via the SlaPolicies. There are 3 different ways to express SLA requirements.
For jobs that need a minimum number of instances to be running all the time,
CountSlaPolicy
provides the ability to express the minimum number of required active instances (i.e. number of
tasks that are RUNNING for at least duration_secs). For instance, if we have a
replicated-service that has 3 instances and needs at least 2 instances every 30 minutes to be
treated healthy, the SLA requirement can be expressed with a
CountSlaPolicy like below,
Job(
name = 'replicated-service',
role = 'www-data',
instances = 3,
sla_policy = CountSlaPolicy(
count = 2,
duration_secs = 1800
)
...
)
For jobs that need a minimum percentage of instances to be running all the time,
PercentageSlaPolicy provides the
ability to express the minimum percentage of required active instances (i.e. percentage of tasks
that are RUNNING for at least duration_secs). For instance, if we have a webservice that
has 10000 instances for handling peak load and cannot have more than 0.1% of the instances down
for every 1 hr, the SLA requirement can be expressed with a
PercentageSlaPolicy like below,
Job(
name = 'frontend-service',
role = 'www-data',
instances = 10000,
sla_policy = PercentageSlaPolicy(
percentage = 99.9,
duration_secs = 3600
)
...
)
When none of the above methods are enough to describe the SLA requirements for a job, then the SLA
calculation can be off-loaded to a custom service called the Coordinator. The Coordinator needs
to expose an endpoint that will be called to check if removal of a task will affect the SLA
requirements for the job. This is useful to control the number of tasks that undergoes maintenance
at a time, without affected the SLA for the application.
Consider the example, where we have a storage-service stores 2 replicas of an object. Each replica
is distributed across the instances, such that replicas are stored on different hosts. In addition
a consistent-hash is used for distributing the data across the instances.
When an instance needs to be drained (say for host-maintenance), we have to make sure that at least 1 of
the 2 replicas remains available. In such a case, a Coordinator service can be used to maintain
the SLA guarantees required for the job.
The job can be configured with a
CoordinatorSlaPolicy to specify the
coordinator endpoint and the field in the response JSON that indicates if the SLA will be affected
or not affected, when the task is removed.
Job(
name = 'storage-service',
role = 'www-data',
sla_policy = CoordinatorSlaPolicy(
coordinator_url = 'http://coordinator.example.com',
status_key = 'drain'
)
...
)
When a CoordinatorSlaPolicy is
specified for a job, any action that requires removing a task
(such as drains) will be required to get approval from the Coordinator before proceeding. The
coordinator service needs to expose a HTTP endpoint, that can take a task-key param
(<cluster>/<role>/<env>/<name>/<instance>) and a json body describing the task
details, force maintenance countdown (ms) and other params and return a response json that will
contain the boolean status for allowing or disallowing the task’s removal.
POST /
?task=<cluster>/<role>/<env>/<name>/<instance>
{
"forceMaintenanceCountdownMs": "604755646",
"task": "cluster/role/devel/job/1",
"taskConfig": {
"assignedTask": {
"taskId": "taskA",
"slaveHost": "a",
"task": {
"job": {
"role": "role",
"environment": "devel",
"name": "job"
},
...
},
"assignedPorts": {
"http": 1000
},
"instanceId": 1
...
},
...
}
}
{
"drain": true
}
If Coordinator allows removal of the task, then the task’s termination lifecycle is triggered. If Coordinator does not allow removal, then the request will be retried again in the future.
Coordinator endpoint get its own lock and this is used to serializes calls to the Coordinator. It guarantees that only one concurrent request is sent to a coordinator endpoint. This allows coordinators to simply look the current state of the tasks to determine its SLA (without having to worry about in-flight and pending requests). However if there are multiple coordinators, maintenance can be done in parallel across all the coordinators.
Note: Single concurrent request to a coordinator endpoint does not translate as exactly-once guarantee. The coordinator must be able to handle duplicate drain requests for the same task.