This document summarizes Aurora storage configuration and maintenance details and is intended for use by anyone deploying and/or maintaining Aurora.
For a high level overview of the Aurora storage architecture refer to this document.
Below is a summary of scheduler storage configuration flags that either don’t have default values or require attention before deploying in a production environment.
Defines the Mesos replicated log quorum size. See the replicated log configuration document on how to choose the right value.
Location of the Mesos replicated log files. Consider allocating a dedicated disk (preferably SSD) for Mesos replicated log files to ensure optimal storage performance.
ZooKeeper path used for Mesos replicated log quorum discovery.
See code) for other available Mesos replicated log configuration options and default values.
Configuration options for the Aurora scheduler backup manager.
The interval on which the scheduler writes local storage backups. The default is every hour.
Directory to write backups to.
Maximum number of backups to retain before deleting the oldest backup(s).
Be sure to read the entire page before attempting to restore from a backup, as it may have unintended consequences.
The restoration procedure replaces the existing (possibly corrupted) Mesos replicated log with an earlier, backed up, version and requires all schedulers to be taken down temporarily while restoring. Once completed, the scheduler state resets to what it was when the backup was created. This means any jobs/tasks created or updated after the backup are unknown to the scheduler and will be killed shortly after the cluster restarts. All other tasks continue operating as normal.
Usually, it is a bad idea to restore a backup that is not extremely recent (i.e. older than a few hours). This is because the scheduler will expect the cluster to look exactly as the backup does, so any tasks that have been rescheduled since the backup was taken will be killed.
Follow these steps to prepare the cluster for restoring from a backup:
Stop all scheduler instances
Consider blocking external traffic on a port defined in
-http_port for all schedulers to
prevent users from interacting with the scheduler during the restoration process. This will help
troubleshooting by reducing the scheduler log noise and prevent users from making changes that will
be erased after the backup snapshot is restored
Next steps are required to put scheduler into a partially disabled state where it would still be able to accept storage recovery requests but unable to schedule or change task states. This may be accomplished by updating the following scheduler configuration options:
-mesos_master_addressto a non-existent zk address. This will prevent scheduler from registering with Mesos. E.g.:
-max_registration_delay- set to sufficiently long interval to prevent registration timeout and as a result scheduler suicide. E.g:
-reconciliation_initial_delayoption is set high enough (e.g.:
365days) to prevent accidental task GC. This is important as scheduler will attempt to reconcile the cluster state and will kill all tasks when restarted with an empty Mesos replicated log.
Restart all schedulers
Get rid of the corrupted files and re-initialize Mesos replicate log:
-native_log_file_pathon all schedulers
mesos-log initialize --path=<-native_log_file_path>
At this point the scheduler is ready to rehydrate from the backup:
Identify the leading scheduler by:
aurora_admin get_scheduler <cluster>- if scheduler is responsive
Locate the desired backup file, copy it to the leading scheduler and stage recovery by running
the following command on a leader
aurora_admin scheduler_stage_recovery <cluster> scheduler-backup-<yyyy-MM-dd-HH-mm>
At this point, the recovery snapshot is staged and available for manual verification/modification
aurora_admin scheduler_print_recovery_tasks and
aurora_admin help <command> for usage details.
Commit recovery. This instructs the scheduler to overwrite the existing Mesosreplicated log with
the provided backup snapshot and initiate a mandatory failover
aurora_admin scheduler_commit_recovery <cluster>
Undo any modification done during Preparation sequence.