""" SnapBack Copyright (c) 2009, Mitch Berkson. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the owner nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- snapback.py Capture a new EBS snapshot of a volume and maintain an existing set of snapshots segregated by snapshot frequency. Revision history 4/23/09 mb Original (beta) -------------------------------------------------------------------------------- Usage: snapback [OPTIONS] volume_id Required: volume_id EBS volume id of which to take a snapshot (string) Environment: Tested with Python 2.5 and expected to work with 2.6 The following environment variables are used by the ec2 tools: EC2_PRIVATE_KEY EC2_CERT EC2_HOME JAVA_HOME Options: -p ss_path path in which snapshot metadata are stored for different frequency snapshots. If absent, uses current directory -y years_to_retain integer number of years (52 weeks) of snapshots to retain default = 5 -q quarters_to_retain integer number of quarters (13 weeks) of snapshots to retain default = 8 (2 years) -w weeks_to_retain integer number of weeks of snapshots to retain default = 26 (6 months) -d days_to_retain integer number of days of snapshots to retain default = 14 (2 weeks) -o hours_to_retain integer number of hours of snapshots to retain default = 120 (5 days) -n minutes_to_retain integer number of minutes of snapshots to retain default = 0 -h help print script usage -t total snapshots taken print the total number of snapshots in all frequency files in ss_path Description: Amazon Web Services (AWS) offers a facility to conveniently take snapshots of Elastic Block Store volumes which are saved to its Simple Storage Service (S3). No additional organization of snapshots in directories is offered natively nor is it possible to name snapshots. In order to use the snapshots in a typical backup scheme, it is desirable to group them by frequency (e.g., yearly, quarterly, hourly) and to be able to specify for how long to retain snapshots of each frequency. Files of snapshot metadata are maintained, using this script, which group the snapshot metadata by frequency and delete snapshots which have aged beyond their retention specifications. Snapshot metadata are kept in any of the following frequency files in directory ss_path: yearly, quarterly, weekly, daily, hourly, minutely. Each frequency file may contain metadata for more than one volume_id. When this script is called, an EBS snapshot is taken and the metadata returned by ec2-create-snapshot is captured. If no frequency files are present in the directory, no other action is taken. Since snapshots will not be deleted, if this script continues to be called, since no snapshots are being deleted, eventually the EBS snapshot quota will be reached. If there are frequency files, the time from the current snapshot is compared to the time of the most recent snapshot of volume_id in each of the frequency files present. The current metadata are appended to the appropriate longest frequency file. For example, if it has been 14 weeks since the most recent snapshot in quarterly, 6 months since the most recent snapshot in yearly and 30 minutes since the last snapshot in hourly, the current snapshot will be appended to quarterly since it has been more than 13 weeks between the current time and the most recent quarterly snapshot. It is not appended to yearly since it has not been a year since the most recent yearly snapshot. After the current snapshot is appended to a frequency file, the metadata in all frequency files is checked and any which are older than the retention periods specified in the command line arguments (or defaults) are deleted (using ec2-delete-snapshot) and their metadata are deleted from the frequency files. A retention option of 0 means that no additional snapshot metadata will be stored in the corresponding frequency file and none of the snapshots in that file will be deleted. In typical use, this script would be called by cron at the frequency of the highest frequency file. It could, however, be called more frequently than that with no ill effects. In that case the snapshot metadata are saved to the highest frequency file. For example, if it is called every 5 minutes and there is an hourly file but no minutely file, metadata from the snapshots will be saved in the hourly file every 5 minutes. This behavior subverts the notion that each frequency file corresponds to snapshots taken at that frequency but it is better than the alternative of not taking a snapshot when the user believes that one is being taken. In order to set up metadata storage files to save backups in the default frequencies, could do, from the appropriate directory: touch yearly quarterly weekly daily hourly """ # snapback.py # # Author: Mitch Berkson from __future__ import with_statement import sys import subprocess import getopt import os import datetime MIN_SNAPSHOT_TABLE_LEN = 4 # For each command line retention option, corresponding: # file names, default retention time, and scale factor to minutes OPT_ARGS = {"-y": ("yearly", 5, 365 * 24 * 60), \ "-q": ("quarterly", 2, 13 * 7 * 24 * 60), \ "-w": ("weekly", 26, 7 * 24 * 60), \ "-d": ("daily", 14, 24 * 60), \ "-o": ("hourly", 120, 60), \ "-n": ("minutely", 0, 1)} # in order from largest period to smallest SORTED_OPT_ARGS = sorted(OPT_ARGS.items(),key=lambda x: x[1][2], reverse=True) # in order from largest period to smallest ALL_FREQUENCY_FILES = [y[0] for x,y in SORTED_OPT_ARGS] CMD_FREQ_OPTS = [x for x,_ in SORTED_OPT_ARGS] FREQ_FILES_OPTS = dict([(y[0], x) for x,y in SORTED_OPT_ARGS]) # multiplier to convert from each of the optional freqency periods to minutes SCALE_TO_MINUTES = [y[2] for _,y in SORTED_OPT_ARGS] OPT_SCALE = dict([(x, y[2]) for x,y in SORTED_OPT_ARGS]) # default retention times in minutes for each frequency of backup DEFAULT_RETENTION = dict([(x, y[1] * y[2]) for x,y in SORTED_OPT_ARGS]) # indices of components of metadata returned by ec2-create-snapshot DATETIME_INDEX = 4 SNAP_ID_INDEX = 1 VOL_ID_INDEX = 2 # indices of components of metadata returned by ec2-describe-snapshot SNAP_STATUS_INDEX = 3 # index in date/time component of ec2-create-snapshot metadata SECONDS_INDEX = 5 # length of date str in metadata DATE_LEN = 10 # index of "+" in the time portion of the metadata PLUS_INDEX = 5 def ss_created(ss_metadata): """Extract creation datatime from snapshot metadata ss_metadata: snapshot metadata str as returned from ec2-create-snapshot Returns creation datetime as datetime """ strdatetime = \ ss_metadata.split()[DATETIME_INDEX][:DATE_LEN].split("-") + \ ss_metadata.split()[DATETIME_INDEX][DATE_LEN+1:].split(":") strdatetime[SECONDS_INDEX] = strdatetime[SECONDS_INDEX][:-PLUS_INDEX] return datetime.datetime(*[int(s) for s in strdatetime]) class EBSSnapshot: """EBSSnapshot class identified by an EBS snapshot ID""" volume_name = None # EBS volume id as str created = None # snapshot time returned by ec2-create-snapshot as datetime status = None # should be "pending" or "completed" name = None # snapshot id as str metadata = None def __init__(self, volume_id=None): self.volume_name = volume_id def updateSnapshotStatus(self): """Update the completion status of this EBSSnapshot""" self.metadata, _ = \ subprocess.Popen(["ec2-describe-snapshots", self.name], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() if self.metadata: self.status = self.metadata.split()[SNAP_STATUS_INDEX] else: self.status = "" def take_snapshot(self): """Make AWS system call to take a snapshot. Returns 2-tuple of str. Return error data returned by ec2-create-snapshot. No error does not imply that the snapshot was taken. It may be pending. There might also be some other unknown condition. The creation date and time of the snapshot is stored in self.created """ self.metadata, errdata = \ subprocess.Popen(["ec2-create-snapshot", self.volume_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate() if self.metadata: self.name = self.metadata.split()[SNAP_ID_INDEX] self.created = ss_created(self.metadata) self.updateSnapshotStatus() return errdata def __str__(self): return self.name def latestSnapshotDate(volume_id, filename): """Find the date of the most recent EBS snapshot of volume_id in file volume_id: an EBS volume-id as a str. filename: an abolute file path as a str Returns: datetime of the most recent EBS snapshot of volume-d in filename. If no snapshot in file has a volume-id that matches or (filename is empty), then datetime.min is returned """ latest_date = datetime.datetime.min try: with open(filename, "r") as file: for ss_metadata in file: created = ss_created(ss_metadata) ss_volume_id = ss_metadata.split()[VOL_ID_INDEX] if (created > latest_date) and (ss_volume_id == volume_id): latest_date = created except IOError: pass return latest_date def deleteOldSnapshots(filename, ss_path, volume_id, retention, keep_ss): """Delete old volume_id snapshots from file. Delete snapshot metadata from file which are older than retention. Also delete the snapshots themselves from S3 filename: str of a frequency file name containing snapshot metadata ss_path: str of path to filename volume_id: EBS volume-id as str retention: in minutes. Any snapshot with a creation date older than retention will be deleted keep_ss: str of snapshot-id which will not be deleted. This becomes relevant if many snapshots are being deleted from a file and the deletion takes long enough that the most recently captured snapshot has expired. In that case, wouldn't want to delete that last snapshot. As filename is read, a temporary file filename.tmp is written excluding expired snapshot metadata. After filename is closed, it is deleted and filename.tmp is renamed as filename. """ file_path = ss_path + filename temp_file_path = ss_path + filename + ".tmp" try: with open(file_path, "r") as in_file: with open(temp_file_path, "w") as temp_file: for ss_metadata in in_file: created = ss_created(ss_metadata) meta_vol_id = ss_metadata.split()[VOL_ID_INDEX] meta_ss_id = ss_metadata.split()[SNAP_ID_INDEX] if (meta_vol_id == volume_id) and \ (meta_ss_id != keep_ss) and \ datetime.datetime.now() - created >= datetime.timedelta(minutes=int(retention)): meta_snap_id = ss_metadata.split()[SNAP_ID_INDEX] _, _ = subprocess.Popen(["ec2-delete-snapshot", meta_snap_id], stdout=subprocess.PIPE).communicate() else: temp_file.write(ss_metadata) _, _ = subprocess.Popen(["mv", temp_file_path, file_path], stdout=subprocess.PIPE).communicate() except IOError: pass def tallySnapshots(ss_path): """Open all frequency files in ss_path and total their snapshot. Since the metadata for each snapshot are represented by one line, only need to count the lines in the files. """ lines = 0 frequency_files = filter(lambda f: f in ALL_FREQUENCY_FILES, os.listdir(ss_path)) if frequency_files: for file in frequency_files: with open(ss_path + file, "r") as in_file: try: for line in in_file: lines += 1 except: print "Cannot open file %s for reading" % file exit(2) print "%s snapshots in %s" % (lines, ss_path) def main(argv): ss_path = os.getcwd() try: opt_list, args = getopt.getopt(argv, "htp:" + "".join(map(lambda x: x + ":", CMD_FREQ_OPTS)).replace("-", "")) except getopt.GetoptError: usage() sys.exit(2) opts = dict(opt_list) try: opts["-h"] usage() sys.exit(0) except KeyError: pass try: ss_path = opts["-p"] except KeyError: pass ss_path = ss_path + ((ss_path[-1] != "/") and "/" or "") try: opts["-t"] tallySnapshots(ss_path) sys.exit(0) except KeyError: pass if (len(args) != 1): usage() sys.exit(2) retention_times = DEFAULT_RETENTION.copy() for opt, optval in opt_list: if opt in CMD_FREQ_OPTS: retention_times[opt] = int(optval) * OPT_SCALE[opt] volume = args[0] snapshot = EBSSnapshot(volume) ss_errdata = snapshot.take_snapshot() if ss_errdata: print ss_errdata sys.exit(1) if len(snapshot.metadata.split()) < MIN_SNAPSHOT_TABLE_LEN: print "ec2-create-snapshot metadata is too short" sys.exit(1) # Assumes a snapshot status other than completed will eventually complete while snapshot.status != "completed": snapshot.updateSnapshotStatus() files_in_dir = os.listdir(ss_path) frequency_files = filter(lambda f: os.path.isfile(ss_path + f) and f in files_in_dir and retention_times[FREQ_FILES_OPTS[f]] != 0, \ ALL_FREQUENCY_FILES) if not frequency_files: sys.exit(0) # find file in which to store the snapshot metadata. That will be the # lowest frequency file in which the differenece between the creation # time of the current snapshot is greater than that file's period. If not # enough time has elapsed to justify storage in any file, add the metadata # to the file with the shortest period.date for f in frequency_files: if (f == frequency_files[-1]) or \ snapshot.created - latestSnapshotDate(snapshot.volume_name, ss_path + f) >= \ datetime.timedelta(minutes=OPT_SCALE[FREQ_FILES_OPTS[f]]): try: with open(ss_path + f, "a") as file: file.write(snapshot.metadata) break except IOError: return # check all the frequency files for volume-id snapshotss which have passed # their expiration dates. Delete them from EC2 and delete the snapshot # metadata from the file for f in frequency_files: deleteOldSnapshots(f, ss_path, volume, retention_times[FREQ_FILES_OPTS[f]], snapshot.name) def usage(): """Print correct script command line usage""" print __doc__ if __name__ == "__main__": main(sys.argv[1:])