Skip to content

Commit

Permalink
Adding new flags for scrubbing
Browse files Browse the repository at this point in the history
  • Loading branch information
@sentry0
sentry0 committed Nov 6, 2024
1 parent 60c3b0c commit 8852fa3
Show file tree
Hide file tree
Showing 8 changed files with 150 additions and 109 deletions.
53 changes: 33 additions & 20 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,21 +75,23 @@ command. Doing so will install VMSnap which includes a vmsnap bin.

The following CLI switches are available when invoking VMSnap.

| Switch | Status | Backup | Scrub | Type | Examples/Notes |
|------------|--------|--------|--------|---------|-------------------------------------------------------------------|
| domains | ✅ | ✅ | ✅ | string | "vm1" or "vm1,vm2,etc" or "*" |
| status | ✅ | - | - | boolean | Querys the domain(s) |
| backup | - | ✅ | - | boolean | Does an incremental backup (if possible) |
| scrub | - | - | ✅ | boolean | Cleans checkpoints and bitmaps off of the domain |
| output | ✅ | ✅ | - | string | A full path to a directory where backups are placed |
| verbose | ✅ | - | - | boolean | Prints out extra information when running a status check |
| machine | ✅ | - | - | boolean | Removes some output from the status command |
| json | ✅ | - | - | boolean | Outputs the status command is JSON |
| yaml | ✅ | - | - | boolean | Output YAML from the status command (aliased to `--yml`) |
| raw | - | ✅ | - | boolean | Enables raw disk handling |
| groupBy | ✅ | ✅ | - | string | Defines how backups are grouped on disk (month, quarter, or year) |
| prune | - | ✅ | - | boolean | Rotates backups by **deleting** last periods backup* |
| pretty | ✅ | - | - | boolean | Pretty prints disk sizes (42.6 GB, 120 GB, etc) |
| Switch | Status | Backup | Scrub | Type | Examples/Notes |
|----------------|--------|--------|--------|---------|--------------------------------------------------------------------|
| domains | ✅ | ✅ | ✅ | string | "vm1" or "vm1,vm2,etc" or "*" |
| status | ✅ | - | - | boolean | Querys the domain(s) |
| backup | - | ✅ | - | boolean | Does an incremental backup (if possible) |
| scrub | - | - | ✅ | boolean | Cleans checkpoints and bitmaps off of the domain |
| output | ✅ | ✅ | - | string | A full path to a directory where backups are placed |
| verbose | ✅ | - | - | boolean | Prints out extra information when running a status check |
| machine | ✅ | - | - | boolean | Removes some output from the status command |
| json | ✅ | - | - | boolean | Outputs the status command is JSON |
| yaml | ✅ | - | - | boolean | Output YAML from the status command (aliased to `--yml`) |
| raw | - | ✅ | - | boolean | Enables raw disk handling |
| groupBy | ✅ | ✅ | - | string | Defines how backups are grouped on disk (month, quarter, or year) |
| prune | - | ✅ | - | boolean | Rotates backups by **deleting** last periods backup* |
| pretty | ✅ | - | - | boolean | Pretty prints disk sizes (42.6 GB, 120 GB, etc) |
| checkpointName | - | - | ✅ | boolean | The name of the checkpoint to delete (no effect when scrubType=*) |
| scrubType | - | - | ✅ | boolean | The type of item to scrub (checkpoint, bitmap, both, or * for ALL) |

*\*This happens on or after the the middle of the current period (15 days monthly, 45 days quarterly or 180 yearly)*

Expand Down Expand Up @@ -180,8 +182,8 @@ on disk. Look at the table below for more information.
#### Pruning (Caution)

**Note:** Pruning is destructive. Be careful when using it and check your
backups frequently!
> **Note:** Pruning is destructive. Be careful when using it and check your
backups frequently!

Pruning backups may be done by setting `--prune` on the backup command.
This flag will automatically delete last periods backup once the middle of the
Expand All @@ -198,12 +200,23 @@ You can turn on raw disk handling by setting the `--raw` flag.

### Scrubbing

**Note:** This is an inherently destructive action, be careful!
> **Note:** These commands are inherently destructive, be careful!
To scrub a VM of checkpoints and bitmaps:
It is occasionally useful to be able to scrub one or more checkpoints or bitmaps
from your domain. Doing so is fairly straight forward with VMSnap but please do
be cautious.

Use this command to scrub a single bitmap from your backup disks. Keep in mind
that bitmaps are stored on a per disk basis. VMSnap will scrub each disk of the
bitmap if it find it.
```sh
vmsnap --domains="dom1" --scrub
vmsnap --domains="dom1" --scrub --scrubType=bitmap --checkpointName=virtnbdbackup.17
```

To scrub a domain of **ALL** checkpoints and bitmaps

```sh
vmsnap --domains="dom1" --scrub --scrubType=*
```

## Contributing
Expand Down
36 changes: 32 additions & 4 deletions libs/general.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,24 @@ import { access } from 'fs/promises';
import commandExists from 'command-exists';
import {
ERR_DOMAINS,
ERR_INVALID_SCRUB_TYPE,
ERR_REQS,
ERR_SCRUB,
ERR_TOO_MANY_COMMANDS,
logger,
} from '../vmsnap.js';
import { cleanupCheckpoints, fetchAllDomains, VIRSH } from './virsh.js';
import { cleanupBitmaps, QEMU_IMG } from './qemu-img.js';
import { BACKUP } from './libnbdbackup.js';

const SCRUB_TYPE_CHECKPOINT = 'checkpoint';

const SCRUB_TYPE_BITMAP = 'bitmap';

const SCRUB_TYPE_BOTH = 'both';

const SCRUB_TYPE_ALL = '*';

/**
* General functions used by vmsnap.
*
Expand Down Expand Up @@ -119,7 +129,11 @@ const fileExists = async (path) => {
* @returns {Promise<boolean>} true if the scrubbing was successful, false if
* there was a failure.
*/
const scrubCheckpointsAndBitmaps = async (domains) => {
const scrubCheckpointsAndBitmaps = async ({
domains,
checkpointName,
scrubType,
}) => {
if (!domains) {
throw new Error('No domains specified', { code: ERR_DOMAINS });
}
Expand All @@ -132,9 +146,23 @@ const scrubCheckpointsAndBitmaps = async (domains) => {
for (const domain of await parseArrayParam(domains, fetchAllDomains)) {
logger.info(`Scrubbing domain: ${domain}`);

await cleanupCheckpoints(domain);

await cleanupBitmaps(domain);
if (scrubType === SCRUB_TYPE_CHECKPOINT) {
await cleanupCheckpoints(domain, checkpointName);
} else if (scrubType === SCRUB_TYPE_BITMAP) {
await cleanupBitmaps(domain, checkpointName);
} else if (scrubType === SCRUB_TYPE_BOTH) {
await cleanupCheckpoints(domain, checkpointName);

await cleanupBitmaps(domain, checkpointName);
} else if (scrubType === '*') {
await cleanupCheckpoints(domain);

await cleanupBitmaps(domain);
} else {
logger.error('No scrub type specified', {
code: ERR_INVALID_SCRUB_TYPE,
});
}
}

scrubbed = true;
Expand Down
129 changes: 58 additions & 71 deletions libs/libnbdbackup.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { spawn } from 'child_process';
import { rm } from 'fs/promises';
import dayjs from 'dayjs';
import advancedFormat from 'dayjs/plugin/advancedFormat.js';
import quarterOfYear from 'dayjs/plugin/quarterOfYear.js';
import { logger } from '../vmsnap.js';
import { cleanupCheckpoints, domainExists, fetchAllDomains } from './virsh.js';
import { fileExists, parseArrayParam } from './general.js';
Expand All @@ -17,6 +18,7 @@ import { cleanupBitmaps } from './qemu-img.js';
// The dayjs library with the advanced format plugin. We need quarter
// resolution for the backup directories.
dayjs.extend(advancedFormat);
dayjs.extend(quarterOfYear);

export const BACKUP = 'virtnbdbackup';

Expand All @@ -33,7 +35,7 @@ const FREQUENCY_QUARTERLY = 'quarter';
const FREQUENCY_YEARLY = 'year';

const PRUNING_FREQUENCIES = [
FREQUENCY_MONTHLY,
FREQUENCY_MONTHLY,
FREQUENCY_QUARTERLY,
FREQUENCY_YEARLY,
];
Expand All @@ -44,22 +46,29 @@ const PRUNING_FREQUENCIES = [
*
* @returns {string} the current months backup folder name
*/
const getBackupFolder = (groupBy = FREQUENCY_MONTHLY) => {
let backupFolder;
const getBackupFolder = (groupBy = FREQUENCY_MONTHLY, current = true) => {
let lastFolder;

switch (groupBy) {
case FREQUENCY_QUARTERLY:
backupFolder = dayjs().format(FORMAT_QUARTERLY);
lastFolder = current
? dayjs().format(FORMAT_QUARTERLY)
: dayjs().subtract(3, 'months').format(FORMAT_QUARTERLY);
break;
case FREQUENCY_YEARLY:
backupFolder = dayjs().format(FORMAT_YEARLY);
lastFolder = current
? dayjs().format(FORMAT_YEARLY)
: dayjs().subtract(1, 'year').format(FORMAT_YEARLY);
break;
case FREQUENCY_MONTHLY:
lastFolder = current
? dayjs().format(FORMAT_MONTHLY)
: dayjs().subtract(1, 'month').format(FORMAT_MONTHLY);
default:
backupFolder = dayjs().format(FORMAT_MONTHLY);
return undefined;
}

return `vmsnap-backup-${groupBy}ly-${backupFolder}`;
return `vmsnap-backup-${groupBy}ly-${lastFolder}`;
};

/**
Expand All @@ -68,13 +77,7 @@ const getBackupFolder = (groupBy = FREQUENCY_MONTHLY) => {
*
* @param {Object} args the command line arguments (domans, output, raw, prune)
*/
const performBackup = async ({
domains,
output,
raw,
groupBy,
prune,
}) => {
const performBackup = async ({ domains, output, raw, groupBy, prune }) => {
if (!domains) {
throw new Error('No domains specified', { code: ERR_DOMAINS });
}
Expand All @@ -84,7 +87,7 @@ const performBackup = async ({
}

for (const domain of await parseArrayParam(domains, fetchAllDomains)) {
if (await isCleanupRequired(domain, groupBy, prune, output)) {
if (await isCleanupRequired(domain, groupBy, output)) {
logger.info('Creating a new backup directory, running bitmap cleanup');

await cleanupCheckpoints(domain);
Expand Down Expand Up @@ -116,22 +119,21 @@ const performBackup = async ({
* @param {*} path the full backup directory path
* @returns {Promise<boolean>} true if cleanup is required, false otherwise
*/
const isCleanupRequired = async (domain, groupBy, pruneFrequency, path) => {
// We're not pruning, so no cleanup is required.
if (pruneFrequency === undefined || pruneFrequency === false) {
return false;
}

const currentBackupFolderExists = await fileExists(
const isCleanupRequired = async (domain, groupBy, path) => {
const backupFolderExists = await fileExists(
`${path}${sep}${domain}${sep}${getBackupFolder(groupBy)}`,
);

// Cleanup is required if the backup folder does not exist. We do this to
// ensure we don't overwrite the previous months backups and to establish a
// full backup for the start of the new period.
//
// If the backup folder exists, we assume the cleanup has already been done.
return currentBackupFolderExists === false;
if (!backupFolderExists) {
logger.info('Backup folder does not exist, cleanup required');

return true;
}

return false;
};

/**
Expand All @@ -144,25 +146,28 @@ const isCleanupRequired = async (domain, groupBy, pruneFrequency, path) => {
* @param {string} path the full backup directory path
* @returns true if pruning is required, false otherwise
*/
const isPruningRequired = async (
domain,
groupBy,
pruneFrequency,
path,
) => {
if (pruneFrequency === undefined || pruneFrequency === false) {
const isPruningRequired = async (domain, groupBy, pruneFrequency, path) => {
if (pruneFrequency === false) {
return false; // No pruning required
}

// If the window is not found, assume no pruning is required.
if (!PRUNING_FREQUENCIES.includes(groupBy)) {
logger.warn(`Invalid prune frequency: ${groupBy}. Pruning disabled`);
logger.warn(`Invalid groupBy: ${groupBy}. Pruning disabled`);

return false;
}

const previousBackupFolder = getBackupFolder(groupBy, false);

if (previousBackupFolder === undefined) {
logger.info('Unable to determine previous backup folder, skipping pruning');

return false;
}

const previousBackupFolderExists = await fileExists(
`${path}${sep}${domain}${sep}${getPreviousBackupFolder(groupBy)}`,
`${path}${sep}${domain}${sep}${previousBackupFolder}`,
);

if (!previousBackupFolderExists) {
Expand All @@ -171,9 +176,11 @@ const isPruningRequired = async (

// The number of days between the current date and the start of the backup
// period.
const days = getBackupStartDate(groupBy).diff(dayjs().date(), 'days');
const days = dayjs().diff(getBackupStartDate(groupBy), 'days');

switch (groupBy) {
logger.info(`Days since the start of the ${groupBy}: ${days}`);

switch (groupBy.toLowerCase()) {
case FREQUENCY_MONTHLY:
return days >= 15;
case FREQUENCY_QUARTERLY:
Expand All @@ -196,10 +203,16 @@ const isPruningRequired = async (
* @param {string} path the full backup directory path
*/
const pruneLastMonthsBackups = async (domain, groupBy, path) => {
const previousBackupFolder = getPreviousBackupFolder(groupBy);
const previousBackupFolder = getBackupFolder(groupBy, false);

if (previousBackupFolder === undefined) {
logger.info('Unable to determine previous backup folder, skipping pruning');

console.info(
`Pruning ${window} backup (${previousBackupFolder}) for ${domain}`,
return;
}

logger.info(
`Pruning ${groupBy}ly backup (${previousBackupFolder}) for ${domain}`,
);

await rm(`${path}${sep}${domain}${sep}${previousBackupFolder}`, {
Expand All @@ -209,42 +222,21 @@ const pruneLastMonthsBackups = async (domain, groupBy, path) => {
};

/**
* Finds and returns the start date for the backup period.
*
* @param {string} groupBy the frequency to prune the backups (monthly,
* quarterly, yearly)
* @param {string} groupBy the frequency to prune the backups (month, quarter,
* year)
* @returns {dayjs} the start date for the backup
*/
const getBackupStartDate = (groupBy) => {
switch (groupBy.toLowerCase) {
case FREQUENCY_MONTHLY:
return dayjs().startOf('month');
switch (groupBy.toLowerCase()) {
case FREQUENCY_QUARTERLY:
return dayjs().startOf('quarter');
case FREQUENCY_YEARLY:
return dayjs().startOf('year');
default:
throw new Error(`Invalid prune groupBy: ${groupBy}`);
}
};

/**
* Returns last months backup folder name.
*
* @param {string} groupBy the frequency to prune the backups (monthly,
* quarterly, yearly)
* @returns {string} Previous month in the format of YYYY-MM or the format for
* the previous period that matches the frequency of the groupBy.
*/
const getPreviousBackupFolder = (groupBy) => {
switch (groupBy.toLowerCase) {
case FREQUENCY_MONTHLY:
return dayjs().subtract(1, 'month').format(FORMAT_MONTHLY);
case FREQUENCY_QUARTERLY:
return dayjs().subtract(3, 'months').format(FORMAT_QUARTERLY);
case FREQUENCY_YEARLY:
return dayjs().subtract(1, 'year').format(FORMAT_YEARLY);
default:
throw new Error(`Invalid prune frequency: ${groupBy}`);
return dayjs().startOf('month');
}
};

Expand All @@ -253,12 +245,7 @@ const getPreviousBackupFolder = (groupBy) => {
*
* @param {Promise<string>} domain the domain to backup
*/
const backup = async (
domain,
outputDir,
raw,
groupBy,
) => {
const backup = async (domain, outputDir, raw, groupBy) => {
if (!(await domainExists(domain))) {
logger.warn(`${domain} does not exist`);

Expand Down
Loading

0 comments on commit 8852fa3

Please sign in to comment.