Perform Full Backup and Restore
Apache Cloudberry supports backing up and restoring the full database in parallel. Parallel operations scale regardless of the number of segments in your system, because segment hosts each write their data to local disk storage at the same time.
gpbackup and gprestore are Apache Cloudberry command-line utilities that create and restore backup sets for Apache Cloudberry. By default, gpbackup stores only the object metadata files and DDL files for a backup in the Apache Cloudberry coordinator data directory. Apache Cloudberry segments use the COPY ... ON SEGMENT command to store their data for backed-up tables in compressed CSV data files, located in each segment's backups directory.
The backup metadata files contain all of the information that gprestore needs to restore a full backup set in parallel. Each gpbackup task uses a single transaction in Apache Cloudberry. During this transaction, metadata is backed up on the coordinator host, and data for each table on each segment host is written to CSV backup files using COPY ... ON SEGMENT commands in parallel. The backup process acquires an ACCESS SHARE lock on each table that is backed up.
Install the gpbackup and gprestore utilities
Before installing the gpbackup and gprestore utilities, make sure that you have the Golang (v1.21 or later) installed and that you have set the Go PATH environment
variable:
## Set the Go `PATH` environment variable
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
-
Pull the
apache/cloudberry-backupGitHub repository to the target machine.go install github.com/apache/cloudberry-backup@latestThe repository is placed in
$GOPATH/pkg/mod/github.com/apache/cloudberry-backup. -
Enter the
apache/cloudberry-backupdirectory. Then, build and install the source code:cd <$GOPATH/pkg/mod/github.com/apache/cloudberry-backup>make dependmake buildYou might encounter the
fatal: Not a git repository (or any of the parent directories): .gitprompt after runningmake depend. Ignore this prompt, because this does not affect the building.The
buildtarget will put thegpbackupandgprestorebinaries in$HOME/go/bin. This operation will also try to copygpbackup_helperto the Apache Cloudberry segments (by retrieving hostnames fromgp_segment_configuration). -
Check whether the build is successful by checking whether your
$HOME/go/bindirectory containsgpback,gprestore, andgpbackup_helper.ls $HOME/go/bin -
Validate whether the installation is successful:
gpbackup --versiongprestore --version
Back up the full database
To perform a complete backup of a database, as well as Apache Cloudberry system metadata, use the command:
gpbackup --dbname <database_name>
For example:
$ gpbackup --dbname test_04
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-gpbackup version = 1.2.7-beta1+dev.7
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Apache Cloudberry Version = oudberry Database 1.0.0 build 5551471267
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Starting backup of database test_04
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Backup Timestamp = 20240108171718
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Backup Database = test_04
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Gathering table state information
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Acquiring ACCESS SHARE locks on tables
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Gathering additional table metadata
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Getting storage information
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[WARNING]:-No tables in backup set contain data. Performing metadata-only backup instead.
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Metadata will be written to /data0/coordinator/gpseg-1/backups/20240108/20240108171718/gpbackup_20240108171718_metadata.sql
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Writing global database metadata
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Global database metadata backup complete
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Writing pre-data metadata
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Pre-data metadata metadata backup complete
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Writing post-data metadata
20240108:17:17:18 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Post-data metadata backup complete
20240108:17:17:19 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Found neither /usr/local/cloudberry-1.0.0/bin/gp_email_contacts.yaml nor /home/gpadmin//gp_email_contacts.yaml
20240108:17:17:19 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Email containing gpbackup report /data0/coordinator/gpseg-1/backups/20240108/20240108171718/gpbackup_20240108171718_report will not be sent
20240108:17:17:19 gpbackup:gpadmin:cbdb-coordinator:001945-[INFO]:-Backup completed successfully
The above command creates a file that contains global and database-specific metadata on the Apache Cloudberry coordinator host in the default directory, $COORDINATOR_DATA_DIRECTORY/backups/<YYYYMMDD>/<YYYYMMDDHHMMSS>/. For example:
ls $COORDINATOR_DATA_DIRECTORY/backups/20240108/20240108171718
gpbackup_20240108171718_config.yaml gpbackup_20240108171718_report
gpbackup_20240108171718_metadata.sql gpbackup_20240108171718_toc.yaml
By default, each segment stores each table's data for the backup in a separate compressed CSV file in <seg_dir>/backups/<YYYYMMDD>/<YYYYMMDDHHMMSS>/. For example:
ls /data1/primary/gpseg1/backups/20240108/20240108171718/
gpbackup_0_20240108171718_17166.gz gpbackup_0_20240108171718_26303.gz
gpbackup_0_20240108171718_21816.gz
To consolidate all backup files into a single directory, include the --backup-dir option. Note that you need to specify an absolute path with this option:
$ gpbackup --dbname test_04 --backup-dir /home/gpadmin/backups
20240108:17:34:10 gpbackup:gpadmin:cbdb-coordinator:003348-[INFO]:-gpbackup version = 1.2.7-beta1+dev.7
20240108:17:34:10 gpbackup:gpadmin:cbdb-coordinator:003348-[INFO]:-Apache Cloudberry Version = oudberry Database 1.0.0 build 5551471267
...
20240108:17:34:12 gpbackup:gpadmin:cbdb-coordinator:003348-[INFO]:-Backup completed successfully
$ find /home/gpadmin/backups/ -type f
/home/gpadmin/backups/gpseg0/backups/20240108/20240108173410/gpbackup_0_20240108173410_16593.gz
/home/gpadmin/backups/gpseg-1/backups/20240108/20240108173410/gpbackup_20240108173410_config.yaml
/home/gpadmin/backups/gpseg-1/backups/20240108/20240108173410/gpbackup_20240108173410_report
/home/gpadmin/backups/gpseg-1/backups/20240108/20240108173410/gpbackup_20240108173410_toc.yaml
/home/gpadmin/backups/gpseg-1/backups/20240108/20240108173410/gpbackup_20240108173410_metadata.sql
/home/gpadmin/backups/gpseg1/backups/20240108/20240108173410/gpbackup_1_20240108173410_16593.gz
When performing a backup operation, you can use the --single-data-file in situations where the additional overhead of multiple files might be prohibitive. For example, if you use a third party storage solution such as Data Domain with backups.
Backing up a materialized view does not back up the materialized view data. Only the materialized view definition is backed up.