- Shut down PostgreSQL if it's running.
- Check if you have current valid backups and/or replicas which are not already corrupt. If so, make sure these are protected.
- If there are no backups, make a full copy of the database files and all transaction logs.
Many things you will try to rescue a corrupt database or a damaged server can cause further destruction of data. Worse, certain kinds of corruption (caused by bad storage or bad RAM, for example) will spread to the replicas and/or your backups given time. In a worst case, you can fiddle with your damaged DB server long enough that the regularly scheduled backup kicks in ... and overwrites a good backup with a corrupt one.
Of course, maintaining a few days or weeks of rotational backups is an even better idea, and protects you against many forms of data loss. I recommend Barman or pgRollDump for this purpose.
Yep really any system and any systems administrator should have auto backups of everything (rotating) and any systems admin worth his salt should be taking a backup before performing maintenance, repairs, upgrades etc etc... I know for many systems admins after doing this job for a long time this can seem trivial when performing something done 1000 times flawlessly before.
ReplyDeleteBut a nightmare just takes one time.