This simple rules, maybe save your time, your data, your customers...
- Build region-free repository (as example for EU and US), make quorum and sync priority
- Any product or service configuration should be in repository (include encrypted secrets)
- Make sure - if you always can have shadow copy of your repository on cold or hot storage
- Always use Clusters and Replicas and one more shadow replica of your replica or master
- Make sure you have three different places with backups - local, network storage, private external cloud
- Use full and incremental databases backup together, encrypt your backup in air (before data saved on target)
- Use network software defined storages for Virtual Machines and Containers
- Make sure you have disaster recovery functions and place where you can migrate
- Try to build infrastructure as hot-spare and region-free
- Make sure you have Duty Engineers with the same skills as day Engineers
- Any major or minor network change can`t affect customers infrastructure
- You have all historical versions of network equipment configurations
- Use Network Software Defined Storages
- Use Network Attach Storages
- Always make sure you have backup drives
- Make shadow copy of your backup storages
- Try to build region-free storages
- Encrypt and anonymize any customer data and backup them - local, network storage, private external cloud
- Isolate any company finance data from you main infrastructure and backup them - local, network storage, private external cloud
- Always use way "All Denied -> Approval to Permit"
- Your engineers always know the best way to get out of an incident (Don't interfere with them, offer help and resources)
- Always listen when your team talks about a lack of infrastructure.
- Hire the best IT Architect
- Make sure Duty Engineers knows how to recover service from backup
- Build tests for recover your backup on isolated infrastructure
- Always use way "3 copy in different locations"
- Combine OpenSource and commercial solutions
- Write documents how to recovery from backups