This guide explains how a IBM Cloud Object Storage System™ maintains data integrity.
This guide explains how IBM Cloud Object Storage System maintains data integrity through a group of software agents collectively known as the Rebuilder. It covers the scanning, rebuild, integrity, and synchronization agents, bandwidth usage for rebuild operations, and the theory of operation of the prioritized rebuilder.
The Rebuilder detects missing or corrupted Slices, reconstructs them from remaining Slices, and deprecates Slices that cannot be reconstructed. Rebuild activity is scheduled adaptively so that it becomes more aggressive when client I/O utilization drops and reduces its rate when client performance is affected.
| Full scan duration | 48 hours |
|---|---|
| Integrity checksum | 4-byte CRC32 checksum |
| Example annual data rebuild rate | 80 TB |
| Example total annual rebuild network traffic | 1600 TB |
| Example individual annual Slicestor rebuild traffic | 2.8 TB |
| Example per-appliance average rebuild traffic | 435.52 KBps |
The Rebuilder includes the Scanning Agent, Rebuild Agent, Integrity Agent, and Synchronization Agent.
A full scan should take 48 hours, so any issue with a Slice is detected within 48 hours.
When client operations increase, the Rebuilder decreases its activity; when client operations reduce, it increases rebuilding activity as much as possible without negatively impacting client operations.
The Slice is deprecated and then rebuilt from the other Slices for that Object on the Slicestor node.