Sometimes Time Machine backups can take a while. The backup process is getting throttled and therefore has not a very high priority.
Set process priority
But you can allow the Time Machine process to gather a higher priority and therefore speed up the backup overall with the following command:
sudo sysctl debug.lowpri_throttle_enabled=0
Reset process priority
To set the process back to it’s original priority we can just set it back to 1.
sudo sysctl debug.lowpri_throttle_enabled=1
For me this speeded up the backup process by other 100%.
See also my tutorial on how to setup a Raspberry Pi 4 as a Time Machine server: