Replication tasks - entire dataset keeps being resent

Description

I replicate my datasets to a remote server running freenas (11.3-U3.1). The snapshot tasks are set to NOT allow taking empty snapshots, and the lifetime is a week.

Some of my datasets aren't very active, and so even though the snapshot task runs every couple of hours, no new snapshots are taken because no new data has been written. If nothing is changed on a dataset, after a week I've noticed that the only remaining snapshot is deleted per it's scheduled lifetime, and then a new snapshot is created in it's place at a later point in time. When replication runs, an incremental stream/snapshot is not able to be generated and I get the following:

Replication "Main/Files - BackupPool/Mike/Files" failed: No incremental base on dataset 'Main/Files/Archive' and replication from scratch is not allowed..

The only way to allow replication to continue is to check "Replicate from scratch if incremental is not possible." However, this results in a large amount of data sent to the remote site (a few TB per month) even when no changes are made to the source dataset. "Hold Pending Snapshots" is checked, but does not resolve this issue.

Obviously this is less than ideal to have such an enormous amount of data sent when no new data has been written. I assume that if I check "Allow taking empty snapshots" then that would fix this problem as there would be a constant stream of snapshots for the replication task to compare against. However, I wonder if there is another way to address this without requiring the creation of hundreds of unneeded snapshots.

What I'm expecting to happen, is that if there is only one snapshot left for a dataset, it won't be deleted by the system until a new snapshot is created regardless of it's lifetime setting, or prevent it from being deleted until any replication tasks that rely on sending incremental data are completed. Even without accounting for replication issues, it doesn't make sense to me to allow the only remaining snapshot on a dataset to be deleted leaving no snapshots until the snapshot task runs again.

If this is all working as expected and I'm just doing it wrong, please let me know!

Problem/Justification

None

Impact

None

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

exo 
July 9, 2021 at 11:47 PM

I have a very similar issue.

I have put the repl task on debug but the error is the same.

I can see 8Tb on the destination and the first snapshot (which still exist in source).

Don't know how else to fix it. it has happened 3 times already.

Should I open a new bug report?

Vladimir Vinogradenko 
August 6, 2020 at 4:12 PM

it should be included in 12.0, we just don't have separate 12.0/12.1 branches for zettarepl yet, that's why there is no "12.0 PR" message. If it is not there yet, then you either have an old build, or it's a technical issue and the fix will be included with other replication fixes that are pending now.

Mike Gittelman 
August 6, 2020 at 4:05 PM

Former user This fix seems to be working well.  I noticed it's not being in included in 12.  Can we add it, or do we need to wait for 12.1 to come out?

Bug Clerk 
June 29, 2020 at 3:01 PM

Bug Clerk 
June 25, 2020 at 1:50 PM

Complete

Details

Assignee

Reporter

Impact

Components

Affects versions

Priority

More fields

Katalon Platform

Created June 18, 2020 at 8:11 PM
Updated July 1, 2022 at 4:52 PM
Resolved June 29, 2020 at 3:01 PM