-----============= acceptance-small: sanity-flr ============----- Thu Jul 30 19:49:19 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg613-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:49:36 (1785455376) === oleg613-client.virtnet: executing check_config_client /mnt/lustre oleg613-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg613-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9dc591a2b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9dc591a2b800.idle_timeout=debug disable quota as required oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:50:13 (1785455413) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanity-flr test 0a: lfs mirror create with -N option == 19:50:16 (1785455416) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:50:33 (1785455433) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:50:36 (1785455436) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:50:40 (1785455440) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:51:05 (1785455465) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:51:08 (1785455468) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:51:12 (1785455472) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9dc591a2b800.stats=clear osc.lustre-OST0001-osc-ffff9dc591a2b800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:51:20 (1785455480) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:51:30 (1785455490) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:51:38 (1785455498) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:51:41 (1785455501) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:51:48 (1785455508) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 19:51:57 (1785455517) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:52:04 (1785455524) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:52:07 (1785455527) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:52:13 (1785455533) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:52:24 (1785455544) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:52:31 (1785455551) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.736357 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217898 s, 4.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:52:41 (1785455561) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0568862 s, 36.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:52:50 (1785455570) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:53:01 (1785455581) Stopping /mnt/lustre-ost1 (opts:) on oleg613-server oleg613-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg613-server: oleg613-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg613-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg613-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg613-server oleg613-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg613-server: oleg613-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg613-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (130s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:55:12 (1785455712) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.373189 s, 16.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg613-server oleg613-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg613-server: oleg613-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg613-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:55:54 (1785455754) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg613-server oleg613-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg613-server: oleg613-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg613-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg613-server oleg613-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg613-server: oleg613-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg613-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:57:05 (1785455825) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.60913 s, 11.4 MB/s -rw-r--r-- 1 root root 52428800 Jul 30 19:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.036s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg613-server oleg613-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9dc591a2b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.044s user 0m0.000s sys 0m0.189s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg613-server: oleg613-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg613-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg613-server oleg613-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.152s user 0m0.002s sys 0m0.174s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg613-server: oleg613-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg613-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg613-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9dc591a2b800.ost_server_uuid in FULL state after 0 sec PASS 33b (86s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:58:31 (1785455911) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:58:34 (1785455914) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:58:38 (1785455918) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 19:58:41 (1785455921) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 19:58:49 (1785455929) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.78611 s, 16.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:59:02 (1785455942) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390907 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364627 s, 28.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:59:09 (1785455949) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:59:18 (1785455958) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000890583 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00476301 s, 215 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00739045 s, 139 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0041277 s, 248 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 19:59:29 (1785455969) create test files with size 15740558 .. 003942eda2adf78799eea494f0ec9893 - 63d00e6a339b984ca73254642b0625b1 - c477a2a99add4c94df1e6e9292d9f6c2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg613-client.virtnet /mnt/lustre (opts:) Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 19:59:51 (1785455991) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:00:06 (1785456006) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:00:12 (1785456012) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142577 s, 7.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191854 s, 54.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 20:00:21 (1785456021) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128684 s, 16.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.375512 s, 11.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.310094 s, 6.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 20:00:36 (1785456036) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:00:40 (1785456040) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:00:43 (1785456043) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255528 s, 41.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 20:00:51 (1785456051) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:00:56 (1785456056) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:01:08 (1785456068) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.240724 s, 43.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg613-client.virtnet /mnt/lustre (opts:) Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:01:19 (1785456079) llite.lustre-ffff9dc583f24800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.317149 s, 33.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg613-client.virtnet /mnt/lustre (opts:) Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9dc586831000.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:01:31 (1785456091) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:02:05 (1785456125) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:02:40 (1785456160) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:02:43 (1785456163) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.317493 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.202414 s, 15.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd47b732 (nosync,stale) Mirror 2: 0xd47b732 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d4cc4371447c96c9f2006925089646fd - mirror 1 checksum: 9ade4727d21b82e964020122663db000 - mirror 2 checksum: d4cc4371447c96c9f2006925089646fd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T20:02:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-30T20:02:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d4cc4371447c96c9f2006925089646fd - mirror 1 checksum: 9ade4727d21b82e964020122663db000 - mirror 2 checksum: 9ade4727d21b82e964020122663db000 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T20:02:46-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-30T20:02:46-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:02:56 (1785456176) Failing mds1 on oleg613-server Stopping /mnt/lustre-mds1 (opts:) on oleg613-server 20:03:02 (1785456182) shut down facet: mds1 facet_host: oleg613-server facet_failover_host: oleg613-server Failover mds1 to oleg613-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg613-server: oleg613-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg613-client: oleg613-server: ssh exited with exit code 1 Started lustre-MDT0000 20:03:25 (1785456205) targets are mounted 20:03:25 (1785456205) facet_failover done oleg613-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232028 s, 45.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:03:42 (1785456222) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:03:45 (1785456225) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:03:48 (1785456228) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0157189 s, 1.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:03:56 (1785456236) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:03:59 (1785456239) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00555523 s, 737 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0972913 s, 42.1 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:04:06 (1785456246) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:04:28 (1785456268) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:04:57 (1785456297) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:05:21 (1785456321) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:05:45 (1785456345) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 20:05:52 (1785456352) mirror create pid 54249 mirror split pid 54250 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54249 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 54250 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 20:06:59 (1785456419) Seed set to 6421 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1785456421.479995 mapwrite 0x01425e thru 0x02079e (0xc541 bytes) 000004 1785456421.629033 read 0x00be97 thru 0x01ab22 (0xec8c bytes) 000008 1785456421.662788 mapwrite 0x01245c thru 0x01d5ee (0xb193 bytes) 000009 1785456421.782541 MIRROR_EXTEND to 2 mirrors 000011 1785456422.675033 mapread 0x00b3dc thru 0x00d615 (0x223a bytes) 000012 1785456422.728815 mapwrite 0x0361d8 thru 0x03ffff (0x9e28 bytes) 000015 1785456423.022660 read 0x01ceda thru 0x01e7cc (0x18f3 bytes) 000016 1785456423.033138 mapwrite 0x032d86 thru 0x035082 (0x22fd bytes) 000021 1785456423.145143 write 0x003b80 thru 0x00d479 (0x98fa bytes) 000022 1785456423.168780 mapwrite 0x012fe2 thru 0x01a2be (0x72dd bytes) 000023 1785456423.325706 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000024 1785456423.560413 mapread 0x025cc6 thru 0x028860 (0x2b9b bytes) 000025 1785456423.626348 mapwrite 0x03c43f thru 0x03ffff (0x3bc1 bytes) truncating to largest ever: 0x7101 000030 1785456423.746660 trunc from 0x040000 to 0x007100 (0x38eff bytes) truncating to largest ever: 0x1b11a 000032 1785456423.780739 trunc from 0x007101 to 0x01b119 (0x14019 bytes) 000034 1785456423.811675 read 0x01adc9 thru 0x01b119 (0x351 bytes) 000035 1785456423.826719 mapread 0x008eed thru 0x00df49 (0x505d bytes) 000036 1785456423.852340 mapwrite 0x0336e3 thru 0x03d4ca (0x9de8 bytes) 000037 1785456424.001198 mapwrite 0x00deb2 thru 0x00f20f (0x135e bytes) 000040 1785456424.065963 MIRROR_EXTEND to 2 mirrors 000041 1785456424.900759 mapread 0x0337f6 thru 0x03d4ca (0x9cd5 bytes) 000046 1785456424.993834 write 0x000802 thru 0x00ede1 (0xe5e0 bytes) 000047 1785456425.094202 read 0x027fae thru 0x02acda (0x2d2d bytes) 000049 1785456425.103509 mapread 0x029e55 thru 0x029f9f (0x14b bytes) 000052 1785456425.107600 mapread 0x008f7d thru 0x012adc (0x9b60 bytes) 000053 1785456425.122320 mapwrite 0x01044f thru 0x017e2e (0x79e0 bytes) 000054 1785456425.190074 trunc from 0x03d4cb to 0x017cd0 (0x257fa bytes) 000056 1785456425.250756 read 0x00eb11 thru 0x0137c6 (0x4cb6 bytes) 000057 1785456425.258986 MIRROR_RESYNC 2 mirrors 000059 1785456426.086426 mapwrite 0x011dcc thru 0x013ea5 (0x20da bytes) 000061 1785456426.280595 mapwrite 0x03a741 thru 0x03c1ad (0x1a6d bytes) truncating to largest ever: 0x1bcef 000062 1785456426.408919 trunc from 0x03c1ae to 0x01bcee (0x204bf bytes) 000063 1785456426.433383 read 0x0129ee thru 0x01bcee (0x9301 bytes) truncating to largest ever: 0x22dec 000065 1785456426.441578 trunc from 0x01bcef to 0x022deb (0x70fd bytes) 000067 1785456426.485556 mapread 0x01e929 thru 0x022deb (0x44c3 bytes) 000068 1785456426.501544 read 0x0183be thru 0x01f223 (0x6e66 bytes) 000069 1785456426.502340 mapwrite 0x03dfdb thru 0x03ffff (0x2025 bytes) 000070 1785456426.599881 trunc from 0x040000 to 0x00c0ee (0x33f11 bytes) 000071 1785456426.639610 mapwrite 0x01567f thru 0x0206aa (0xb02c bytes) 000073 1785456426.836563 read 0x005c31 thru 0x01577e (0xfb4e bytes) 000074 1785456426.863761 trunc from 0x0206ab to 0x01288d (0xde1d bytes) 000075 1785456426.904416 mapread 0x00ce35 thru 0x01288d (0x5a59 bytes) 000076 1785456426.920617 mapread 0x000b2f thru 0x00b3a5 (0xa877 bytes) 000077 1785456426.925671 write 0x012728 thru 0x013167 (0xa40 bytes) truncating to largest ever: 0x3c6a8 000079 1785456426.939303 trunc from 0x013168 to 0x03c6a7 (0x29540 bytes) 000080 1785456426.972468 mapread 0x030c93 thru 0x03c6a7 (0xba15 bytes) 000081 1785456427.021981 write 0x01c819 thru 0x02723a (0xaa22 bytes) 000082 1785456427.030229 read 0x013a49 thru 0x0229df (0xef97 bytes) 000084 1785456427.044355 mapread 0x01a8da thru 0x028edf (0xe606 bytes) 000085 1785456427.050803 MIRROR_RESYNC 2 mirrors 000086 1785456427.811046 read 0x02d236 thru 0x03aa9e (0xd869 bytes) 000087 1785456427.832277 mapread 0x018774 thru 0x0230c7 (0xa954 bytes) 000091 1785456427.923835 write 0x0383f4 thru 0x03ffff (0x7c0c bytes) 000092 1785456428.067989 MIRROR_RESYNC 2 mirrors 000093 1785456428.940941 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000095 1785456429.100080 MIRROR_EXTEND to 2 mirrors 000096 1785456429.998481 mapread 0x0246a2 thru 0x02b815 (0x7174 bytes) 000097 1785456430.078270 write 0x0224a4 thru 0x0224f8 (0x55 bytes) 000099 1785456430.190965 read 0x00fd30 thru 0x012eb7 (0x3188 bytes) 000102 1785456430.201508 trunc from 0x040000 to 0x0326ab (0xd954 bytes) 000105 1785456430.226362 write 0x0086eb thru 0x00deb8 (0x57ce bytes) 000106 1785456430.243490 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x326ac): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ac): Mirror 2: 0xa62bf443 (stale) CRC-32 checksum value for chunk [0, 0x326ac): Mirror 1: 0xd6756613 Mirror 3: 0xd6756613 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ac: skipped 000107 1785456431.343354 mapwrite 0x00a0c7 thru 0x00c20e (0x2148 bytes) 000109 1785456431.536560 read 0x00357d thru 0x0054ca (0x1f4e bytes) 000110 1785456431.548558 read 0x0221c8 thru 0x0271c4 (0x4ffd bytes) 000112 1785456431.564830 mapwrite 0x03ce94 thru 0x03ffff (0x316c bytes) 000113 1785456431.673793 write 0x02ea34 thru 0x0391c7 (0xa794 bytes) 000114 1785456431.676718 mapwrite 0x03fe8c thru 0x03ffff (0x174 bytes) 000115 1785456431.725502 write 0x039b55 thru 0x03efa5 (0x5451 bytes) 000116 1785456431.731765 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xce9ee413 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xce9ee413 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000117 1785456432.041494 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000118 1785456432.167146 read 0x03a740 thru 0x03ffff (0x58c0 bytes) 000119 1785456432.176356 mapwrite 0x0169b2 thru 0x01d50f (0x6b5e bytes) 000120 1785456432.236442 read 0x02ffe9 thru 0x0320a9 (0x20c1 bytes) 000121 1785456432.247230 write 0x00be6a thru 0x00c908 (0xa9f bytes) 000123 1785456432.248787 mapread 0x01458f thru 0x0193a6 (0x4e18 bytes) 000124 1785456432.250849 trunc from 0x040000 to 0x009840 (0x367bf bytes) 000126 1785456432.270050 trunc from 0x009841 to 0x0004c8 (0x9378 bytes) 000129 1785456432.288618 mapwrite 0x01073d thru 0x016ee5 (0x67a9 bytes) 000130 1785456432.383385 mapwrite 0x00f838 thru 0x0183ff (0x8bc8 bytes) 000131 1785456432.461595 read 0x010c65 thru 0x0183ff (0x779b bytes) 000133 1785456432.462874 trunc from 0x018400 to 0x019e3a (0x1a3b bytes) 000134 1785456432.484272 mapwrite 0x02ccf2 thru 0x02de13 (0x1122 bytes) truncating to largest ever: 0x3fd84 000135 1785456432.547019 trunc from 0x02de14 to 0x03fd83 (0x11f70 bytes) 000136 1785456432.562479 write 0x02bcff thru 0x0313e1 (0x56e3 bytes) 000137 1785456432.568197 mapwrite 0x03cafb thru 0x03ffff (0x3505 bytes) 000140 1785456432.669588 mapread 0x036810 thru 0x03c7d3 (0x5fc4 bytes) 000141 1785456432.670873 write 0x0330cb thru 0x03ffff (0xcf35 bytes) 000143 1785456432.675521 mapread 0x036076 thru 0x0383a8 (0x2333 bytes) 000144 1785456432.677091 write 0x0254a0 thru 0x033717 (0xe278 bytes) 000145 1785456432.679960 mapread 0x006290 thru 0x00731e (0x108f bytes) 000146 1785456432.681918 write 0x02f6b0 thru 0x03b3b6 (0xbd07 bytes) 000147 1785456432.683531 read 0x001a41 thru 0x004e62 (0x3422 bytes) 000157 1785456432.694349 mapwrite 0x03df37 thru 0x03ffff (0x20c9 bytes) 000158 1785456432.770804 mapread 0x017356 thru 0x01de6c (0x6b17 bytes) 000159 1785456432.773951 write 0x03fa17 thru 0x03ffff (0x5e9 bytes) 000161 1785456432.777584 mapread 0x0209cc thru 0x02a162 (0x9797 bytes) 000162 1785456432.781462 trunc from 0x040000 to 0x02f179 (0x10e86 bytes) 000164 1785456432.808607 mapwrite 0x032225 thru 0x0379c8 (0x57a4 bytes) 000168 1785456432.936717 trunc from 0x0379c9 to 0x00c7a6 (0x2b222 bytes) 000173 1785456432.957245 mapread 0x00ab50 thru 0x00c7a6 (0x1c57 bytes) 000174 1785456432.958409 trunc from 0x00c7a7 to 0x01ac93 (0xe4ed bytes) 000175 1785456432.997970 trunc from 0x01ac94 to 0x01f17c (0x44e9 bytes) 000179 1785456433.024816 mapread 0x0118b4 thru 0x01cb19 (0xb266 bytes) 000180 1785456433.087112 mapwrite 0x00e47a thru 0x01de4d (0xf9d4 bytes) 000183 1785456433.185680 mapwrite 0x015252 thru 0x01d69e (0x844d bytes) 000185 1785456433.227875 read 0x010241 thru 0x017e17 (0x7bd7 bytes) 000186 1785456433.233581 MIRROR_EXTEND to 2 mirrors 000187 1785456433.855102 mapwrite 0x030632 thru 0x03990f (0x92de bytes) 000189 1785456434.198567 mapread 0x0207da thru 0x029b82 (0x93a9 bytes) 000190 1785456434.227828 mapread 0x0243a0 thru 0x02525b (0xebc bytes) 000192 1785456434.230814 mapread 0x028993 thru 0x02ffd4 (0x7642 bytes) 000194 1785456434.235578 write 0x0320a8 thru 0x03acdf (0x8c38 bytes) 000196 1785456434.239594 write 0x01bdfc thru 0x01f331 (0x3536 bytes) 000197 1785456434.244434 write 0x01dbd0 thru 0x02709c (0x94cd bytes) 000199 1785456434.255868 MIRROR_RESYNC 2 mirrors 000200 1785456435.283924 mapread 0x0028f1 thru 0x0070fc (0x480c bytes) 000202 1785456435.305614 mapread 0x030c51 thru 0x037060 (0x6410 bytes) 000204 1785456435.324230 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000205 1785456435.452515 write 0x007fd9 thru 0x013de0 (0xbe08 bytes) 000206 1785456435.474604 mapwrite 0x016dcf thru 0x026dbc (0xffee bytes) 000207 1785456435.585356 MIRROR_EXTEND to 2 mirrors 000208 1785456436.351821 mapread 0x0293a3 thru 0x034c2e (0xb88c bytes) 000209 1785456436.390912 mapread 0x00dad4 thru 0x013b18 (0x6045 bytes) 000210 1785456436.415384 mapwrite 0x006af6 thru 0x00f9dd (0x8ee8 bytes) 000212 1785456436.591040 mapwrite 0x034687 thru 0x03ffff (0xb979 bytes) 000213 1785456436.699468 write 0x012fac thru 0x01d3bd (0xa412 bytes) 000219 1785456436.724480 mapread 0x03ad7d thru 0x03ffff (0x5283 bytes) 000221 1785456436.741047 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000223 1785456436.985833 read 0x0189b7 thru 0x01e587 (0x5bd1 bytes) 000224 1785456436.997599 read 0x0025f4 thru 0x005245 (0x2c52 bytes) 000225 1785456437.019692 mapread 0x02a793 thru 0x032297 (0x7b05 bytes) 000227 1785456437.025031 write 0x030a06 thru 0x034bf1 (0x41ec bytes) 000228 1785456437.026366 read 0x036ab6 thru 0x039fbe (0x3509 bytes) 000230 1785456437.030084 write 0x0008bc thru 0x00cab7 (0xc1fc bytes) 000231 1785456437.032519 trunc from 0x040000 to 0x02a923 (0x156dc bytes) 000233 1785456437.056192 mapread 0x003a8f thru 0x0093da (0x594c bytes) 000237 1785456437.061252 mapwrite 0x00b660 thru 0x011c1e (0x65bf bytes) 000238 1785456437.106485 MIRROR_EXTEND to 2 mirrors 000240 1785456437.752101 trunc from 0x02a924 to 0x004530 (0x263f3 bytes) 000241 1785456437.931006 mapwrite 0x02b5be thru 0x031cb2 (0x66f5 bytes) 000243 1785456438.137901 read 0x02d20e thru 0x02dd0d (0xb00 bytes) 000244 1785456438.149063 MIRROR_RESYNC 2 mirrors 000245 1785456438.984516 mapread 0x004315 thru 0x0115a5 (0xd291 bytes) 000247 1785456439.016092 mapread 0x00a796 thru 0x0127e5 (0x8050 bytes) 000248 1785456439.017269 read 0x01b2f0 thru 0x021161 (0x5e72 bytes) 000249 1785456439.023851 trunc from 0x031cb3 to 0x02aa11 (0x72a1 bytes) 000253 1785456439.141183 read 0x000840 thru 0x00c305 (0xbac6 bytes) 000254 1785456439.152430 mapwrite 0x03e44d thru 0x03ffff (0x1bb3 bytes) 000256 1785456439.282679 MIRROR_RESYNC 2 mirrors 000258 1785456440.222108 write 0x01ffcc thru 0x02f582 (0xf5b7 bytes) 000261 1785456440.298100 trunc from 0x040000 to 0x01641c (0x29be3 bytes) 000266 1785456440.366224 mapread 0x00c70c thru 0x01641c (0x9d11 bytes) 000268 1785456440.407557 mapread 0x007615 thru 0x0131e9 (0xbbd5 bytes) 000269 1785456440.412801 trunc from 0x01641d to 0x031f4c (0x1bb30 bytes) 000270 1785456440.433008 write 0x01a320 thru 0x028ff4 (0xecd5 bytes) 000271 1785456440.450074 read 0x02136f thru 0x02c43a (0xb0cc bytes) 000273 1785456440.458384 mapwrite 0x00b2ae thru 0x01b289 (0xffdc bytes) 000276 1785456440.525925 trunc from 0x031f4d to 0x00ff52 (0x21ffa bytes) 000278 1785456440.561233 read 0x008dc8 thru 0x00ff52 (0x718b bytes) 000280 1785456440.565310 mapwrite 0x00cf8d thru 0x019740 (0xc7b4 bytes) 000281 1785456440.655121 mapread 0x008dcc thru 0x00c79b (0x39d0 bytes) 000283 1785456440.658306 mapread 0x018fdb thru 0x019150 (0x176 bytes) 000284 1785456440.659905 mapwrite 0x023a6a thru 0x029268 (0x57ff bytes) 000286 1785456440.733703 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29269): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29269): Mirror 2: 0x2fbc4393 (stale) CRC-32 checksum value for chunk [0, 0x29269): Mirror 1: 0xffecd645 Mirror 3: 0xffecd645 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29269: skipped 000287 1785456441.858701 write 0x00c8d2 thru 0x00d120 (0x84f bytes) 000288 1785456441.962586 write 0x01239d thru 0x01ff99 (0xdbfd bytes) 000289 1785456441.976128 mapread 0x000cc7 thru 0x004ab2 (0x3dec bytes) 000291 1785456442.007415 trunc from 0x029269 to 0x015a66 (0x13802 bytes) 000292 1785456442.038512 trunc from 0x015a67 to 0x0198e1 (0x3e7b bytes) 000293 1785456442.055929 trunc from 0x0198e2 to 0x001d60 (0x17b81 bytes) 000294 1785456442.079772 mapwrite 0x0220e4 thru 0x02d1c4 (0xb0e1 bytes) 000295 1785456442.158956 trunc from 0x02d1c5 to 0x0180b7 (0x1510d bytes) 000297 1785456442.178602 trunc from 0x0180b8 to 0x03f620 (0x27569 bytes) 000298 1785456442.200252 write 0x01fd57 thru 0x0206f8 (0x9a2 bytes) 000299 1785456442.216617 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f621): Mirror 2: 0xe82f7eae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f621): Mirror 2: 0xe82f7eae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f621: skipped 000301 1785456442.568234 write 0x01f7e8 thru 0x025a00 (0x6219 bytes) 000302 1785456442.578050 trunc from 0x03f621 to 0x029bb7 (0x15a69 bytes) 000303 1785456442.592222 mapread 0x00df1e thru 0x00e7f0 (0x8d3 bytes) 000306 1785456442.603784 trunc from 0x029bb8 to 0x02325e (0x6959 bytes) 000307 1785456442.622403 mapwrite 0x004cc0 thru 0x00e9df (0x9d20 bytes) 000308 1785456442.675558 write 0x02af7a thru 0x02ed66 (0x3ded bytes) 000309 1785456442.676931 write 0x0044cc thru 0x011f49 (0xda7e bytes) 000310 1785456442.683313 trunc from 0x02ed67 to 0x02dcd9 (0x108d bytes) 000311 1785456442.701097 trunc from 0x02dcda to 0x0377a8 (0x9acf bytes) 000312 1785456442.714152 trunc from 0x0377a9 to 0x01d346 (0x1a462 bytes) 000313 1785456442.731556 trunc from 0x01d347 to 0x03da69 (0x20723 bytes) 000316 1785456442.752343 write 0x02622f thru 0x02b454 (0x5226 bytes) 000317 1785456442.771144 trunc from 0x03da6a to 0x01f2d0 (0x1e799 bytes) 000318 1785456442.803676 mapwrite 0x037b10 thru 0x03a701 (0x2bf2 bytes) 000319 1785456442.921450 read 0x024d50 thru 0x0301e5 (0xb496 bytes) 000322 1785456442.960823 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000324 1785456443.118372 write 0x0361cb thru 0x03ffff (0x9e35 bytes) 000325 1785456443.127957 read 0x022f79 thru 0x02adef (0x7e77 bytes) 000326 1785456443.161268 MIRROR_EXTEND to 2 mirrors 000328 1785456444.274232 MIRROR_RESYNC 2 mirrors 000329 1785456444.662556 mapread 0x03ac34 thru 0x03ec91 (0x405e bytes) 000330 1785456444.726501 trunc from 0x040000 to 0x02e920 (0x116df bytes) 000331 1785456444.869835 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e921): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e921): Mirror 2: 0x9699f7c (stale) CRC-32 checksum value for chunk [0, 0x2e921): Mirror 1: 0x9699f7c Mirror 3: 0x9699f7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e921: skipped 000333 1785456445.928894 trunc from 0x02e921 to 0x03c243 (0xd923 bytes) 000335 1785456446.096280 mapwrite 0x0057ff thru 0x00ca62 (0x7264 bytes) 000337 1785456446.164514 mapwrite 0x03ead8 thru 0x03ffff (0x1528 bytes) 000338 1785456446.226516 trunc from 0x040000 to 0x02df7a (0x12085 bytes) 000340 1785456446.245017 write 0x0069f0 thru 0x00d45d (0x6a6e bytes) 000341 1785456446.251826 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2df7b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df7b): Mirror 2: 0xd9ff6867 (stale) Mirror 3: 0xd9ff6867 (stale) CRC-32 checksum value for chunk [0, 0x2df7b): Mirror 1: 0x24b9e190 Mirror 4: 0x24b9e190 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df7b: skipped 000343 1785456447.481667 write 0x03be93 thru 0x03fb08 (0x3c76 bytes) 000344 1785456447.565159 write 0x02703e thru 0x036bd4 (0xfb97 bytes) 000346 1785456447.581253 mapwrite 0x0399b7 thru 0x03ffff (0x6649 bytes) 000349 1785456447.676721 mapwrite 0x0256ff thru 0x029600 (0x3f02 bytes) 000350 1785456447.726035 mapwrite 0x0199d9 thru 0x022f4f (0x9577 bytes) 000354 1785456447.775391 read 0x03a766 thru 0x03e0f9 (0x3994 bytes) 000355 1785456447.776171 mapread 0x03fbbb thru 0x03ffff (0x445 bytes) 000356 1785456447.778201 trunc from 0x040000 to 0x02df65 (0x1209a bytes) 000358 1785456447.808868 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2df66): Mirror 2: 0x1604e487 (stale) Mirror 4: 0x70e55a5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df66): Mirror 2: 0x1604e487 (stale) Mirror 4: 0x70e55a5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df66: skipped 000360 1785456448.199896 trunc from 0x02df66 to 0x01e802 (0xf763 bytes) 000363 1785456448.233830 trunc from 0x01e803 to 0x000149 (0x1e6b9 bytes) 000365 1785456448.275303 MIRROR_RESYNC 3 mirrors 000366 1785456449.133508 read 0x00002f thru 0x000149 (0x11b bytes) 000369 1785456449.140220 trunc from 0x00014a to 0x0090b2 (0x8f69 bytes) 000370 1785456449.208208 mapread 0x0078b7 thru 0x0090b2 (0x17fc bytes) 000372 1785456449.228050 mapwrite 0x0088bc thru 0x00e698 (0x5ddd bytes) 000373 1785456449.302923 MIRROR_RESYNC 3 mirrors 000374 1785456449.931281 read 0x00cb06 thru 0x00e698 (0x1b93 bytes) 000375 1785456449.938262 write 0x002d99 thru 0x00ddcb (0xb033 bytes) 000377 1785456450.038159 MIRROR_RESYNC 3 mirrors 000379 1785456451.211534 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000380 1785456451.521600 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000384 1785456451.719200 write 0x022ebb thru 0x0278c8 (0x4a0e bytes) 000385 1785456451.733810 mapread 0x00ec52 thru 0x01abd6 (0xbf85 bytes) 000386 1785456451.761023 write 0x01b3b2 thru 0x02659c (0xb1eb bytes) 000387 1785456451.762509 write 0x01cd92 thru 0x02247f (0x56ee bytes) 000389 1785456451.765372 read 0x013c4a thru 0x019dad (0x6164 bytes) 000391 1785456451.767288 write 0x007303 thru 0x01418a (0xce88 bytes) 000392 1785456451.772926 trunc from 0x0278c9 to 0x0250b5 (0x2813 bytes) 000394 1785456451.789478 mapwrite 0x02acca thru 0x02b4c4 (0x7fb bytes) 000395 1785456451.866247 mapwrite 0x031e3a thru 0x03ffff (0xe1c6 bytes) 000398 1785456451.941237 mapread 0x02499f thru 0x032038 (0xd69a bytes) 000399 1785456451.943728 trunc from 0x040000 to 0x0221b1 (0x1de4e bytes) 000400 1785456451.958744 mapwrite 0x0352f1 thru 0x03f4a9 (0xa1b9 bytes) 000401 1785456452.097016 read 0x02fba8 thru 0x030bec (0x1045 bytes) 000402 1785456452.099059 mapread 0x012631 thru 0x01ba00 (0x93d0 bytes) 000403 1785456452.102601 read 0x0094a0 thru 0x00ed65 (0x58c6 bytes) 000405 1785456452.106245 mapwrite 0x0362e0 thru 0x03ffff (0x9d20 bytes) 000406 1785456452.213611 trunc from 0x040000 to 0x023994 (0x1c66b bytes) 000407 1785456452.240597 read 0x01d216 thru 0x023994 (0x677f bytes) 000408 1785456452.266533 write 0x02910d thru 0x035023 (0xbf17 bytes) 000410 1785456452.272478 read 0x00d86b thru 0x01b4d2 (0xdc68 bytes) 000411 1785456452.275234 mapread 0x012f22 thru 0x0179ad (0x4a8c bytes) 000414 1785456452.282148 write 0x003ac0 thru 0x008a10 (0x4f51 bytes) 000415 1785456452.284878 read 0x01a851 thru 0x020dc7 (0x6577 bytes) 000417 1785456452.287907 write 0x028c66 thru 0x02fd8e (0x7129 bytes) 000418 1785456452.308573 mapwrite 0x000804 thru 0x008745 (0x7f42 bytes) 000419 1785456452.387580 MIRROR_EXTEND to 2 mirrors 000421 1785456453.008011 MIRROR_EXTEND to 3 mirrors 000423 1785456453.821542 trunc from 0x035024 to 0x03e2e6 (0x92c3 bytes) 000425 1785456453.960216 MIRROR_RESYNC 3 mirrors 000426 1785456454.803957 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000429 1785456455.179347 mapread 0x016871 thru 0x025c20 (0xf3b0 bytes) 000431 1785456455.213271 trunc from 0x03e2e7 to 0x010b8e (0x2d758 bytes) 000434 1785456455.316593 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000436 1785456455.458570 write 0x03f701 thru 0x03ffff (0x8ff bytes) 000437 1785456455.461829 write 0x0022b0 thru 0x008f81 (0x6cd2 bytes) 000438 1785456455.492145 mapread 0x007ec5 thru 0x0091e9 (0x1325 bytes) 000440 1785456455.514580 trunc from 0x040000 to 0x027a67 (0x18598 bytes) 000442 1785456455.559201 mapread 0x01aa8d thru 0x02182e (0x6da2 bytes) 000443 1785456455.593445 trunc from 0x027a68 to 0x012365 (0x15702 bytes) 000445 1785456455.617550 MIRROR_EXTEND to 2 mirrors 000448 1785456456.572072 mapwrite 0x013caf thru 0x01a323 (0x6675 bytes) 000449 1785456456.744423 trunc from 0x01a324 to 0x00f935 (0xa9ee bytes) 000450 1785456456.775205 write 0x022b2f thru 0x02f482 (0xc954 bytes) 000452 1785456456.780200 trunc from 0x02f483 to 0x01990c (0x15b76 bytes) 000453 1785456456.810506 trunc from 0x01990d to 0x016af2 (0x2e1a bytes) 000455 1785456456.865046 mapwrite 0x005f68 thru 0x01316a (0xd203 bytes) 000456 1785456456.989768 mapwrite 0x03ca10 thru 0x03f45d (0x2a4e bytes) 000458 1785456457.098319 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000459 1785456457.301985 write 0x025944 thru 0x030ff4 (0xb6b1 bytes) 000467 1785456457.343494 mapwrite 0x029640 thru 0x03352b (0x9eec bytes) 000469 1785456457.437337 write 0x02d473 thru 0x02e537 (0x10c5 bytes) 000470 1785456457.439202 write 0x00d535 thru 0x014889 (0x7355 bytes) 000473 1785456457.462223 write 0x039784 thru 0x03ffff (0x687c bytes) 000474 1785456457.464228 mapwrite 0x039925 thru 0x03ffff (0x66db bytes) 000475 1785456457.508191 read 0x02c3c9 thru 0x03bd58 (0xf990 bytes) 000477 1785456457.511570 trunc from 0x040000 to 0x03744a (0x8bb5 bytes) 000478 1785456457.536785 mapwrite 0x0079b1 thru 0x00f06b (0x76bb bytes) 000479 1785456457.626091 trunc from 0x03744b to 0x03f453 (0x8009 bytes) 000480 1785456457.653565 MIRROR_EXTEND to 2 mirrors 000482 1785456458.697068 mapread 0x01725a thru 0x0186ec (0x1493 bytes) 000483 1785456458.725307 MIRROR_RESYNC 2 mirrors 000484 1785456459.133196 read 0x018f2b thru 0x01d31b (0x43f1 bytes) 000485 1785456459.213031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000486 1785456459.427598 mapread 0x0112a4 thru 0x015797 (0x44f4 bytes) 000489 1785456459.482190 mapread 0x039e50 thru 0x03f453 (0x5604 bytes) 000491 1785456459.496176 read 0x00b375 thru 0x00cff9 (0x1c85 bytes) 000492 1785456459.503682 write 0x038ad5 thru 0x03942f (0x95b bytes) 000493 1785456459.534796 mapwrite 0x03c3f2 thru 0x03ffff (0x3c0e bytes) 000495 1785456459.619353 mapread 0x006de8 thru 0x0081c4 (0x13dd bytes) 000496 1785456459.636569 mapread 0x010070 thru 0x01ecc4 (0xec55 bytes) 000499 1785456459.666800 trunc from 0x040000 to 0x010e2f (0x2f1d0 bytes) 000500 1785456459.687900 trunc from 0x010e30 to 0x0175c2 (0x6793 bytes) 000501 1785456459.711282 write 0x03710f thru 0x03cf93 (0x5e85 bytes) 000502 1785456459.716097 mapwrite 0x018a94 thru 0x028a7a (0xffe7 bytes) 000504 1785456459.837408 write 0x0297cf thru 0x02d4fa (0x3d2c bytes) 000508 1785456459.868082 MIRROR_EXTEND to 2 mirrors 000512 1785456460.708557 trunc from 0x03cf94 to 0x013bec (0x293a7 bytes) 000514 1785456460.856742 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000516 1785456460.994378 mapwrite 0x03574d thru 0x03653f (0xdf3 bytes) 000518 1785456461.099711 mapwrite 0x0359a3 thru 0x03851a (0x2b78 bytes) 000520 1785456461.169030 mapread 0x014ad8 thru 0x017ac0 (0x2fe9 bytes) 000521 1785456461.186033 write 0x002609 thru 0x00d398 (0xad90 bytes) 000523 1785456461.193555 read 0x0049c3 thru 0x005736 (0xd74 bytes) 000525 1785456461.195155 mapread 0x033290 thru 0x03851a (0x528b bytes) 000527 1785456461.198549 mapread 0x026f01 thru 0x027533 (0x633 bytes) 000528 1785456461.211191 mapwrite 0x00464b thru 0x00a8db (0x6291 bytes) 000530 1785456461.260113 mapread 0x02fb69 thru 0x034de9 (0x5281 bytes) 000532 1785456461.268116 mapread 0x033486 thru 0x03851a (0x5095 bytes) 000535 1785456461.275032 read 0x016c70 thru 0x021efc (0xb28d bytes) 000536 1785456461.285733 write 0x03ff61 thru 0x03ffff (0x9f bytes) 000540 1785456461.290206 mapread 0x0058ef thru 0x009b5a (0x426c bytes) 000541 1785456461.297543 MIRROR_EXTEND to 2 mirrors 000543 1785456462.247168 mapwrite 0x01a1f3 thru 0x01c847 (0x2655 bytes) 000547 1785456462.462484 trunc from 0x040000 to 0x012b29 (0x2d4d6 bytes) 000550 1785456462.521557 read 0x000bac thru 0x00baf5 (0xaf4a bytes) 000551 1785456462.534008 mapwrite 0x0388d2 thru 0x03ffff (0x772e bytes) 000554 1785456462.616410 mapwrite 0x021c2a thru 0x026f90 (0x5367 bytes) 000556 1785456462.666716 mapread 0x03b5dc thru 0x03ffff (0x4a24 bytes) 000557 1785456462.668040 trunc from 0x040000 to 0x00c681 (0x3397e bytes) 000560 1785456462.700104 MIRROR_RESYNC 2 mirrors 000561 1785456463.553076 trunc from 0x00c682 to 0x013d82 (0x7701 bytes) 000562 1785456463.626763 read 0x0125a7 thru 0x013d82 (0x17dc bytes) 000564 1785456463.639985 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000565 1785456463.803158 trunc from 0x013d83 to 0x02e269 (0x1a4e7 bytes) 000568 1785456463.828726 mapwrite 0x03a2c8 thru 0x03ffff (0x5d38 bytes) 000570 1785456463.983463 mapwrite 0x03f4f8 thru 0x03ffff (0xb08 bytes) 000572 1785456464.058861 trunc from 0x040000 to 0x0348d2 (0xb72d bytes) 000574 1785456464.100208 read 0x03237b thru 0x0348d2 (0x2558 bytes) 000575 1785456464.114675 read 0x024cc2 thru 0x033c3d (0xef7c bytes) 000577 1785456464.146164 trunc from 0x0348d3 to 0x0246ef (0x101e3 bytes) 000578 1785456464.162705 mapread 0x001a4a thru 0x0033a5 (0x195c bytes) 000580 1785456464.171284 read 0x00a164 thru 0x00d352 (0x31ef bytes) 000582 1785456464.173611 mapread 0x00dbc6 thru 0x016952 (0x8d8d bytes) 000583 1785456464.175753 read 0x003578 thru 0x004cad (0x1736 bytes) 000586 1785456464.179387 mapread 0x01df87 thru 0x0246ef (0x6769 bytes) 000587 1785456464.181237 mapread 0x02258b thru 0x02371e (0x1194 bytes) 000588 1785456464.185489 trunc from 0x0246f0 to 0x020c0b (0x3ae4 bytes) 000595 1785456464.244147 write 0x000164 thru 0x005434 (0x52d1 bytes) 000596 1785456464.258361 read 0x01d74e thru 0x0200dc (0x298f bytes) 000598 1785456464.269571 MIRROR_EXTEND to 2 mirrors 000601 1785456465.247201 mapwrite 0x02f6cb thru 0x03cd7b (0xd6b1 bytes) 000603 1785456465.497325 mapread 0x03c6aa thru 0x03cd7b (0x6d2 bytes) 000604 1785456465.505146 read 0x026f49 thru 0x032c7f (0xbd37 bytes) 000606 1785456465.526573 mapwrite 0x0372f4 thru 0x03eb4f (0x785c bytes) 000607 1785456465.615949 write 0x031c07 thru 0x03ffff (0xe3f9 bytes) 000609 1785456465.623664 trunc from 0x040000 to 0x012792 (0x2d86d bytes) 000610 1785456465.651563 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x12793): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12793): Mirror 2: 0xbee2414f (stale) CRC-32 checksum value for chunk [0, 0x12793): Mirror 1: 0xbee2414f Mirror 3: 0xbee2414f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12793: skipped truncating to largest ever: 0x3feed 000615 1785456466.946827 trunc from 0x012793 to 0x03feec (0x2d75a bytes) 000616 1785456467.043664 read 0x028f28 thru 0x02a0a0 (0x1179 bytes) 000618 1785456467.053096 trunc from 0x03feed to 0x032c70 (0xd27c bytes) 000619 1785456467.087413 trunc from 0x032c71 to 0x00ebba (0x240b6 bytes) 000620 1785456467.107514 read 0x00950b thru 0x00ebba (0x56b0 bytes) 000621 1785456467.120465 mapwrite 0x01f9f5 thru 0x02c3d9 (0xc9e5 bytes) 000622 1785456467.266224 read 0x025dae thru 0x02c3d9 (0x662c bytes) 000625 1785456467.268592 mapwrite 0x009f4e thru 0x016148 (0xc1fb bytes) 000627 1785456467.354527 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c3da): Mirror 3: 0x25099f09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3da): Mirror 3: 0x25099f09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3da: skipped 000628 1785456467.709197 MIRROR_RESYNC 2 mirrors 000629 1785456468.558170 mapread 0x00d958 thru 0x0130dc (0x5785 bytes) 000631 1785456468.580143 write 0x00617c thru 0x00b558 (0x53dd bytes) 000632 1785456468.640959 trunc from 0x02c3da to 0x000a9e (0x2b93b bytes) 000634 1785456468.674569 mapread 0x0005a7 thru 0x000a9e (0x4f8 bytes) 000637 1785456468.688016 mapread 0x0006a4 thru 0x000a9e (0x3fb bytes) 000638 1785456468.693837 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000639 1785456468.895394 mapwrite 0x013659 thru 0x01beed (0x8895 bytes) 000640 1785456468.984978 write 0x024a43 thru 0x034278 (0xf836 bytes) 000642 1785456468.992993 mapwrite 0x01132b thru 0x01ebd9 (0xd8af bytes) 000644 1785456469.119685 MIRROR_EXTEND to 2 mirrors 000648 1785456470.020858 write 0x03b7fe thru 0x03ffff (0x4802 bytes) 000649 1785456470.099414 mapwrite 0x00c01b thru 0x00e6c2 (0x26a8 bytes) 000650 1785456470.215532 trunc from 0x040000 to 0x013f98 (0x2c067 bytes) 000653 1785456470.250066 trunc from 0x013f99 to 0x020f26 (0xcf8e bytes) 000654 1785456470.267174 read 0x016d78 thru 0x020f26 (0xa1af bytes) 000658 1785456470.285493 mapread 0x01a5e0 thru 0x020f26 (0x6947 bytes) 000659 1785456470.289300 trunc from 0x020f27 to 0x02d05b (0xc135 bytes) 000660 1785456470.311480 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000662 1785456470.467153 write 0x025eb3 thru 0x028cc1 (0x2e0f bytes) 000663 1785456470.492347 MIRROR_EXTEND to 2 mirrors 000664 1785456471.114251 read 0x003e4a thru 0x004d06 (0xebd bytes) 000665 1785456471.126133 mapread 0x00a4ca thru 0x011d9c (0x78d3 bytes) 000666 1785456471.154252 write 0x005d92 thru 0x00749d (0x170c bytes) 000667 1785456471.224793 read 0x00fce2 thru 0x013a90 (0x3daf bytes) 000668 1785456471.241135 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000669 1785456471.392716 trunc from 0x02d05c to 0x02776f (0x58ec bytes) 000670 1785456471.431378 write 0x01a073 thru 0x025f1d (0xbeab bytes) 000671 1785456471.444552 read 0x00dbe3 thru 0x01410d (0x652b bytes) 000672 1785456471.458360 write 0x03f514 thru 0x03ffff (0xaec bytes) 000673 1785456471.459552 read 0x00c8a6 thru 0x012f42 (0x669d bytes) 000676 1785456471.476137 write 0x018668 thru 0x027402 (0xed9b bytes) 000678 1785456471.481185 mapread 0x03a711 thru 0x03ffff (0x58ef bytes) 000681 1785456471.488463 trunc from 0x040000 to 0x038d81 (0x727e bytes) 000683 1785456471.514987 mapwrite 0x039d77 thru 0x03ffff (0x6289 bytes) 000685 1785456471.646317 mapread 0x02afb2 thru 0x02d86e (0x28bd bytes) 000689 1785456471.653854 mapread 0x03df26 thru 0x03ffff (0x20da bytes) 000690 1785456471.655265 trunc from 0x040000 to 0x03734e (0x8cb1 bytes) 000691 1785456471.680231 write 0x023417 thru 0x026847 (0x3431 bytes) 000692 1785456471.681354 write 0x00268a thru 0x0039f0 (0x1367 bytes) 000694 1785456471.682744 write 0x03396c thru 0x03d4a1 (0x9b36 bytes) 000695 1785456471.684109 mapwrite 0x02a61c thru 0x02c69f (0x2084 bytes) 000696 1785456471.731146 trunc from 0x03d4a2 to 0x02b674 (0x11e2d bytes) 000697 1785456471.750739 read 0x0036e8 thru 0x0132f3 (0xfc0c bytes) 000699 1785456471.754566 mapwrite 0x0053f6 thru 0x011152 (0xbd5d bytes) 000700 1785456471.822425 write 0x03383f thru 0x03ef75 (0xb737 bytes) 000702 1785456471.829461 mapread 0x03297c thru 0x0375b9 (0x4c3e bytes) 000704 1785456471.842958 write 0x015187 thru 0x023d0c (0xeb86 bytes) 000709 1785456471.850381 read 0x0073a6 thru 0x0130ba (0xbd15 bytes) 000710 1785456471.851177 trunc from 0x03ef76 to 0x00e003 (0x30f72 bytes) 000711 1785456471.879224 read 0x0007dc thru 0x00a5f1 (0x9e16 bytes) 000712 1785456471.881088 mapwrite 0x03eb6b thru 0x03ffff (0x1495 bytes) 000713 1785456471.991119 read 0x02bd45 thru 0x035b3d (0x9df9 bytes) 000716 1785456471.996144 mapread 0x037fa6 thru 0x03c835 (0x4890 bytes) 000719 1785456472.006452 write 0x009ee6 thru 0x0186e1 (0xe7fc bytes) 000720 1785456472.008925 mapread 0x000046 thru 0x00434a (0x4305 bytes) 000722 1785456472.011191 trunc from 0x040000 to 0x01979e (0x26861 bytes) 000724 1785456472.035896 write 0x03811a thru 0x03ab5a (0x2a41 bytes) 000725 1785456472.037112 mapwrite 0x03027c thru 0x03e74e (0xe4d3 bytes) 000727 1785456472.148803 trunc from 0x03e74f to 0x005a64 (0x38cea bytes) 000728 1785456472.166059 trunc from 0x005a65 to 0x0390e3 (0x3367f bytes) 000731 1785456472.189302 mapread 0x034ac7 thru 0x0390e3 (0x461d bytes) 000733 1785456472.226002 mapread 0x02e4f4 thru 0x02e867 (0x374 bytes) 000734 1785456472.234504 read 0x013824 thru 0x022f10 (0xf6ed bytes) 000736 1785456472.256320 MIRROR_EXTEND to 2 mirrors 000737 1785456473.090530 write 0x01b21c thru 0x01f9e8 (0x47cd bytes) 000738 1785456473.236150 read 0x019c49 thru 0x020417 (0x67cf bytes) 000739 1785456473.248934 write 0x0307be thru 0x03ffff (0xf842 bytes) 000741 1785456473.256295 mapread 0x031f3a thru 0x03d504 (0xb5cb bytes) 000743 1785456473.262962 read 0x000f77 thru 0x00ce11 (0xbe9b bytes) 000745 1785456473.275158 mapwrite 0x012c6f thru 0x014432 (0x17c4 bytes) 000748 1785456473.346949 MIRROR_RESYNC 2 mirrors 000749 1785456474.206915 read 0x01de04 thru 0x028f63 (0xb160 bytes) 000750 1785456474.218942 read 0x01ddf8 thru 0x01fd03 (0x1f0c bytes) 000752 1785456474.219882 read 0x02a90e thru 0x02cbde (0x22d1 bytes) 000754 1785456474.229852 read 0x0139ef thru 0x0157b3 (0x1dc5 bytes) 000755 1785456474.230499 read 0x034fd1 thru 0x03694d (0x197d bytes) 000757 1785456474.234723 MIRROR_EXTEND to 3 mirrors 000759 1785456475.067040 read 0x00d655 thru 0x0193a4 (0xbd50 bytes) 000764 1785456475.077389 mapwrite 0x012171 thru 0x0188cb (0x675b bytes) 000768 1785456475.206621 write 0x029923 thru 0x031a68 (0x8146 bytes) 000771 1785456475.226453 trunc from 0x040000 to 0x025047 (0x1afb8 bytes) 000772 1785456475.294894 mapread 0x0206c8 thru 0x025047 (0x4980 bytes) 000773 1785456475.310221 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25048): Mirror 3: 0xc2787d61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25048): Mirror 3: 0xc2787d61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25048: skipped 000774 1785456475.611151 trunc from 0x025048 to 0x030bee (0xbba7 bytes) 000775 1785456475.630269 read 0x029a58 thru 0x030bee (0x7197 bytes) 000776 1785456475.638146 mapwrite 0x01d9c2 thru 0x028347 (0xa986 bytes) 000778 1785456475.723070 MIRROR_RESYNC 2 mirrors 000779 1785456476.474451 mapread 0x0171fc thru 0x01c5f7 (0x53fc bytes) 000782 1785456476.504620 mapread 0x004594 thru 0x00cebb (0x8928 bytes) 000785 1785456476.522154 trunc from 0x030bef to 0x0155f9 (0x1b5f5 bytes) 000787 1785456476.631772 trunc from 0x0155fa to 0x0133f6 (0x2203 bytes) 000789 1785456476.660351 mapread 0x004844 thru 0x00cb05 (0x82c2 bytes) 000791 1785456476.683732 mapread 0x012cb4 thru 0x0133f6 (0x743 bytes) 000792 1785456476.697422 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000793 1785456476.870312 MIRROR_EXTEND to 2 mirrors 000794 1785456477.615850 write 0x0204b4 thru 0x029c00 (0x974d bytes) 000795 1785456477.676236 mapwrite 0x00d10c thru 0x012174 (0x5069 bytes) 000797 1785456477.755517 mapread 0x019d3d thru 0x027d7d (0xe041 bytes) 000798 1785456477.770382 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000800 1785456477.935420 mapread 0x024015 thru 0x029c00 (0x5bec bytes) 000802 1785456477.970744 mapread 0x00c3be thru 0x01b0b6 (0xecf9 bytes) 000803 1785456478.010159 mapread 0x0036bc thru 0x012bd9 (0xf51e bytes) 000804 1785456478.019772 mapwrite 0x002f4a thru 0x007e8c (0x4f43 bytes) 000806 1785456478.102980 write 0x00f4e2 thru 0x01991f (0xa43e bytes) 000808 1785456478.112968 trunc from 0x029c01 to 0x01d092 (0xcb6e bytes) 000809 1785456478.149154 write 0x024677 thru 0x026e45 (0x27cf bytes) 000810 1785456478.154443 trunc from 0x026e46 to 0x026cac (0x199 bytes) 000811 1785456478.178412 mapwrite 0x027d98 thru 0x032b06 (0xad6f bytes) 000812 1785456478.264314 mapwrite 0x033b10 thru 0x034b7c (0x106d bytes) 000814 1785456478.340100 write 0x01ec0c thru 0x02768c (0x8a81 bytes) 000818 1785456478.356867 mapread 0x01709e thru 0x022603 (0xb566 bytes) 000819 1785456478.358111 write 0x011f61 thru 0x01c1da (0xa27a bytes) 000822 1785456478.366028 mapread 0x025009 thru 0x029204 (0x41fc bytes) 000824 1785456478.369160 mapwrite 0x018c55 thru 0x019243 (0x5ef bytes) 000825 1785456478.430109 mapwrite 0x02a144 thru 0x02dbe1 (0x3a9e bytes) 000826 1785456478.466666 mapread 0x02bc3f thru 0x02c314 (0x6d6 bytes) 000827 1785456478.468256 mapwrite 0x02e636 thru 0x03c1da (0xdba5 bytes) 000834 1785456478.560511 mapread 0x00f764 thru 0x01b675 (0xbf12 bytes) 000837 1785456478.565161 mapwrite 0x020dd6 thru 0x02ef2e (0xe159 bytes) 000838 1785456478.612778 write 0x02ce12 thru 0x03548a (0x8679 bytes) 000840 1785456478.616203 trunc from 0x03c1db to 0x0329ed (0x97ed bytes) 000841 1785456478.637509 mapwrite 0x01dee7 thru 0x0208eb (0x2a05 bytes) 000844 1785456478.694952 trunc from 0x0329ee to 0x03195c (0x1091 bytes) 000845 1785456478.720294 mapwrite 0x0200db thru 0x028d95 (0x8cbb bytes) 000846 1785456478.778327 read 0x01c1f3 thru 0x022c63 (0x6a71 bytes) 000849 1785456478.781306 read 0x02553b thru 0x028acf (0x3595 bytes) 000850 1785456478.788595 MIRROR_EXTEND to 2 mirrors 000852 1785456479.595111 MIRROR_RESYNC 2 mirrors 000853 1785456479.887511 mapwrite 0x00e732 thru 0x01ac17 (0xc4e6 bytes) 000855 1785456480.099154 trunc from 0x03195d to 0x02fc3f (0x1d1d bytes) 000856 1785456480.120708 trunc from 0x02fc40 to 0x02aaa5 (0x519a bytes) 000858 1785456480.135733 mapread 0x00a1c1 thru 0x01761f (0xd45f bytes) 000859 1785456480.137209 trunc from 0x02aaa6 to 0x028754 (0x2351 bytes) 000861 1785456480.156685 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28755): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28755): Mirror 2: 0x50eba571 (stale) CRC-32 checksum value for chunk [0, 0x28755): Mirror 1: 0x6ac1a322 Mirror 3: 0x6ac1a322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28755: skipped 000863 1785456481.071928 read 0x01955a thru 0x01e4ba (0x4f61 bytes) 000864 1785456481.079968 trunc from 0x028755 to 0x01ba06 (0xcd4e bytes) 000865 1785456481.134755 write 0x011bf1 thru 0x013cb9 (0x20c9 bytes) 000866 1785456481.150981 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: 0x823a9a36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: 0x823a9a36 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba07: skipped 000867 1785456481.482810 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 3: 0x823a9a36 (stale) CRC-32 checksum value for chunk [0, 0x1ba07): Mirror 1: 0xbecc7b5e Mirror 4: 0xbecc7b5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba07: skipped 000869 1785456482.437563 mapread 0x007ad5 thru 0x0165bc (0xeae8 bytes) 000870 1785456482.493510 write 0x0099f9 thru 0x00ef44 (0x554c bytes) 000871 1785456482.620594 trunc from 0x01ba07 to 0x001d56 (0x19cb0 bytes) 000873 1785456482.658987 read 0x0013df thru 0x001d56 (0x978 bytes) 000876 1785456482.675398 MIRROR_RESYNC 3 mirrors 000877 1785456483.396746 read 0x0018f3 thru 0x001d56 (0x464 bytes) 000878 1785456483.405834 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000879 1785456483.708846 mapwrite 0x019eac thru 0x025340 (0xb495 bytes) 000880 1785456483.908280 MIRROR_RESYNC 2 mirrors 000884 1785456484.517445 mapread 0x024511 thru 0x025340 (0xe30 bytes) 000887 1785456484.529411 write 0x003a32 thru 0x010970 (0xcf3f bytes) 000888 1785456484.612060 read 0x017be4 thru 0x019df3 (0x2210 bytes) 000889 1785456484.619224 mapread 0x005be3 thru 0x00ac16 (0x5034 bytes) 000890 1785456484.621190 mapwrite 0x00a411 thru 0x019b92 (0xf782 bytes) 000891 1785456484.696317 trunc from 0x025341 to 0x02ecf7 (0x99b7 bytes) 000892 1785456484.715005 trunc from 0x02ecf8 to 0x023702 (0xb5f5 bytes) 000893 1785456484.736561 mapwrite 0x019bc9 thru 0x0223a0 (0x87d8 bytes) 000895 1785456484.786578 mapread 0x017964 thru 0x01b6ae (0x3d4b bytes) 000896 1785456484.788954 trunc from 0x023703 to 0x01db05 (0x5bfd bytes) 000897 1785456484.803373 trunc from 0x01db06 to 0x0127a6 (0xb35f bytes) 000898 1785456484.827175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000899 1785456484.965615 mapread 0x00ee6f thru 0x0127a6 (0x3938 bytes) 000900 1785456484.989955 write 0x037e85 thru 0x038865 (0x9e1 bytes) 000901 1785456484.991836 trunc from 0x038866 to 0x0163a0 (0x224c5 bytes) 000902 1785456485.010290 write 0x03135b thru 0x038b88 (0x782e bytes) 000904 1785456485.017595 MIRROR_EXTEND to 2 mirrors 000905 1785456485.827447 trunc from 0x038b89 to 0x01cebf (0x1bcc9 bytes) 000906 1785456485.941182 mapread 0x013374 thru 0x019d8c (0x6a19 bytes) 000907 1785456485.981167 write 0x03b112 thru 0x03ffff (0x4eee bytes) 000910 1785456485.987019 read 0x026ce8 thru 0x035810 (0xeb29 bytes) 000911 1785456485.999092 write 0x004bc0 thru 0x00d4e9 (0x892a bytes) 000913 1785456486.010941 mapread 0x00bb53 thru 0x013fb3 (0x8461 bytes) 000915 1785456486.014444 write 0x0216ca thru 0x02d74e (0xc085 bytes) 000916 1785456486.019985 mapread 0x00cc35 thru 0x0190c2 (0xc48e bytes) 000917 1785456486.021287 trunc from 0x040000 to 0x02329b (0x1cd64 bytes) 000918 1785456486.049172 MIRROR_RESYNC 2 mirrors 000919 1785456486.860337 MIRROR_EXTEND to 3 mirrors 000922 1785456487.577322 read 0x006e8f thru 0x012e90 (0xc002 bytes) 000923 1785456487.589261 mapread 0x00f256 thru 0x019e08 (0xabb3 bytes) 000924 1785456487.611477 read 0x018d44 thru 0x01df07 (0x51c4 bytes) 000927 1785456487.612663 mapwrite 0x03a76e thru 0x03ffff (0x5892 bytes) 000928 1785456487.813719 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x669d231a (stale) Mirror 3: 0x669d231a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5cfecf1 Mirror 4: 0xf5cfecf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000929 1785456488.838231 read 0x014069 thru 0x01e16a (0xa102 bytes) 000930 1785456488.865950 read 0x0382df thru 0x039acd (0x17ef bytes) 000931 1785456488.899756 MIRROR_RESYNC 4 mirrors 000933 1785456489.813944 mapread 0x03378e thru 0x03ffff (0xc872 bytes) 000936 1785456489.840195 read 0x0179b0 thru 0x01f203 (0x7854 bytes) 000937 1785456489.848076 mapread 0x02c4c2 thru 0x03c084 (0xfbc3 bytes) 000938 1785456489.858773 mapread 0x011ddb thru 0x01ab49 (0x8d6f bytes) 000940 1785456489.878970 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) 000942 1785456490.222724 trunc from 0x040000 to 0x002037 (0x3dfc8 bytes) 000946 1785456490.347658 write 0x0208cc thru 0x029dd0 (0x9505 bytes) 000949 1785456490.350622 mapread 0x015705 thru 0x0190ac (0x39a8 bytes) 000950 1785456490.383914 MIRROR_RESYNC 3 mirrors 000951 1785456491.247718 trunc from 0x029dd1 to 0x010d09 (0x190c7 bytes) 000952 1785456491.352951 MIRROR_RESYNC 3 mirrors 000953 1785456492.150594 trunc from 0x010d0a to 0x026310 (0x15607 bytes) 000954 1785456492.242668 trunc from 0x026311 to 0x03965a (0x1334a bytes) 000955 1785456492.290798 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: 0xa3056f67 (stale) Mirror 4: 0xa3056f67 (stale) CRC-32 checksum value for chunk [0, 0x3965b): Mirror 1: 0xa3056f67 Mirror 5: 0xa3056f67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3965b: skipped 000962 1785456493.614054 write 0x026fc5 thru 0x031df4 (0xae30 bytes) 000964 1785456493.687623 mapwrite 0x013567 thru 0x01fb40 (0xc5da bytes) 000965 1785456493.796605 MIRROR_RESYNC 4 mirrors 000967 1785456494.917128 mapwrite 0x02c1ca thru 0x038630 (0xc467 bytes) 000969 1785456495.200122 mapread 0x013ba1 thru 0x020818 (0xcc78 bytes) 000970 1785456495.228035 mapread 0x02d952 thru 0x03815c (0xa80b bytes) 000971 1785456495.236192 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: 0x57490682 (stale) Mirror 4: 0x57490682 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3965b): Mirror 2: 0x57490682 (stale) Mirror 4: 0x57490682 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3965b: skipped 000973 1785456495.621344 mapread 0x00c328 thru 0x00dca9 (0x1982 bytes) 000975 1785456495.637368 trunc from 0x03965b to 0x00d5d0 (0x2c08a bytes) 000976 1785456495.663176 write 0x0261c1 thru 0x02d60c (0x744c bytes) 000980 1785456495.679675 MIRROR_RESYNC 3 mirrors 000981 1785456496.835990 write 0x007e9e thru 0x0100da (0x823d bytes) 000982 1785456496.954484 mapread 0x02ac49 thru 0x02d60c (0x29c4 bytes) 000986 1785456496.999793 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: 0x93c51409 (stale) Mirror 4: 0x93c51409 (stale) CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 1: 0xae76d7f Mirror 5: 0xae76d7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d60d: skipped 000988 1785456498.000987 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: 0x93c51409 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 2: 0x93c51409 (stale) CRC-32 checksum value for chunk [0, 0x2d60d): Mirror 1: 0xae76d7f Mirror 5: 0xae76d7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d60d: skipped 000989 1785456498.591723 trunc from 0x02d60d to 0x03ce8a (0xf87e bytes) 000990 1785456498.687397 read 0x007a9c thru 0x00b415 (0x397a bytes) 000991 1785456498.696292 write 0x015efb thru 0x01dda2 (0x7ea8 bytes) 000992 1785456498.704279 read 0x011a32 thru 0x0147b0 (0x2d7f bytes) 000993 1785456498.714545 read 0x017305 thru 0x01afb8 (0x3cb4 bytes) 000994 1785456498.715444 mapwrite 0x0383c9 thru 0x03ffff (0x7c37 bytes) 001000 1785456498.821984 read 0x028510 thru 0x029690 (0x1181 bytes) All operations completed A-OK! PASS 70a (84s) == sanity-flr test 71: check mirror extend parent fid ==== 20:08:23 (1785456503) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 20:08:26 (1785456506) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22816 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153263 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 20:08:33 (1785456513) Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre2 Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8603072 .. Extending file size to 6781440 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8017568 .. Extending file size to 8839616 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8050656 .. Extending file size to 8162464 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8284000 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8708576 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57281 57282 57283 57285 57289 failed 192.168.206.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg613-client.virtnet /mnt/lustre2 (opts:) 192.168.206.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg613-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:09:54 (1785456594) Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre2 Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3737152 .. Extending file size to 3734080 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 5936928 .. Extending file size to 7032064 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7201760 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7625440 .. Extending file size to 7657888 .. Extending file size to 8481824 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8068320 .. Extending file size to 9017056 .. Extending file size to 9184256 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8840032 .. Extending file size to 9020480 .. Waiting 58715 58716 58717 58720 58722 done 192.168.206.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg613-client.virtnet /mnt/lustre2 (opts:) 192.168.206.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg613-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:11:13 (1785456673) Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109465 s, 19.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.495159 s, 4.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe810fa12 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg613-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:11:32 (1785456692) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0737906 s, 28.4 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 20:11:38 (1785456698) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1db:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:11:45 (1785456705) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138106 s, 15.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:11:54 (1785456714) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.341779 s, 6.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:12:02 (1785456722) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.264043 s, 7.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:12:35 (1785456755) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg613-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14622 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:13:30 (1785456810) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg613-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.444765 s, 4.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:14:30 (1785456870) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg613-server mds-ost sync done. Creating new pool test_204f oleg613-server: Pool lustre.test_204f created Adding targets to pool oleg613-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg613-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.315445 s, 6.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg613-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg613-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg613-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:15:47 (1785456947) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:15:55 (1785456955) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:16:04 (1785456964) oleg613-server: Pool lustre.test_206 created Adding targets to pool oleg613-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg613-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg613-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg613-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg613-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:16:30 (1785456990) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767725 s, 13.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540593 s, 19.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6ca9ab48 Mirror 3: 0x6ca9ab48 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:16:41 (1785457001) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:16:45 (1785457005) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:16:48 (1785457008) Starting client: oleg613-client.virtnet: -o user_xattr,flock 192.168.206.113@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.17803 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x25a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x780000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x780000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x780000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1660000 Read this (4 bytes): BBBB 192.168.206.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg613-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:17:25 (1785457045) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.029226 s, 280 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:17:33 (1785457053) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053103 s, 19.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834432 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-30 20:17:35.000000000 -0400 Modify: 2026-07-30 20:17:35.000000000 -0400 Change: 2026-07-30 20:17:36.000000000 -0400 Birth: 2026-07-30 20:17:34.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:17:41 (1785457061) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:17:45 (1785457065) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.702671 s, 58.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.047012 s, 87.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 30 20:17 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 30 20:17 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:17:53 (1785457073) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:18:01 (1785457081) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1731 sec =========== 20:18:11 (1785457091) === sanity-flr: start cleanup 20:18:13 (1785457093) === === sanity-flr: finish cleanup 20:18:19 (1785457099) ===