-----============= acceptance-small: sanity-flr ============----- Wed Dec 3 01:51:56 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg231-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:52:03 (1764744723) === oleg231-client.virtnet: executing check_config_client /mnt/lustre oleg231-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg231-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa07ad02de000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa07ad02de000.idle_timeout=debug disable quota as required oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:52:16 (1764744736) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7549] == sanity-flr test 0a: lfs mirror create with -N option == 01:52:17 (1764744737) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:52:24 (1764744744) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:52:25 (1764744745) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:52:27 (1764744747) PASS 0d (10s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:52:37 (1764744757) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:52:38 (1764744758) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 01:52:40 (1764744760) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa07ad02de000.stats=clear osc.lustre-OST0001-osc-ffffa07ad02de000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:52:43 (1764744763) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:52:47 (1764744767) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:52:51 (1764744771) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:52:52 (1764744772) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 01:52:55 (1764744775) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:52:58 (1764744778) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:52:59 (1764744779) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:53:02 (1764744782) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:53:06 (1764744786) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:53:08 (1764744788) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.756243 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572661 s, 18.3 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:53:13 (1764744793) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0578453 s, 90.6 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:53:16 (1764744796) Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg231-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (30s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:53:46 (1764744826) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0965212 s, 43.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (16s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:54:02 (1764744842) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg231-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (27s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:54:29 (1764744869) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5386 s, 34.1 MB/s -rw-r--r-- 1 root root 52428800 Dec 3 01:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.211s user 0m0.001s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa07ad02de000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.256s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg231-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg231-server oleg231-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.246s user 0m0.002s sys 0m0.180s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg231-server: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg231-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa07ad02de000.ost_server_uuid in FULL state after 0 sec PASS 33b (30s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:54:59 (1764744899) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:55:01 (1764744901) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:55:02 (1764744902) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 01:55:04 (1764744904) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 01:55:06 (1764744906) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.150983 s, 83.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:55:10 (1764744910) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104363 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135817 s, 77.2 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:55:12 (1764744912) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:55:16 (1764744916) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000794189 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00078113 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000707217 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000714736 s, 1.4 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 01:55:19 (1764744919) create test files with size 15729440 .. df56bd5466f31a010cbc388eee62268b - 95295dd10b3b1039e2e09f2c61fb145f - fa62a0ebacb57ca7420b6d97feff7c35 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 01:55:26 (1764744926) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:55:31 (1764744931) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:55:33 (1764744933) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417855 s, 25.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120759 s, 86.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 01:55:36 (1764744936) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0316228 s, 66.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.100095 s, 41.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0474299 s, 44.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 01:55:41 (1764744941) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 01:55:42 (1764744942) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:55:43 (1764744943) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012774 s, 82.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 01:55:45 (1764744945) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:55:46 (1764744946) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:55:49 (1764744949) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.098598 s, 106 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:56:09 (1764744969) llite.lustre-ffffa07ac8576800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0939996 s, 112 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg231-client.virtnet /mnt/lustre (opts:) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffffa07ad02dd000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:56:13 (1764744973) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:56:25 (1764744985) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:56:36 (1764744996) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 01:56:37 (1764744997) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0664002 s, 47.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0511008 s, 61.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x302cbed4 (stale) Mirror 2: 0x302cbed4 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 0f2ae1c7d29ec97ff39b584a4469a77e - mirror 1 checksum: 5fd7ac5ab40983ca69c71ad77b35236f - mirror 2 checksum: 0f2ae1c7d29ec97ff39b584a4469a77e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Dec 3 01:56:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Dec 3 01:56:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0f2ae1c7d29ec97ff39b584a4469a77e - mirror 1 checksum: 5fd7ac5ab40983ca69c71ad77b35236f - mirror 2 checksum: 5fd7ac5ab40983ca69c71ad77b35236f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Dec 3 01:56:37 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Dec 3 01:56:37 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:56:40 (1764745000) Failing mds1 on oleg231-server Stopping /mnt/lustre-mds1 (opts:) on oleg231-server 01:56:42 (1764745002) shut down facet: mds1 facet_host: oleg231-server facet_failover_host: oleg231-server Failover mds1 to oleg231-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 01:56:56 (1764745016) targets are mounted 01:56:56 (1764745016) facet_failover done oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00993128 s, 106 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:57:02 (1764745022) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:57:03 (1764745023) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:57:04 (1764745024) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0025569 s, 6.4 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:57:06 (1764745026) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:57:07 (1764745027) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:57:09 (1764745029) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:57:15 (1764745035) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:57:34 (1764745054) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:57:52 (1764745072) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:58:09 (1764745089) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 01:58:11 (1764745091) mirror create pid 52196 mirror split pid 52198 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52196 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52198 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 01:59:14 (1764745154) Seed set to 5154 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000006 1764745154.694277 MIRROR_EXTEND to 2 mirrors skipping zero size read truncating to largest ever: 0xa79a 000008 1764745154.809523 trunc from 00000000 to 0x00a799 (0xa79a bytes) 000009 1764745154.838760 mapwrite 0x01fe35 thru 0x0293b7 (0x9583 bytes) 000010 1764745154.880730 write 0x00346b thru 0x00fe8f (0xca25 bytes) 000011 1764745154.884845 mapwrite 0x034b3b thru 0x035c97 (0x115d bytes) 000012 1764745154.905446 read 0x02298c thru 0x023696 (0xd0b bytes) 000014 1764745154.906237 read 0x02d7f7 thru 0x035c97 (0x84a1 bytes) truncating to largest ever: 0xe10c 000016 1764745154.907202 trunc from 0x035c98 to 0x00e10b (0x27b8c bytes) 000019 1764745154.912987 mapread 0x00b6d7 thru 0x00e10b (0x2a35 bytes) 000020 1764745154.915567 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe10c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe10c): Mirror 2: 0x6ea97e29 (stale) CRC-32 checksum value for chunk [0, 0xe10c): Mirror 1: 0x361ade Mirror 3: 0x361ade lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe10c: skipped 000022 1764745155.160705 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x245c9 000023 1764745155.348501 trunc from 0x00e10c to 0x0245c8 (0x164bd bytes) 000025 1764745155.371547 mapread 0x023910 thru 0x0245c8 (0xcb9 bytes) truncating to largest ever: 0x3374f 000026 1764745155.376345 trunc from 0x0245c9 to 0x03374e (0xf186 bytes) 000028 1764745155.383022 write 0x03c86c thru 0x03ffff (0x3794 bytes) 000029 1764745155.384008 mapread 0x0324aa thru 0x036a73 (0x45ca bytes) 000030 1764745155.393982 mapread 0x00d5fb thru 0x01ca01 (0xf407 bytes) 000031 1764745155.410948 mapwrite 0x024129 thru 0x0292a8 (0x5180 bytes) 000034 1764745155.434999 trunc from 0x040000 to 0x01d533 (0x22acc bytes) 000035 1764745155.442837 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d534): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d534): Mirror 2: 0xafb86770 (stale) Mirror 3: 0xafb86770 (stale) CRC-32 checksum value for chunk [0, 0x1d534): Mirror 1: 0xafb86770 Mirror 4: 0xafb86770 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d534: skipped 000036 1764745155.681691 write 0x017247 thru 0x0216e4 (0xa49e bytes) 000037 1764745155.703401 MIRROR_RESYNC 4 mirrors 000041 1764745155.930056 MIRROR_EXTEND to 5 mirrors 000044 1764745156.131408 read 0x01da84 thru 0x0216e4 (0x3c61 bytes) 000045 1764745156.134538 mapwrite 0x00bd3c thru 0x015e0a (0xa0cf bytes) 000046 1764745156.202067 read 0x016b83 thru 0x0216e4 (0xab62 bytes) 000051 1764745156.207604 write 0x00c03b thru 0x011b0c (0x5ad2 bytes) 000052 1764745156.210500 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x216e5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x216e5): Mirror 2: 0xa05918e6 (stale) Mirror 3: 0xa05918e6 (stale) Mirror 4: 0xa05918e6 (stale) Mirror 5: 0xa05918e6 (stale) CRC-32 checksum value for chunk [0, 0x216e5): Mirror 1: 0x285051df Mirror 6: 0x285051df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x216e5: skipped 000053 1764745156.448421 write 0x02cf63 thru 0x0394ec (0xc58a bytes) 000054 1764745156.468222 mapread 0x001667 thru 0x002ab1 (0x144b bytes) 000056 1764745156.472681 trunc from 0x0394ed to 0x013dfb (0x256f1 bytes) 000057 1764745156.487190 MIRROR_RESYNC 6 mirrors 000058 1764745156.739762 mapread 0x00caec thru 0x013dfb (0x7310 bytes) 000059 1764745156.744994 write 0x03566a thru 0x03ffff (0xa996 bytes) 000060 1764745156.763051 read 0x013a0a thru 0x022a1c (0xf013 bytes) 000062 1764745156.766568 read 0x0251c1 thru 0x02816d (0x2fad bytes) 000065 1764745156.771441 trunc from 0x040000 to 0x022534 (0x1dacb bytes) 000066 1764745156.782021 mapwrite 0x029771 thru 0x03447b (0xad0b bytes) 000067 1764745156.816690 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3447c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3447c): Mirror 2: 0x9af27ccd (stale) Mirror 3: 0x9af27ccd (stale) Mirror 4: 0x9af27ccd (stale) Mirror 5: 0x9af27ccd (stale) Mirror 6: 0x9af27ccd (stale) CRC-32 checksum value for chunk [0, 0x3447c): Mirror 1: 0xa5dd766a Mirror 7: 0xa5dd766a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3447c: skipped 000069 1764745157.043810 mapwrite 0x01ccdd thru 0x02b72c (0xea50 bytes) 000070 1764745157.110059 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3447c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3447c): Mirror 2: 0x9af27ccd (stale) Mirror 3: 0x9af27ccd (stale) Mirror 4: 0x9af27ccd (stale) Mirror 5: 0x9af27ccd (stale) Mirror 6: 0x9af27ccd (stale) Mirror 7: 0xa5dd766a (stale) CRC-32 checksum value for chunk [0, 0x3447c): Mirror 1: 0xe74b451f Mirror 8: 0xe74b451f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3447c: skipped 000073 1764745157.361211 mapread 0x029add thru 0x02f99c (0x5ec0 bytes) 000074 1764745157.370533 read 0x020ac3 thru 0x022ce3 (0x2221 bytes) 000075 1764745157.374520 trunc from 0x03447c to 0x02e30f (0x616c bytes) 000077 1764745157.397094 read 0x0299ff thru 0x02e30f (0x4911 bytes) 000078 1764745157.401034 write 0x026b23 thru 0x0286d9 (0x1bb7 bytes) 000080 1764745157.402358 mapwrite 0x0182d9 thru 0x025818 (0xd540 bytes) truncating to largest ever: 0x377c1 000083 1764745157.421944 trunc from 0x02e310 to 0x0377c0 (0x94b1 bytes) 000084 1764745157.429696 write 0x0194f9 thru 0x021399 (0x7ea1 bytes) 000087 1764745157.431451 trunc from 0x0377c1 to 0x01044e (0x27372 bytes) 000088 1764745157.437607 mapread 0x000d17 thru 0x003a5d (0x2d47 bytes) truncating to largest ever: 0x3f004 000089 1764745157.438446 trunc from 0x01044f to 0x03f003 (0x2ebb5 bytes) 000090 1764745157.443625 trunc from 0x03f004 to 0x015ae0 (0x29523 bytes) 000092 1764745157.449216 trunc from 0x015ae1 to 0x02bd64 (0x16284 bytes) 000093 1764745157.454939 write 0x035993 thru 0x03ddbb (0x8429 bytes) 000095 1764745157.456398 read 0x03a977 thru 0x03c8fe (0x1f88 bytes) 000096 1764745157.456771 read 0x002d60 thru 0x0051e5 (0x2486 bytes) 000098 1764745157.459944 MIRROR_RESYNC 8 mirrors 000099 1764745157.753873 mapread 0x017a75 thru 0x01c787 (0x4d13 bytes) 000100 1764745157.758546 read 0x028b1b thru 0x02efa0 (0x6486 bytes) 000102 1764745157.761622 MIRROR_EXTEND to 9 mirrors 000104 1764745157.933588 mapwrite 0x00b746 thru 0x0192a9 (0xdb64 bytes) 000105 1764745157.993808 mapwrite 0x00f78a thru 0x012d19 (0x3590 bytes) 000106 1764745158.006148 MIRROR_RESYNC 9 mirrors 000108 1764745158.315530 read 0x02645f thru 0x0313db (0xaf7d bytes) 000109 1764745158.318488 trunc from 0x03ddbc to 0x00a6e9 (0x336d2 bytes) 000110 1764745158.338126 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xa6ea): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa6ea): Mirror 2: 0xe1e1946b (stale) Mirror 3: 0xe1e1946b (stale) Mirror 4: 0xe1e1946b (stale) Mirror 5: 0xe1e1946b (stale) Mirror 6: 0xe1e1946b (stale) Mirror 7: 0xe1e1946b (stale) Mirror 8: 0xe1e1946b (stale) Mirror 9: 0xe1e1946b (stale) CRC-32 checksum value for chunk [0, 0xa6ea): Mirror 1: 0xe1e1946b Mirror 10: 0xe1e1946b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa6ea: skipped 000111 1764745158.569168 mapwrite 0x021094 thru 0x02f121 (0xe08e bytes) 000112 1764745158.618310 write 0x0190f2 thru 0x024306 (0xb215 bytes) 000115 1764745158.622265 trunc from 0x02f122 to 0x039356 (0xa235 bytes) 000117 1764745158.628424 trunc from 0x039357 to 0x02f950 (0x9a06 bytes) 000118 1764745158.632613 write 0x025d3b thru 0x031c5e (0xbf24 bytes) 000119 1764745158.633713 trunc from 0x031c5f to 0x035567 (0x3909 bytes) 000121 1764745158.637884 trunc from 0x035568 to 0x03422d (0x133a bytes) 000122 1764745158.641534 read 0x00ac0b thru 0x013204 (0x85fa bytes) 000123 1764745158.644291 read 0x02073e thru 0x02c05e (0xb921 bytes) 000124 1764745158.646164 MIRROR_RESYNC 10 mirrors 000125 1764745158.984602 mapwrite 0x02dd98 thru 0x038747 (0xa9b0 bytes) 000128 1764745159.030369 mapwrite 0x00306b thru 0x00e593 (0xb529 bytes) 000129 1764745159.048172 mapwrite 0x029ef3 thru 0x036b74 (0xcc82 bytes) 000131 1764745159.064915 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38748): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38748): Mirror 2: 0x96702e98 (stale) Mirror 3: 0x96702e98 (stale) Mirror 4: 0x96702e98 (stale) Mirror 5: 0x96702e98 (stale) Mirror 6: 0x96702e98 (stale) Mirror 7: 0x96702e98 (stale) Mirror 8: 0x96702e98 (stale) Mirror 9: 0x96702e98 (stale) Mirror 10: 0x96702e98 (stale) CRC-32 checksum value for chunk [0, 0x38748): Mirror 1: 0x72ae8c28 Mirror 11: 0x72ae8c28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38748: skipped 000132 1764745159.323827 mapread 0x022fed thru 0x032618 (0xf62c bytes) 000133 1764745159.336891 mapread 0x015499 thru 0x01bd2e (0x6896 bytes) 000134 1764745159.346634 MIRROR_RESYNC 11 mirrors truncating to largest ever: 0x3fc53 000135 1764745159.663566 trunc from 0x038748 to 0x03fc52 (0x750b bytes) 000136 1764745159.686800 mapwrite 0x03042f thru 0x03e0f9 (0xdccb bytes) 000138 1764745159.717096 write 0x03ab90 thru 0x03ffff (0x5470 bytes) 000139 1764745159.717841 trunc from 0x040000 to 0x01d3ac (0x22c53 bytes) 000141 1764745159.725087 trunc from 0x01d3ad to 0x012d30 (0xa67c bytes) 000142 1764745159.730751 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12d31): Mirror 2: 0xc39d27f7 (stale) Mirror 3: 0xc39d27f7 (stale) Mirror 4: 0xc39d27f7 (stale) Mirror 5: 0xc39d27f7 (stale) Mirror 6: 0xc39d27f7 (stale) Mirror 7: 0xc39d27f7 (stale) Mirror 8: 0xc39d27f7 (stale) Mirror 9: 0xc39d27f7 (stale) Mirror 11: 0xc39d27f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12d31): Mirror 2: 0xc39d27f7 (stale) Mirror 3: 0xc39d27f7 (stale) Mirror 4: 0xc39d27f7 (stale) Mirror 5: 0xc39d27f7 (stale) Mirror 6: 0xc39d27f7 (stale) Mirror 7: 0xc39d27f7 (stale) Mirror 8: 0xc39d27f7 (stale) Mirror 9: 0xc39d27f7 (stale) Mirror 11: 0xc39d27f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12d31: skipped 000145 1764745159.847682 trunc from 0x012d31 to 0x03d191 (0x2a461 bytes) 000147 1764745159.851961 read 0x00b123 thru 0x014016 (0x8ef4 bytes) 000149 1764745159.854744 read 0x0207fa thru 0x02ee4f (0xe656 bytes) 000150 1764745159.859272 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3d192): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d192): Mirror 2: 0x7389c799 (stale) Mirror 3: 0x7389c799 (stale) Mirror 4: 0x7389c799 (stale) Mirror 5: 0x7389c799 (stale) Mirror 6: 0x7389c799 (stale) Mirror 7: 0x7389c799 (stale) Mirror 8: 0x7389c799 (stale) Mirror 9: 0x7389c799 (stale) Mirror 11: 0x7389c799 (stale) CRC-32 checksum value for chunk [0, 0x3d192): Mirror 1: 0xf2ea1253 Mirror 12: 0xf2ea1253 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d192: skipped 000151 1764745160.089010 mapwrite 0x02e856 thru 0x02eafa (0x2a5 bytes) 000153 1764745160.139110 trunc from 0x03d192 to 0x004cb2 (0x384df bytes) 000158 1764745160.150156 read 0x0018e4 thru 0x004cb2 (0x33cf bytes) 000159 1764745160.152185 read 0x001cc5 thru 0x004cb2 (0x2fee bytes) 000160 1764745160.152540 read 0x0044af thru 0x004cb2 (0x804 bytes) 000161 1764745160.153919 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x4cb3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4cb3): Mirror 2: 0xbd0cb337 (stale) Mirror 3: 0xbd0cb337 (stale) Mirror 4: 0xbd0cb337 (stale) Mirror 5: 0xbd0cb337 (stale) Mirror 6: 0xbd0cb337 (stale) Mirror 7: 0xbd0cb337 (stale) Mirror 8: 0xbd0cb337 (stale) Mirror 9: 0xbd0cb337 (stale) Mirror 11: 0xbd0cb337 (stale) Mirror 12: 0xbd0cb337 (stale) CRC-32 checksum value for chunk [0, 0x4cb3): Mirror 1: 0xbd0cb337 Mirror 13: 0xbd0cb337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4cb3: skipped 000162 1764745160.362084 trunc from 0x004cb3 to 0x011496 (0xc7e4 bytes) 000164 1764745160.385924 write 0x039960 thru 0x03a230 (0x8d1 bytes) 000165 1764745160.388924 read 0x031354 thru 0x031c44 (0x8f1 bytes) 000166 1764745160.390782 trunc from 0x03a231 to 0x02a560 (0xfcd0 bytes) 000171 1764745160.399022 MIRROR_RESYNC 12 mirrors 000172 1764745160.747867 write 0x037b63 thru 0x039a27 (0x1ec5 bytes) 000174 1764745160.764898 mapread 0x0370ee thru 0x039a27 (0x293a bytes) 000177 1764745160.767414 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x39a28): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a28): Mirror 2: 0x4a1bafb8 (stale) Mirror 3: 0x4a1bafb8 (stale) Mirror 4: 0x4a1bafb8 (stale) Mirror 5: 0x4a1bafb8 (stale) Mirror 6: 0x4a1bafb8 (stale) Mirror 7: 0x4a1bafb8 (stale) Mirror 8: 0x4a1bafb8 (stale) Mirror 9: 0x4a1bafb8 (stale) Mirror 11: 0x4a1bafb8 (stale) Mirror 12: 0x4a1bafb8 (stale) Mirror 13: 0x4a1bafb8 (stale) CRC-32 checksum value for chunk [0, 0x39a28): Mirror 1: 0xf55c1ada Mirror 14: 0xf55c1ada lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a28: skipped 000183 1764745161.017483 mapwrite 0x01e456 thru 0x020e2e (0x29d9 bytes) 000185 1764745161.071286 mapwrite 0x02e83c thru 0x032705 (0x3eca bytes) 000189 1764745161.089570 read 0x00d560 thru 0x015ceb (0x878c bytes) 000190 1764745161.092512 mapread 0x00e7a9 thru 0x011325 (0x2b7d bytes) 000191 1764745161.093159 read 0x01234e thru 0x01ce42 (0xaaf5 bytes) 000192 1764745161.096361 mapread 0x00c87b thru 0x01708f (0xa815 bytes) 000193 1764745161.097049 write 0x000ca4 thru 0x006b1d (0x5e7a bytes) 000194 1764745161.097947 mapread 0x0149b5 thru 0x019dde (0x542a bytes) 000196 1764745161.098854 write 0x02d94b thru 0x03c685 (0xed3b bytes) 000198 1764745161.100051 write 0x00b3e8 thru 0x018762 (0xd37b bytes) 000201 1764745161.101679 trunc from 0x03c686 to 0x0215d2 (0x1b0b3 bytes) 000202 1764745161.113006 write 0x01f043 thru 0x02b2ea (0xc2a8 bytes) 000204 1764745161.114408 trunc from 0x02b2eb to 0x034989 (0x969f bytes) 000205 1764745161.118740 write 0x0080d5 thru 0x00b33a (0x3266 bytes) 000206 1764745161.119505 read 0x0335b1 thru 0x034989 (0x13d9 bytes) 000208 1764745161.122361 mapread 0x0337e9 thru 0x034989 (0x11a1 bytes) 000209 1764745161.124352 MIRROR_RESYNC 13 mirrors 000210 1764745161.521230 mapread 0x01458d thru 0x01e6ba (0xa12e bytes) 000211 1764745161.528889 MIRROR_EXTEND to 14 mirrors 000212 1764745161.732146 write 0x02b9b7 thru 0x02d4ce (0x1b18 bytes) 000214 1764745161.753284 mapread 0x029a34 thru 0x02e87e (0x4e4b bytes) 000215 1764745161.757564 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3498a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3498a): Mirror 2: 0x47b303de (stale) Mirror 3: 0x47b303de (stale) Mirror 4: 0x47b303de (stale) Mirror 5: 0x47b303de (stale) Mirror 6: 0x47b303de (stale) Mirror 7: 0x47b303de (stale) Mirror 8: 0x47b303de (stale) Mirror 9: 0x47b303de (stale) Mirror 11: 0x47b303de (stale) Mirror 12: 0x47b303de (stale) Mirror 13: 0x47b303de (stale) Mirror 14: 0x47b303de (stale) Mirror 15: 0x47b303de (stale) CRC-32 checksum value for chunk [0, 0x3498a): Mirror 1: 0x571d39bc Mirror 16: 0x571d39bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3498a: skipped 000216 1764745162.023108 write 0x00b50c thru 0x011a05 (0x64fa bytes) 000217 1764745162.043240 read 0x015bdb thru 0x02197c (0xbda2 bytes) 000218 1764745162.045852 write 0x01be38 thru 0x027fac (0xc175 bytes) 000222 1764745162.048969 write 0x012a31 thru 0x0130f2 (0x6c2 bytes) 000223 1764745162.051924 mapread 0x01742b thru 0x023be3 (0xc7b9 bytes) 000224 1764745162.052568 write 0x031e0f thru 0x032251 (0x443 bytes) 000225 1764745162.055262 mapwrite 0x01f9f2 thru 0x02260b (0x2c1a bytes) 000226 1764745162.070067 write 0x001972 thru 0x00640c (0x4a9b bytes) 000228 1764745162.073462 mapread 0x005309 thru 0x00c6d1 (0x73c9 bytes) 000229 1764745162.074076 write 0x037ce8 thru 0x03f67a (0x7993 bytes) 000231 1764745162.075112 trunc from 0x03f67b to 0x000f10 (0x3e76a bytes) 000232 1764745162.085373 write 0x022cac thru 0x02d4b6 (0xa80b bytes) 000234 1764745162.086680 read 0x0181bc thru 0x023970 (0xb7b5 bytes) 000235 1764745162.089873 mapread 0x020435 thru 0x022b09 (0x26d5 bytes) 000237 1764745162.091211 trunc from 0x02d4b7 to 0x008077 (0x2543f bytes) 000240 1764745162.096660 write 0x03c51b thru 0x03ffff (0x3ae5 bytes) 000241 1764745162.097498 read 0x029500 thru 0x035b74 (0xc675 bytes) 000242 1764745162.102016 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd6abcde7 (stale) Mirror 3: 0xd6abcde7 (stale) Mirror 4: 0xd6abcde7 (stale) Mirror 5: 0xd6abcde7 (stale) Mirror 6: 0xd6abcde7 (stale) Mirror 7: 0xd6abcde7 (stale) Mirror 8: 0xd6abcde7 (stale) Mirror 9: 0xd6abcde7 (stale) Mirror 11: 0xd6abcde7 (stale) Mirror 12: 0xd6abcde7 (stale) Mirror 13: 0xd6abcde7 (stale) Mirror 14: 0xd6abcde7 (stale) Mirror 15: 0xd6abcde7 (stale) Mirror 16: 0x4de03be6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x311f59a7 Mirror 17: 0x311f59a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000244 1764745162.363950 write 0x003248 thru 0x006496 (0x324f bytes) 000245 1764745162.386447 mapread 0x020cf0 thru 0x026233 (0x5544 bytes) 000248 1764745162.397476 read 0x01c899 thru 0x0230ec (0x6854 bytes) 000249 1764745162.398114 mapread 0x015962 thru 0x024242 (0xe8e1 bytes) 000252 1764745162.406883 MIRROR_RESYNC 16 mirrors 000253 1764745162.856357 trunc from 0x040000 to 0x02ccbe (0x13341 bytes) 000254 1764745162.879804 trunc from 0x02ccbf to 0x030c53 (0x3f95 bytes) 000256 1764745162.885663 write 0x00c267 thru 0x01128c (0x5026 bytes) 000259 1764745162.889837 trunc from 0x030c54 to 0x027932 (0x9321 bytes) 000260 1764745162.894490 mapwrite 0x021cb3 thru 0x02c751 (0xaa9f bytes) 000261 1764745162.920164 read 0x0197a3 thru 0x02825e (0xeabc bytes) 000263 1764745162.922918 read 0x000337 thru 0x00cbaa (0xc874 bytes) 000265 1764745162.926201 mapwrite 0x03ec1b thru 0x03ffff (0x13e5 bytes) 000270 1764745162.947557 write 0x01769e thru 0x0251d0 (0xdb33 bytes) 000272 1764745162.949236 read 0x01f125 thru 0x02b7dd (0xc6b9 bytes) 000274 1764745162.950242 mapwrite 0x03ce9e thru 0x03ffff (0x3162 bytes) 000275 1764745162.963230 mapwrite 0x03a411 thru 0x03ffff (0x5bef bytes) 000276 1764745162.976026 mapwrite 0x0264f5 thru 0x030d3e (0xa84a bytes) 000277 1764745162.989185 read 0x03ad69 thru 0x03ffff (0x5297 bytes) 000278 1764745162.989614 mapwrite 0x025bc8 thru 0x0322c4 (0xc6fd bytes) 000279 1764745163.003249 read 0x002576 thru 0x00a16a (0x7bf5 bytes) 000280 1764745163.003866 read 0x000e8d thru 0x00282e (0x19a2 bytes) 000282 1764745163.004536 mapwrite 0x01ab61 thru 0x026282 (0xb722 bytes) 000283 1764745163.017443 read 0x01c8e1 thru 0x02ae1c (0xe53c bytes) 000284 1764745163.019650 MIRROR_RESYNC 16 mirrors 000285 1764745163.504706 MIRROR_RESYNC 16 mirrors 000286 1764745163.692474 write 0x01175c thru 0x0214a8 (0xfd4d bytes) 000288 1764745163.714946 write 0x0253af thru 0x02b6d2 (0x6324 bytes) 000289 1764745163.719369 read 0x035ce7 thru 0x03ffff (0xa319 bytes) 000291 1764745163.722746 write 0x00e96f thru 0x011c7d (0x330f bytes) 000292 1764745163.726001 write 0x00ae0f thru 0x014bc0 (0x9db2 bytes) 000294 1764745163.727603 trunc from 0x040000 to 0x024d67 (0x1b298 bytes) 000297 1764745163.740103 trunc from 0x024d68 to 0x0027cb (0x2259c bytes) 000302 1764745163.746317 mapread 0x00175a thru 0x0027cb (0x1072 bytes) 000303 1764745163.748760 mapwrite 0x020f0c thru 0x030938 (0xfa2d bytes) 000304 1764745163.785152 read 0x0039af thru 0x006dcb (0x341d bytes) 000307 1764745163.788824 trunc from 0x030939 to 0x02395a (0xcfde bytes) 000308 1764745163.793514 mapread 0x00c278 thru 0x014d5d (0x8ae6 bytes) 000309 1764745163.795576 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2395b): Mirror 2: 0xc3285ea7 (stale) Mirror 3: 0xc3285ea7 (stale) Mirror 4: 0xc3285ea7 (stale) Mirror 5: 0xc3285ea7 (stale) Mirror 6: 0xc3285ea7 (stale) Mirror 7: 0xc3285ea7 (stale) Mirror 8: 0xc3285ea7 (stale) Mirror 9: 0xc3285ea7 (stale) Mirror 11: 0xc3285ea7 (stale) Mirror 12: 0xc3285ea7 (stale) Mirror 13: 0xc3285ea7 (stale) Mirror 14: 0xc3285ea7 (stale) Mirror 15: 0xc3285ea7 (stale) Mirror 16: 0xc3285ea7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2395b): Mirror 2: 0xc3285ea7 (stale) Mirror 3: 0xc3285ea7 (stale) Mirror 4: 0xc3285ea7 (stale) Mirror 5: 0xc3285ea7 (stale) Mirror 6: 0xc3285ea7 (stale) Mirror 7: 0xc3285ea7 (stale) Mirror 8: 0xc3285ea7 (stale) Mirror 9: 0xc3285ea7 (stale) Mirror 11: 0xc3285ea7 (stale) Mirror 12: 0xc3285ea7 (stale) Mirror 13: 0xc3285ea7 (stale) Mirror 14: 0xc3285ea7 (stale) Mirror 15: 0xc3285ea7 (stale) Mirror 16: 0xc3285ea7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2395b: skipped 000312 1764745163.953312 MIRROR_RESYNC 15 mirrors 000313 1764745164.400159 MIRROR_EXTEND to 16 mirrors 000317 1764745164.615039 mapread 0x0209e2 thru 0x02395a (0x2f79 bytes) 000322 1764745164.621524 read 0x004b75 thru 0x011c35 (0xd0c1 bytes) 000325 1764745164.624800 read 0x007207 thru 0x0102f9 (0x90f3 bytes) 000326 1764745164.625183 write 0x0031d3 thru 0x006724 (0x3552 bytes) 000327 1764745164.649505 trunc from 0x02395b to 0x0184fa (0xb460 bytes) 000328 1764745164.654201 mapwrite 0x011372 thru 0x020627 (0xf2b6 bytes) 000329 1764745164.682029 read 0x007480 thru 0x009447 (0x1fc8 bytes) 000330 1764745164.682594 mapread 0x009d80 thru 0x00b7b8 (0x1a39 bytes) 000333 1764745164.683633 read 0x01806c thru 0x020627 (0x85bc bytes) 000337 1764745164.684912 read 0x0117b3 thru 0x01b9b2 (0xa200 bytes) 000344 1764745164.687175 mapwrite 0x026207 thru 0x02e017 (0x7e11 bytes) 000345 1764745164.706213 trunc from 0x02e018 to 0x001971 (0x2c6a6 bytes) 000347 1764745164.711270 trunc from 0x001972 to 0x020f97 (0x1f626 bytes) 000348 1764745164.715456 trunc from 0x020f98 to 0x007a4c (0x1954b bytes) 000349 1764745164.721365 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7a4d): Mirror 3: 0xf65f509d (stale) Mirror 4: 0xf65f509d (stale) Mirror 5: 0xf65f509d (stale) Mirror 6: 0xf65f509d (stale) Mirror 7: 0xf65f509d (stale) Mirror 8: 0xf65f509d (stale) Mirror 9: 0xf65f509d (stale) Mirror 11: 0xf65f509d (stale) Mirror 12: 0xf65f509d (stale) Mirror 13: 0xf65f509d (stale) Mirror 14: 0xf65f509d (stale) Mirror 15: 0xf65f509d (stale) Mirror 16: 0xf65f509d (stale) Mirror 17: 0xf65f509d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7a4d): Mirror 3: 0xf65f509d (stale) Mirror 4: 0xf65f509d (stale) Mirror 5: 0xf65f509d (stale) Mirror 6: 0xf65f509d (stale) Mirror 7: 0xf65f509d (stale) Mirror 8: 0xf65f509d (stale) Mirror 9: 0xf65f509d (stale) Mirror 11: 0xf65f509d (stale) Mirror 12: 0xf65f509d (stale) Mirror 13: 0xf65f509d (stale) Mirror 14: 0xf65f509d (stale) Mirror 15: 0xf65f509d (stale) Mirror 16: 0xf65f509d (stale) Mirror 17: 0xf65f509d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7a4d: skipped 000350 1764745164.868998 trunc from 0x007a4d to 0x030746 (0x28cfa bytes) 000352 1764745164.873241 write 0x00471d thru 0x00df31 (0x9815 bytes) 000353 1764745164.876658 trunc from 0x030747 to 0x008b28 (0x27c1e bytes) 000355 1764745164.882827 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8b29): Mirror 3: 0x707e7dbc (stale) Mirror 4: 0x707e7dbc (stale) Mirror 5: 0x707e7dbc (stale) Mirror 6: 0x707e7dbc (stale) Mirror 7: 0x707e7dbc (stale) Mirror 8: 0x707e7dbc (stale) Mirror 9: 0x707e7dbc (stale) Mirror 11: 0x707e7dbc (stale) Mirror 12: 0x707e7dbc (stale) Mirror 13: 0x707e7dbc (stale) Mirror 14: 0x707e7dbc (stale) Mirror 15: 0x707e7dbc (stale) Mirror 16: 0x707e7dbc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8b29): Mirror 3: 0x707e7dbc (stale) Mirror 4: 0x707e7dbc (stale) Mirror 5: 0x707e7dbc (stale) Mirror 6: 0x707e7dbc (stale) Mirror 7: 0x707e7dbc (stale) Mirror 8: 0x707e7dbc (stale) Mirror 9: 0x707e7dbc (stale) Mirror 11: 0x707e7dbc (stale) Mirror 12: 0x707e7dbc (stale) Mirror 13: 0x707e7dbc (stale) Mirror 14: 0x707e7dbc (stale) Mirror 15: 0x707e7dbc (stale) Mirror 16: 0x707e7dbc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8b29: skipped 000356 1764745165.032046 read 0x008976 thru 0x008b28 (0x1b3 bytes) 000357 1764745165.033850 read 0x00362f thru 0x0055c2 (0x1f94 bytes) 000358 1764745165.036851 mapwrite 0x03bb5e thru 0x03f875 (0x3d18 bytes) 000359 1764745165.059665 MIRROR_RESYNC 14 mirrors 000361 1764745165.469317 mapread 0x022819 thru 0x0238fa (0x10e2 bytes) 000362 1764745165.472340 mapread 0x01b9ff thru 0x02abae (0xf1b0 bytes) 000363 1764745165.485161 read 0x0308ad thru 0x037261 (0x69b5 bytes) 000367 1764745165.488780 trunc from 0x03f876 to 0x006821 (0x39054 bytes) 000369 1764745165.519470 read 0x006149 thru 0x006821 (0x6d9 bytes) 000370 1764745165.522445 mapread 0x00357e thru 0x006821 (0x32a4 bytes) 000372 1764745165.523391 trunc from 0x006822 to 0x029397 (0x22b76 bytes) 000373 1764745165.529734 mapwrite 0x017bfd thru 0x0274ef (0xf8f3 bytes) 000374 1764745165.555394 mapwrite 0x02e1bd thru 0x039dbc (0xbc00 bytes) 000375 1764745165.578166 trunc from 0x039dbd to 0x031e62 (0x7f5a bytes) 000377 1764745165.583954 mapwrite 0x020cc4 thru 0x028a7d (0x7dba bytes) 000378 1764745165.598831 mapread 0x025b41 thru 0x0298f2 (0x3db2 bytes) 000380 1764745165.599811 trunc from 0x031e63 to 0x03f40f (0xd5ad bytes) 000382 1764745165.605668 write 0x03fbbc thru 0x03ffff (0x444 bytes) 000386 1764745165.610354 write 0x03f1f1 thru 0x03ffff (0xe0f bytes) 000387 1764745165.610884 write 0x018d09 thru 0x022943 (0x9c3b bytes) 000390 1764745165.615111 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc7d91bc4 (stale) Mirror 4: 0xc7d91bc4 (stale) Mirror 5: 0xc7d91bc4 (stale) Mirror 6: 0xc7d91bc4 (stale) Mirror 7: 0xc7d91bc4 (stale) Mirror 8: 0xc7d91bc4 (stale) Mirror 9: 0xc7d91bc4 (stale) Mirror 11: 0xc7d91bc4 (stale) Mirror 12: 0xc7d91bc4 (stale) Mirror 13: 0xc7d91bc4 (stale) Mirror 14: 0xc7d91bc4 (stale) Mirror 15: 0xc7d91bc4 (stale) Mirror 16: 0xc7d91bc4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef80e0f7 Mirror 17: 0xef80e0f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000391 1764745165.897865 write 0x029278 thru 0x02d793 (0x451c bytes) 000392 1764745165.917604 read 0x02bf7a thru 0x02eaf6 (0x2b7d bytes) 000394 1764745165.919500 mapwrite 0x028eea thru 0x032960 (0x9a77 bytes) 000395 1764745165.939185 trunc from 0x040000 to 0x036eaa (0x9155 bytes) 000396 1764745165.949274 trunc from 0x036eab to 0x007447 (0x2fa63 bytes) 000399 1764745165.954265 mapwrite 0x00a41b thru 0x00f9d8 (0x55be bytes) 000401 1764745165.973566 read 0x005538 thru 0x007950 (0x2419 bytes) 000402 1764745165.974074 trunc from 0x00f9d9 to 0x026ca5 (0x172cd bytes) 000403 1764745165.978351 read 0x00817c thru 0x00f6a2 (0x7527 bytes) 000404 1764745165.978928 trunc from 0x026ca6 to 0x0080a5 (0x1ec00 bytes) 000405 1764745165.982824 write 0x026e76 thru 0x029272 (0x23fd bytes) 000406 1764745165.983595 write 0x00e4b8 thru 0x013ec2 (0x5a0b bytes) 000407 1764745165.988037 MIRROR_RESYNC 15 mirrors 000409 1764745166.430607 read 0x025b05 thru 0x02871a (0x2c16 bytes) 000412 1764745166.432483 mapread 0x021ef6 thru 0x029272 (0x737d bytes) 000414 1764745166.436679 write 0x009947 thru 0x00ddf3 (0x44ad bytes) 000415 1764745166.455031 read 0x004a33 thru 0x00a2ee (0x58bc bytes) 000416 1764745166.455469 read 0x00a7b8 thru 0x010558 (0x5da1 bytes) 000417 1764745166.458500 mapwrite 0x027a19 thru 0x028b30 (0x1118 bytes) 000418 1764745166.472949 read 0x0118bf thru 0x01c63b (0xad7d bytes) 000419 1764745166.473391 read 0x0096e2 thru 0x00bd15 (0x2634 bytes) 000420 1764745166.475159 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x29273): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29273): Mirror 3: 0xe1a66e1c (stale) Mirror 4: 0xe1a66e1c (stale) Mirror 5: 0xe1a66e1c (stale) Mirror 6: 0xe1a66e1c (stale) Mirror 7: 0xe1a66e1c (stale) Mirror 8: 0xe1a66e1c (stale) Mirror 9: 0xe1a66e1c (stale) Mirror 11: 0xe1a66e1c (stale) Mirror 12: 0xe1a66e1c (stale) Mirror 13: 0xe1a66e1c (stale) Mirror 14: 0xe1a66e1c (stale) Mirror 15: 0xe1a66e1c (stale) Mirror 16: 0xe1a66e1c (stale) Mirror 17: 0xe1a66e1c (stale) CRC-32 checksum value for chunk [0, 0x29273): Mirror 1: 0x3f33adc Mirror 18: 0x3f33adc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29273: skipped 000421 1764745166.727470 trunc from 0x029273 to 0x0261f6 (0x307c bytes) 000423 1764745166.752168 write 0x028559 thru 0x03686f (0xe317 bytes) 000425 1764745166.753631 trunc from 0x036870 to 0x03bbba (0x534b bytes) 000429 1764745166.759895 write 0x00f670 thru 0x014a58 (0x53e9 bytes) 000430 1764745166.762752 trunc from 0x03bbbb to 0x0200f8 (0x1bac2 bytes) 000432 1764745166.767844 mapread 0x00658c thru 0x00e998 (0x840d bytes) 000433 1764745166.770688 read 0x00fee7 thru 0x01bae3 (0xbbfd bytes) 000434 1764745166.773334 trunc from 0x0200f9 to 0x00c512 (0x13be6 bytes) 000435 1764745166.777945 mapwrite 0x0171ab thru 0x0217d4 (0xa62a bytes) 000438 1764745166.799625 mapread 0x00f37e thru 0x018876 (0x94f9 bytes) 000439 1764745166.803282 MIRROR_RESYNC 16 mirrors 000440 1764745167.228927 mapread 0x006c52 thru 0x016488 (0xf837 bytes) 000441 1764745167.240690 write 0x025e5c thru 0x0288c6 (0x2a6b bytes) 000442 1764745167.256774 trunc from 0x0288c7 to 0x012b69 (0x15d5d bytes) 000444 1764745167.262719 MIRROR_RESYNC 16 mirrors 000446 1764745167.707398 write 0x01f863 thru 0x021205 (0x19a3 bytes) 000447 1764745167.727770 mapwrite 0x02bb35 thru 0x02eb59 (0x3025 bytes) 000449 1764745167.756721 mapread 0x018268 thru 0x02391c (0xb6b5 bytes) 000451 1764745167.762692 read 0x02e0b0 thru 0x02eb59 (0xaaa bytes) 000452 1764745167.763116 mapread 0x00b552 thru 0x00d958 (0x2407 bytes) 000453 1764745167.766199 trunc from 0x02eb5a to 0x029a06 (0x5153 bytes) 000456 1764745167.772454 mapread 0x007534 thru 0x00e81f (0x72ec bytes) 000459 1764745167.774124 mapwrite 0x01a027 thru 0x01ff49 (0x5f23 bytes) 000460 1764745167.789181 read 0x025d49 thru 0x029a06 (0x3cbe bytes) 000461 1764745167.792410 write 0x00d113 thru 0x017cc7 (0xabb5 bytes) 000462 1764745167.793110 mapread 0x00efce thru 0x0179bd (0x89f0 bytes) 000463 1764745167.795097 MIRROR_RESYNC 16 mirrors 000464 1764745168.266949 mapwrite 0x00dd52 thru 0x01274b (0x49fa bytes) 000466 1764745168.317447 mapread 0x0243fa thru 0x027d26 (0x392d bytes) 000467 1764745168.320545 write 0x02141e thru 0x0246fb (0x32de bytes) 000469 1764745168.321558 trunc from 0x029a07 to 0x029bd5 (0x1cf bytes) 000471 1764745168.327232 write 0x02a4f3 thru 0x02fb41 (0x564f bytes) 000473 1764745168.328631 mapwrite 0x0241f3 thru 0x02f89f (0xb6ad bytes) 000474 1764745168.342417 mapwrite 0x03c4f3 thru 0x03ffff (0x3b0d bytes) 000475 1764745168.362534 trunc from 0x040000 to 0x02efcf (0x11030 bytes) 000476 1764745168.367142 write 0x00b38c thru 0x01b13a (0xfdaf bytes) 000478 1764745168.370118 write 0x01d4bf thru 0x02a7cb (0xd30d bytes) 000480 1764745168.371677 read 0x00c739 thru 0x018d20 (0xc5e8 bytes) 000481 1764745168.372224 mapwrite 0x0056cd thru 0x006175 (0xaa9 bytes) 000482 1764745168.390688 trunc from 0x02efd0 to 0x01e295 (0x10d3a bytes) 000483 1764745168.396079 read 0x00982a thru 0x014565 (0xad3c bytes) 000485 1764745168.396807 write 0x03035f thru 0x03384a (0x34ec bytes) 000487 1764745168.397923 write 0x01f604 thru 0x02a840 (0xb23d bytes) 000488 1764745168.401202 write 0x039ca8 thru 0x03ffff (0x6358 bytes) 000489 1764745168.402019 read 0x034929 thru 0x03ffff (0xb6d7 bytes) 000491 1764745168.404860 mapwrite 0x0292a1 thru 0x03751a (0xe27a bytes) 000494 1764745168.420826 write 0x01e12f thru 0x02ad6e (0xcc40 bytes) 000495 1764745168.423360 MIRROR_RESYNC 16 mirrors 000497 1764745168.848227 read 0x02622a thru 0x02cdc5 (0x6b9c bytes) 000500 1764745168.850895 trunc from 0x040000 to 0x0242e2 (0x1bd1d bytes) 000503 1764745168.876376 mapread 0x00782a thru 0x00812e (0x905 bytes) 000504 1764745168.880646 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x242e3): Mirror 3: 0x29a54fee (stale) Mirror 4: 0x29a54fee (stale) Mirror 6: 0x29a54fee (stale) Mirror 7: 0x29a54fee (stale) Mirror 8: 0x29a54fee (stale) Mirror 9: 0x29a54fee (stale) Mirror 11: 0x29a54fee (stale) Mirror 12: 0x29a54fee (stale) Mirror 13: 0x29a54fee (stale) Mirror 14: 0x29a54fee (stale) Mirror 15: 0x29a54fee (stale) Mirror 16: 0x29a54fee (stale) Mirror 17: 0x29a54fee (stale) Mirror 18: 0x29a54fee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242e3): Mirror 3: 0x29a54fee (stale) Mirror 4: 0x29a54fee (stale) Mirror 6: 0x29a54fee (stale) Mirror 7: 0x29a54fee (stale) Mirror 8: 0x29a54fee (stale) Mirror 9: 0x29a54fee (stale) Mirror 11: 0x29a54fee (stale) Mirror 12: 0x29a54fee (stale) Mirror 13: 0x29a54fee (stale) Mirror 14: 0x29a54fee (stale) Mirror 15: 0x29a54fee (stale) Mirror 16: 0x29a54fee (stale) Mirror 17: 0x29a54fee (stale) Mirror 18: 0x29a54fee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242e3: skipped 000507 1764745169.043529 MIRROR_RESYNC 15 mirrors 000509 1764745169.479492 write 0x01cda7 thru 0x024a65 (0x7cbf bytes) 000510 1764745169.502481 mapwrite 0x013388 thru 0x01a9e7 (0x7660 bytes) 000511 1764745169.537847 write 0x010034 thru 0x01a8ed (0xa8ba bytes) 000512 1764745169.539029 write 0x03be65 thru 0x03ffff (0x419b bytes) 000515 1764745169.540865 read 0x01fc82 thru 0x02cb59 (0xced8 bytes) 000516 1764745169.544493 write 0x02a949 thru 0x039b57 (0xf20f bytes) 000517 1764745169.547479 trunc from 0x040000 to 0x01b164 (0x24e9b bytes) 000518 1764745169.553789 write 0x01ec80 thru 0x02bd1a (0xd09b bytes) 000519 1764745169.555142 trunc from 0x02bd1b to 0x01bdc0 (0xff5a bytes) 000520 1764745169.559765 mapwrite 0x020bba thru 0x02d9ec (0xce33 bytes) 000521 1764745169.585986 mapwrite 0x00d96d thru 0x01b0f2 (0xd786 bytes) 000524 1764745169.599749 read 0x02cf88 thru 0x02d9ec (0xa65 bytes) 000525 1764745169.600174 write 0x01816f thru 0x020187 (0x8019 bytes) 000527 1764745169.601353 mapwrite 0x0235f1 thru 0x02e342 (0xad52 bytes) 000530 1764745169.626098 write 0x021364 thru 0x02e8b1 (0xd54e bytes) 000531 1764745169.627311 mapread 0x01c237 thru 0x02a7a1 (0xe56b bytes) 000532 1764745169.627865 trunc from 0x02e8b2 to 0x00b743 (0x2316e bytes) 000535 1764745169.634243 trunc from 0x00b744 to 0x00c30f (0xbcc bytes) 000536 1764745169.640468 MIRROR_RESYNC 15 mirrors 000537 1764745170.061622 mapwrite 0x01e1a9 thru 0x0218ae (0x3706 bytes) 000541 1764745170.108930 mapread 0x013246 thru 0x01bce4 (0x8a9f bytes) 000542 1764745170.116606 write 0x025a1e thru 0x02c653 (0x6c36 bytes) 000544 1764745170.117750 write 0x031506 thru 0x0341de (0x2cd9 bytes) 000546 1764745170.118763 mapwrite 0x000e14 thru 0x00dd0f (0xcefc bytes) 000547 1764745170.135220 write 0x007ea4 thru 0x0081b5 (0x312 bytes) 000548 1764745170.136082 mapwrite 0x01b22a thru 0x0213cc (0x61a3 bytes) 000550 1764745170.151386 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x341df): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341df): Mirror 3: 0xeaebe97c (stale) Mirror 4: 0xeaebe97c (stale) Mirror 6: 0xeaebe97c (stale) Mirror 7: 0xeaebe97c (stale) Mirror 8: 0xeaebe97c (stale) Mirror 9: 0xeaebe97c (stale) Mirror 11: 0xeaebe97c (stale) Mirror 12: 0xeaebe97c (stale) Mirror 13: 0xeaebe97c (stale) Mirror 14: 0xeaebe97c (stale) Mirror 15: 0xeaebe97c (stale) Mirror 16: 0xeaebe97c (stale) Mirror 17: 0xeaebe97c (stale) Mirror 18: 0xeaebe97c (stale) CRC-32 checksum value for chunk [0, 0x341df): Mirror 1: 0xfe230da8 Mirror 19: 0xfe230da8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341df: skipped 000551 1764745170.462253 write 0x009bd6 thru 0x01121b (0x7646 bytes) 000555 1764745170.493500 mapwrite 0x033cdc thru 0x039b50 (0x5e75 bytes) 000556 1764745170.536122 mapwrite 0x02bfe0 thru 0x038732 (0xc753 bytes) 000558 1764745170.558497 write 0x03e160 thru 0x03ffff (0x1ea0 bytes) 000559 1764745170.559244 read 0x0062ed thru 0x00dbf9 (0x790d bytes) 000560 1764745170.562383 write 0x021cde thru 0x02c7f1 (0xab14 bytes) 000565 1764745170.568033 mapwrite 0x00f5fb thru 0x018f8c (0x9992 bytes) 000566 1764745170.598934 trunc from 0x040000 to 0x015177 (0x2ae88 bytes) 000569 1764745170.606257 write 0x03d22f thru 0x03ffff (0x2dd1 bytes) 000570 1764745170.607096 mapwrite 0x001300 thru 0x002526 (0x1227 bytes) 000572 1764745170.632105 MIRROR_RESYNC 16 mirrors 000573 1764745171.230791 mapwrite 0x00c7ec thru 0x01128c (0x4aa1 bytes) 000574 1764745171.287376 mapread 0x01a83b thru 0x020430 (0x5bf6 bytes) 000575 1764745171.292791 mapread 0x002f92 thru 0x007b34 (0x4ba3 bytes) 000576 1764745171.296098 mapwrite 0x01506c thru 0x020549 (0xb4de bytes) 000577 1764745171.312276 trunc from 0x040000 to 0x001d66 (0x3e299 bytes) 000578 1764745171.319122 mapread 0x001d2a thru 0x001d66 (0x3d bytes) 000580 1764745171.322346 MIRROR_RESYNC 16 mirrors 000582 1764745171.747833 write 0x014bf3 thru 0x01f203 (0xa611 bytes) 000583 1764745171.768624 mapwrite 0x02f900 thru 0x032660 (0x2d61 bytes) 000584 1764745171.803541 mapwrite 0x013a52 thru 0x020446 (0xc9f5 bytes) 000585 1764745171.823559 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32661): Mirror 3: 0xeaadc21c (stale) Mirror 6: 0xeaadc21c (stale) Mirror 7: 0xeaadc21c (stale) Mirror 8: 0xeaadc21c (stale) Mirror 9: 0xeaadc21c (stale) Mirror 11: 0xeaadc21c (stale) Mirror 12: 0xeaadc21c (stale) Mirror 13: 0xeaadc21c (stale) Mirror 14: 0xeaadc21c (stale) Mirror 15: 0xeaadc21c (stale) Mirror 16: 0xeaadc21c (stale) Mirror 17: 0xeaadc21c (stale) Mirror 18: 0xeaadc21c (stale) Mirror 19: 0xeaadc21c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32661): Mirror 3: 0xeaadc21c (stale) Mirror 6: 0xeaadc21c (stale) Mirror 7: 0xeaadc21c (stale) Mirror 8: 0xeaadc21c (stale) Mirror 9: 0xeaadc21c (stale) Mirror 11: 0xeaadc21c (stale) Mirror 12: 0xeaadc21c (stale) Mirror 13: 0xeaadc21c (stale) Mirror 14: 0xeaadc21c (stale) Mirror 15: 0xeaadc21c (stale) Mirror 16: 0xeaadc21c (stale) Mirror 17: 0xeaadc21c (stale) Mirror 18: 0xeaadc21c (stale) Mirror 19: 0xeaadc21c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32661: skipped 000586 1764745172.008043 mapwrite 0x034aa6 thru 0x039db7 (0x5312 bytes) 000587 1764745172.030363 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39db8): Mirror 3: 0xfd4e63ea (stale) Mirror 6: 0xfd4e63ea (stale) Mirror 7: 0xfd4e63ea (stale) Mirror 8: 0xfd4e63ea (stale) Mirror 9: 0xfd4e63ea (stale) Mirror 12: 0xfd4e63ea (stale) Mirror 13: 0xfd4e63ea (stale) Mirror 14: 0xfd4e63ea (stale) Mirror 15: 0xfd4e63ea (stale) Mirror 16: 0xfd4e63ea (stale) Mirror 17: 0xfd4e63ea (stale) Mirror 18: 0xfd4e63ea (stale) Mirror 19: 0xfd4e63ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39db8): Mirror 3: 0xfd4e63ea (stale) Mirror 6: 0xfd4e63ea (stale) Mirror 7: 0xfd4e63ea (stale) Mirror 8: 0xfd4e63ea (stale) Mirror 9: 0xfd4e63ea (stale) Mirror 12: 0xfd4e63ea (stale) Mirror 13: 0xfd4e63ea (stale) Mirror 14: 0xfd4e63ea (stale) Mirror 15: 0xfd4e63ea (stale) Mirror 16: 0xfd4e63ea (stale) Mirror 17: 0xfd4e63ea (stale) Mirror 18: 0xfd4e63ea (stale) Mirror 19: 0xfd4e63ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39db8: skipped 000588 1764745172.199159 write 0x007f23 thru 0x00cc71 (0x4d4f bytes) 000589 1764745172.202659 mapwrite 0x021e4b thru 0x02b64b (0x9801 bytes) 000591 1764745172.230183 trunc from 0x039db8 to 0x01c5c0 (0x1d7f7 bytes) 000592 1764745172.237216 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c5c1): Mirror 3: 0x35a5d597 (stale) Mirror 7: 0x35a5d597 (stale) Mirror 8: 0x35a5d597 (stale) Mirror 9: 0x35a5d597 (stale) Mirror 12: 0x35a5d597 (stale) Mirror 13: 0x35a5d597 (stale) Mirror 14: 0x35a5d597 (stale) Mirror 15: 0x35a5d597 (stale) Mirror 16: 0x35a5d597 (stale) Mirror 17: 0x35a5d597 (stale) Mirror 18: 0x35a5d597 (stale) Mirror 19: 0x35a5d597 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c5c1): Mirror 3: 0x35a5d597 (stale) Mirror 7: 0x35a5d597 (stale) Mirror 8: 0x35a5d597 (stale) Mirror 9: 0x35a5d597 (stale) Mirror 12: 0x35a5d597 (stale) Mirror 13: 0x35a5d597 (stale) Mirror 14: 0x35a5d597 (stale) Mirror 15: 0x35a5d597 (stale) Mirror 16: 0x35a5d597 (stale) Mirror 17: 0x35a5d597 (stale) Mirror 18: 0x35a5d597 (stale) Mirror 19: 0x35a5d597 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c5c1: skipped 000594 1764745172.392442 write 0x01e7ea thru 0x02af39 (0xc750 bytes) 000595 1764745172.393425 mapread 0x01380a thru 0x0187c3 (0x4fba bytes) 000597 1764745172.401539 write 0x0131da thru 0x0181ad (0x4fd4 bytes) 000601 1764745172.403601 mapwrite 0x009e81 thru 0x00e3fb (0x457b bytes) 000602 1764745172.420261 write 0x01b1cf thru 0x01c390 (0x11c2 bytes) 000604 1764745172.421194 mapwrite 0x01b93b thru 0x026e13 (0xb4d9 bytes) 000605 1764745172.435873 read 0x021223 thru 0x02af39 (0x9d17 bytes) 000606 1764745172.436454 read 0x013e03 thru 0x01d81d (0x9a1b bytes) 000608 1764745172.437070 write 0x016628 thru 0x01f982 (0x935b bytes) 000610 1764745172.438557 read 0x02142e thru 0x02af39 (0x9b0c bytes) 000613 1764745172.440007 trunc from 0x02af3a to 0x016a8a (0x144af bytes) 000614 1764745172.445375 mapread 0x00b9e0 thru 0x00fb28 (0x4149 bytes) 000615 1764745172.445902 mapwrite 0x016b72 thru 0x02244a (0xb8d9 bytes) 000618 1764745172.467593 mapread 0x0125a6 thru 0x0202ac (0xdd07 bytes) 000619 1764745172.470104 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2244b): Mirror 7: 0x5104dab1 (stale) Mirror 8: 0x5104dab1 (stale) Mirror 9: 0x5104dab1 (stale) Mirror 12: 0x5104dab1 (stale) Mirror 13: 0x5104dab1 (stale) Mirror 14: 0x5104dab1 (stale) Mirror 15: 0x5104dab1 (stale) Mirror 16: 0x5104dab1 (stale) Mirror 17: 0x5104dab1 (stale) Mirror 18: 0x5104dab1 (stale) Mirror 19: 0x5104dab1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2244b): Mirror 7: 0x5104dab1 (stale) Mirror 8: 0x5104dab1 (stale) Mirror 9: 0x5104dab1 (stale) Mirror 12: 0x5104dab1 (stale) Mirror 13: 0x5104dab1 (stale) Mirror 14: 0x5104dab1 (stale) Mirror 15: 0x5104dab1 (stale) Mirror 16: 0x5104dab1 (stale) Mirror 17: 0x5104dab1 (stale) Mirror 18: 0x5104dab1 (stale) Mirror 19: 0x5104dab1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2244b: skipped 000620 1764745172.626078 mapwrite 0x01593c thru 0x0226e4 (0xcda9 bytes) 000622 1764745172.657824 mapread 0x0043e0 thru 0x004446 (0x67 bytes) 000625 1764745172.663413 mapread 0x017417 thru 0x01fe6c (0x8a56 bytes) 000627 1764745172.664417 read 0x00b383 thru 0x01b2a0 (0xff1e bytes) 000629 1764745172.665368 write 0x032128 thru 0x0345b7 (0x2490 bytes) 000632 1764745172.667336 mapwrite 0x03836f thru 0x03ffff (0x7c91 bytes) 000633 1764745172.693135 write 0x032f49 thru 0x03983a (0x68f2 bytes) 000634 1764745172.693993 trunc from 0x040000 to 0x01ab01 (0x254fe bytes) 000635 1764745172.699711 read 0x004bc8 thru 0x00b5ea (0x6a23 bytes) 000636 1764745172.700203 read 0x016a56 thru 0x01ab01 (0x40ac bytes) 000638 1764745172.701228 mapwrite 0x02b449 thru 0x0348fe (0x94b6 bytes) 000640 1764745172.723763 read 0x00e467 thru 0x016c66 (0x8800 bytes) 000641 1764745172.724346 trunc from 0x0348ff to 0x01e69b (0x16263 bytes) 000644 1764745172.730104 mapwrite 0x03120a thru 0x0351b8 (0x3faf bytes) 000646 1764745172.751715 mapwrite 0x0086cc thru 0x0140f0 (0xba25 bytes) 000650 1764745172.769171 trunc from 0x0351b9 to 0x0123ea (0x22dce bytes) 000651 1764745172.774916 read 0x00409a thru 0x006eac (0x2e13 bytes) 000653 1764745172.775890 read 0x00cd0e thru 0x0123ea (0x56dd bytes) 000654 1764745172.778565 MIRROR_RESYNC 12 mirrors 000655 1764745173.165361 write 0x00f8ef thru 0x0129cf (0x30e1 bytes) 000656 1764745173.187120 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x129d0): Mirror 8: 0x5e54a22d (stale) Mirror 9: 0x5e54a22d (stale) Mirror 12: 0x5e54a22d (stale) Mirror 13: 0x5e54a22d (stale) Mirror 14: 0x5e54a22d (stale) Mirror 15: 0x5e54a22d (stale) Mirror 16: 0x5e54a22d (stale) Mirror 17: 0x5e54a22d (stale) Mirror 18: 0x5e54a22d (stale) Mirror 19: 0x5e54a22d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x129d0): Mirror 8: 0x5e54a22d (stale) Mirror 9: 0x5e54a22d (stale) Mirror 12: 0x5e54a22d (stale) Mirror 13: 0x5e54a22d (stale) Mirror 14: 0x5e54a22d (stale) Mirror 15: 0x5e54a22d (stale) Mirror 16: 0x5e54a22d (stale) Mirror 17: 0x5e54a22d (stale) Mirror 18: 0x5e54a22d (stale) Mirror 19: 0x5e54a22d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x129d0: skipped 000657 1764745173.327013 trunc from 0x0129d0 to 0x01bde7 (0x9418 bytes) 000658 1764745173.331717 read 0x001a24 thru 0x00e9e0 (0xcfbd bytes) 000660 1764745173.335323 mapread 0x00c765 thru 0x015907 (0x91a3 bytes) 000662 1764745173.343617 write 0x0105bf thru 0x0167c5 (0x6207 bytes) 000663 1764745173.344557 mapwrite 0x02b438 thru 0x02fcd1 (0x489a bytes) 000665 1764745173.367237 mapwrite 0x013932 thru 0x0186de (0x4dad bytes) 000666 1764745173.380661 trunc from 0x02fcd2 to 0x013ecd (0x1be04 bytes) 000668 1764745173.386122 write 0x027f4f thru 0x02fa92 (0x7b44 bytes) 000672 1764745173.388275 read 0x00663b thru 0x011692 (0xb058 bytes) 000673 1764745173.388840 mapwrite 0x010166 thru 0x0186ac (0x8547 bytes) 000676 1764745173.406003 trunc from 0x02fa93 to 0x02a0d6 (0x59bc bytes) 000679 1764745173.411227 trunc from 0x02a0d7 to 0x023eef (0x61e7 bytes) 000681 1764745173.415495 mapread 0x021c07 thru 0x023eef (0x22e9 bytes) 000682 1764745173.416184 mapwrite 0x036284 thru 0x03905e (0x2ddb bytes) 000683 1764745173.436481 read 0x025241 thru 0x033ffb (0xedbb bytes) 000685 1764745173.439964 mapread 0x02685b thru 0x02ff22 (0x96c8 bytes) 000686 1764745173.442176 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3905f): Mirror 8: 0x7aa6d4d6 (stale) Mirror 9: 0x7aa6d4d6 (stale) Mirror 12: 0x7aa6d4d6 (stale) Mirror 13: 0x7aa6d4d6 (stale) Mirror 14: 0x7aa6d4d6 (stale) Mirror 15: 0x7aa6d4d6 (stale) Mirror 16: 0x7aa6d4d6 (stale) Mirror 17: 0x7aa6d4d6 (stale) Mirror 18: 0x7aa6d4d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3905f): Mirror 8: 0x7aa6d4d6 (stale) Mirror 9: 0x7aa6d4d6 (stale) Mirror 12: 0x7aa6d4d6 (stale) Mirror 13: 0x7aa6d4d6 (stale) Mirror 14: 0x7aa6d4d6 (stale) Mirror 15: 0x7aa6d4d6 (stale) Mirror 16: 0x7aa6d4d6 (stale) Mirror 17: 0x7aa6d4d6 (stale) Mirror 18: 0x7aa6d4d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3905f: skipped 000687 1764745173.578776 write 0x00b28a thru 0x0166ca (0xb441 bytes) 000688 1764745173.584070 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3905f): Mirror 8: 0x7aa6d4d6 (stale) Mirror 9: 0x7aa6d4d6 (stale) Mirror 13: 0x7aa6d4d6 (stale) Mirror 14: 0x7aa6d4d6 (stale) Mirror 15: 0x7aa6d4d6 (stale) Mirror 16: 0x7aa6d4d6 (stale) Mirror 17: 0x7aa6d4d6 (stale) Mirror 18: 0x7aa6d4d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3905f): Mirror 8: 0x7aa6d4d6 (stale) Mirror 9: 0x7aa6d4d6 (stale) Mirror 13: 0x7aa6d4d6 (stale) Mirror 14: 0x7aa6d4d6 (stale) Mirror 15: 0x7aa6d4d6 (stale) Mirror 16: 0x7aa6d4d6 (stale) Mirror 17: 0x7aa6d4d6 (stale) Mirror 18: 0x7aa6d4d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3905f: skipped 000690 1764745173.722426 read 0x020693 thru 0x02d8b2 (0xd220 bytes) 000691 1764745173.726291 mapread 0x006ae8 thru 0x0106f1 (0x9c0a bytes) 000693 1764745173.735440 MIRROR_RESYNC 9 mirrors 000695 1764745174.081396 read 0x011f79 thru 0x01911b (0x71a3 bytes) 000696 1764745174.085213 mapread 0x01501f thru 0x01b977 (0x6959 bytes) 000700 1764745174.091923 MIRROR_RESYNC 9 mirrors 000703 1764745174.277427 MIRROR_RESYNC 9 mirrors 000704 1764745174.465054 read 0x0164df thru 0x019e9b (0x39bd bytes) 000706 1764745174.465595 trunc from 0x03905f to 0x00538c (0x33cd2 bytes) 000707 1764745174.489771 read 0x00325c thru 0x00538c (0x2131 bytes) 000708 1764745174.493171 read 0x003d36 thru 0x00538c (0x1657 bytes) 000709 1764745174.493708 read 0x002f4b thru 0x00538c (0x2442 bytes) 000714 1764745174.495849 mapwrite 0x02663e thru 0x03183f (0xb202 bytes) 000719 1764745174.528580 mapwrite 0x035f05 thru 0x03ae4b (0x4f47 bytes) 000720 1764745174.555108 mapwrite 0x016e7b thru 0x0253ed (0xe573 bytes) 000721 1764745174.572572 mapwrite 0x02bc6c thru 0x038171 (0xc506 bytes) 000723 1764745174.591121 trunc from 0x03ae4c to 0x02ec8e (0xc1bd bytes) 000725 1764745174.596795 trunc from 0x02ec8f to 0x03f065 (0x103d7 bytes) 000726 1764745174.601527 mapwrite 0x02f682 thru 0x031ad9 (0x2458 bytes) 000728 1764745174.620719 mapread 0x02bb58 thru 0x02da64 (0x1f0d bytes) 000729 1764745174.621570 write 0x03defd thru 0x03ffff (0x2103 bytes) 000730 1764745174.622457 write 0x0381b4 thru 0x039b6d (0x19ba bytes) 000732 1764745174.627208 trunc from 0x040000 to 0x03f22f (0xdd0 bytes) 000733 1764745174.633377 write 0x039d09 thru 0x03bcb2 (0x1faa bytes) 000734 1764745174.634174 read 0x03b31a thru 0x03f22f (0x3f16 bytes) 000735 1764745174.634657 trunc from 0x03f230 to 0x0047a8 (0x3aa87 bytes) 000737 1764745174.644033 MIRROR_RESYNC 9 mirrors 000738 1764745175.001017 trunc from 0x0047a9 to 0x02f4f9 (0x2ad51 bytes) 000740 1764745175.026483 mapread 0x0264b6 thru 0x02f40f (0x8f5a bytes) 000741 1764745175.032610 mapwrite 0x03ecaf thru 0x03ffff (0x1351 bytes) 000743 1764745175.057463 trunc from 0x040000 to 0x0326ec (0xd913 bytes) 000744 1764745175.064269 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326ed): Mirror 9: 0x1474af43 (stale) Mirror 13: 0x1474af43 (stale) Mirror 14: 0x1474af43 (stale) Mirror 15: 0x1474af43 (stale) Mirror 16: 0x1474af43 (stale) Mirror 17: 0x1474af43 (stale) Mirror 18: 0x1474af43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ed): Mirror 9: 0x1474af43 (stale) Mirror 13: 0x1474af43 (stale) Mirror 14: 0x1474af43 (stale) Mirror 15: 0x1474af43 (stale) Mirror 16: 0x1474af43 (stale) Mirror 17: 0x1474af43 (stale) Mirror 18: 0x1474af43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ed: skipped 000748 1764745175.196140 mapread 0x029930 thru 0x02d7d1 (0x3ea2 bytes) 000749 1764745175.203395 read 0x01ef73 thru 0x02d9c5 (0xea53 bytes) 000752 1764745175.209388 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x326ed): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ed): Mirror 9: 0x1474af43 (stale) Mirror 13: 0x1474af43 (stale) Mirror 14: 0x1474af43 (stale) Mirror 15: 0x1474af43 (stale) Mirror 16: 0x1474af43 (stale) Mirror 17: 0x1474af43 (stale) Mirror 18: 0x1474af43 (stale) CRC-32 checksum value for chunk [0, 0x326ed): Mirror 1: 0x1474af43 Mirror 19: 0x1474af43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ed: skipped 000753 1764745175.502810 mapread 0x024187 thru 0x029531 (0x53ab bytes) 000754 1764745175.510154 read 0x01829f thru 0x025c38 (0xd99a bytes) 000756 1764745175.515343 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326ed): Mirror 9: 0x1474af43 (stale) Mirror 13: 0x1474af43 (stale) Mirror 14: 0x1474af43 (stale) Mirror 15: 0x1474af43 (stale) Mirror 16: 0x1474af43 (stale) Mirror 18: 0x1474af43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326ed): Mirror 9: 0x1474af43 (stale) Mirror 13: 0x1474af43 (stale) Mirror 14: 0x1474af43 (stale) Mirror 15: 0x1474af43 (stale) Mirror 16: 0x1474af43 (stale) Mirror 18: 0x1474af43 (stale) CRC-32 checksum value for chunk [0, 0x326ed): Mirror 1: 0x1474af43 Mirror 19: 0x1474af43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326ed: skipped 000757 1764745175.749454 mapread 0x03241b thru 0x0326ec (0x2d2 bytes) 000760 1764745175.760103 mapread 0x028f1f thru 0x02fdef (0x6ed1 bytes) 000761 1764745175.767728 write 0x03a3f5 thru 0x03ffff (0x5c0b bytes) 000763 1764745175.789198 trunc from 0x040000 to 0x004d56 (0x3b2a9 bytes) 000765 1764745175.800103 write 0x03d197 thru 0x03ffff (0x2e69 bytes) 000766 1764745175.800892 read 0x02928d thru 0x02df4e (0x4cc2 bytes) 000767 1764745175.803615 read 0x01c06d thru 0x023ed3 (0x7e67 bytes) 000768 1764745175.807597 trunc from 0x040000 to 0x01f364 (0x20c9b bytes) 000769 1764745175.812726 write 0x03742e thru 0x03fb31 (0x8704 bytes) 000771 1764745175.814488 mapread 0x01ef4d thru 0x02934c (0xa400 bytes) 000772 1764745175.830902 mapread 0x01d19a thru 0x02790e (0xa775 bytes) 000773 1764745175.831545 mapread 0x00358f thru 0x005069 (0x1adb bytes) 000776 1764745175.834886 MIRROR_RESYNC 8 mirrors 000779 1764745176.250931 read 0x02ab21 thru 0x02f632 (0x4b12 bytes) 000780 1764745176.254070 trunc from 0x03fb32 to 0x026992 (0x1919f bytes) 000783 1764745176.283082 MIRROR_RESYNC 8 mirrors 000784 1764745176.633993 read 0x021066 thru 0x026992 (0x592d bytes) 000785 1764745176.636373 mapread 0x0193a0 thru 0x020377 (0x6fd8 bytes) 000786 1764745176.642318 read 0x018148 thru 0x026992 (0xe84b bytes) 000790 1764745176.643257 trunc from 0x026993 to 0x014392 (0x12600 bytes) 000793 1764745176.663691 read 0x0093eb thru 0x014392 (0xafa8 bytes) 000794 1764745176.667369 trunc from 0x014393 to 0x03928b (0x24ef9 bytes) 000795 1764745176.673688 trunc from 0x03928c to 0x0072e2 (0x31fa9 bytes) 000798 1764745176.679538 mapread 0x000409 thru 0x001ad1 (0x16c9 bytes) 000799 1764745176.680409 write 0x03ebe1 thru 0x03ffff (0x141f bytes) 000800 1764745176.681165 mapwrite 0x0056a8 thru 0x006c1d (0x1576 bytes) 000801 1764745176.701387 mapread 0x00c602 thru 0x018f8a (0xc989 bytes) 000802 1764745176.723425 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 15: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 19: 0x9bfd37c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 15: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 19: 0x9bfd37c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000804 1764745176.861695 mapwrite 0x02e874 thru 0x035bc3 (0x7350 bytes) 000806 1764745176.888585 read 0x014669 thru 0x01fe96 (0xb82e bytes) 000807 1764745176.891751 mapwrite 0x00c451 thru 0x00c9dc (0x58c bytes) 000811 1764745176.906730 mapread 0x028946 thru 0x02c77a (0x3e35 bytes) 000812 1764745176.910106 mapwrite 0x030caa thru 0x039f0b (0x9262 bytes) 000813 1764745176.928754 read 0x01e733 thru 0x028b88 (0xa456 bytes) 000816 1764745176.932570 read 0x013180 thru 0x01d2c2 (0xa143 bytes) 000817 1764745176.933002 write 0x03f3fc thru 0x03ffff (0xc04 bytes) 000818 1764745176.933571 mapread 0x010bc4 thru 0x012bfc (0x2039 bytes) 000819 1764745176.934164 mapread 0x01c7b7 thru 0x023f75 (0x77bf bytes) 000822 1764745176.935478 trunc from 0x040000 to 0x039fa4 (0x605b bytes) 000823 1764745176.940093 write 0x028b7b thru 0x02ffc4 (0x744a bytes) 000824 1764745176.940897 mapwrite 0x0138f8 thru 0x023648 (0xfd51 bytes) 000826 1764745176.957697 mapwrite 0x03c720 thru 0x03ffff (0x38e0 bytes) 000827 1764745176.979637 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 15: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 19: 0x9bfd37c9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa32c5add Mirror 20: 0xa32c5add lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000828 1764745177.274336 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 19: 0x9bfd37c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 19: 0x9bfd37c9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa32c5add Mirror 20: 0xa32c5add lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000830 1764745177.525466 write 0x03953f thru 0x03ffff (0x6ac1 bytes) 000832 1764745177.547472 mapread 0x02a953 thru 0x02b3aa (0xa58 bytes) 000835 1764745177.552956 mapwrite 0x0362d3 thru 0x038b20 (0x284e bytes) 000836 1764745177.577972 mapread 0x00809c thru 0x0172ae (0xf213 bytes) 000837 1764745177.592536 mapwrite 0x00fc0f thru 0x0171f0 (0x75e2 bytes) 000838 1764745177.608729 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 19: 0x9bfd37c9 (stale) Mirror 20: 0xa32c5add (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x28f6b01b Mirror 21: 0x28f6b01b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000839 1764745177.864800 mapread 0x01afe0 thru 0x022a4c (0x7a6d bytes) 000841 1764745177.878074 write 0x00eed9 thru 0x01252b (0x3653 bytes) 000843 1764745177.907477 mapwrite 0x023c6c thru 0x025e36 (0x21cb bytes) 000844 1764745177.938969 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 20: 0xa32c5add (stale) Mirror 21: 0x28f6b01b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9bfd37c9 (stale) Mirror 13: 0x9bfd37c9 (stale) Mirror 14: 0x9bfd37c9 (stale) Mirror 18: 0x9bfd37c9 (stale) Mirror 20: 0xa32c5add (stale) Mirror 21: 0x28f6b01b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1764745178.104334 write 0x00f671 thru 0x01764a (0x7fda bytes) 000846 1764745178.108677 read 0x0153f9 thru 0x017351 (0x1f59 bytes) 000849 1764745178.109981 mapwrite 0x00ed74 thru 0x01c64d (0xd8da bytes) 000850 1764745178.134104 trunc from 0x040000 to 0x02b34b (0x14cb4 bytes) 000852 1764745178.147493 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b34c): Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b34c): Mirror 9: 0x887ea961 (stale) Mirror 13: 0x887ea961 (stale) Mirror 14: 0x887ea961 (stale) Mirror 18: 0x887ea961 (stale) Mirror 20: 0x31b943c2 (stale) Mirror 21: 0xddf3db91 (stale) CRC-32 checksum value for chunk [0, 0x2b34c): Mirror 1: 0xfcd4555b Mirror 22: 0xfcd4555b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b34c: skipped 000853 1764745178.479272 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b34c): Mirror 9: 0x887ea961 (stale) Mirror 13: 0x887ea961 (stale) Mirror 14: 0x887ea961 (stale) Mirror 18: 0x887ea961 (stale) Mirror 21: 0xddf3db91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b34c): Mirror 9: 0x887ea961 (stale) Mirror 13: 0x887ea961 (stale) Mirror 14: 0x887ea961 (stale) Mirror 18: 0x887ea961 (stale) Mirror 21: 0xddf3db91 (stale) CRC-32 checksum value for chunk [0, 0x2b34c): Mirror 1: 0xfcd4555b Mirror 22: 0xfcd4555b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b34c: skipped 000854 1764745178.725725 mapread 0x01a958 thru 0x020067 (0x5710 bytes) 000855 1764745178.736850 trunc from 0x02b34c to 0x0028dc (0x28a6f bytes) 000858 1764745178.761623 write 0x01a140 thru 0x027b65 (0xda26 bytes) 000859 1764745178.763281 trunc from 0x027b66 to 0x034de9 (0xd284 bytes) 000860 1764745178.770191 write 0x02b3c0 thru 0x037223 (0xbe64 bytes) 000861 1764745178.773977 trunc from 0x037224 to 0x02298a (0x14899 bytes) 000862 1764745178.779811 mapread 0x00ea6c thru 0x017ba0 (0x9135 bytes) 000863 1764745178.799771 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2298b): Mirror 9: 0x65fad520 (stale) Mirror 14: 0x65fad520 (stale) Mirror 18: 0x65fad520 (stale) Mirror 21: 0x2556d502 (stale) Mirror 22: 0xca739dec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2298b): Mirror 9: 0x65fad520 (stale) Mirror 14: 0x65fad520 (stale) Mirror 18: 0x65fad520 (stale) Mirror 21: 0x2556d502 (stale) Mirror 22: 0xca739dec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2298b: skipped 000864 1764745178.925966 mapread 0x012c5d thru 0x016766 (0x3b0a bytes) 000865 1764745178.937474 trunc from 0x02298b to 0x0353a1 (0x12a17 bytes) 000866 1764745178.942907 mapread 0x002f6e thru 0x00f328 (0xc3bb bytes) 000868 1764745178.948731 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x353a2): Mirror 9: 0x4f12f90c (stale) Mirror 14: 0x4f12f90c (stale) Mirror 21: 0x2b5e096f (stale) Mirror 22: 0x147a3462 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x353a2): Mirror 9: 0x4f12f90c (stale) Mirror 14: 0x4f12f90c (stale) Mirror 21: 0x2b5e096f (stale) Mirror 22: 0x147a3462 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x353a2: skipped 000871 1764745179.069422 mapwrite 0x0118aa thru 0x018795 (0x6eec bytes) 000872 1764745179.114385 mapwrite 0x02b279 thru 0x0317d5 (0x655d bytes) 000875 1764745179.136124 write 0x028d90 thru 0x030edf (0x8150 bytes) 000877 1764745179.137489 mapread 0x013215 thru 0x018b8a (0x5976 bytes) 000878 1764745179.138442 trunc from 0x0353a2 to 0x01c9f1 (0x189b0 bytes) 000879 1764745179.145554 read 0x00f462 thru 0x0120fe (0x2c9d bytes) 000881 1764745179.146552 mapwrite 0x02710d thru 0x033358 (0xc24c bytes) 000882 1764745179.173346 mapread 0x00dc7f thru 0x01bb0b (0xde8d bytes) 000883 1764745179.174225 trunc from 0x033359 to 0x037dda (0x4a82 bytes) 000885 1764745179.180138 mapwrite 0x03312b thru 0x03bb9e (0x8a74 bytes) 000886 1764745179.206816 mapwrite 0x00c896 thru 0x011bbd (0x5328 bytes) 000887 1764745179.224950 mapread 0x03a0a7 thru 0x03bb9e (0x1af8 bytes) 000888 1764745179.225699 trunc from 0x03bb9f to 0x00439d (0x37801 bytes) 000889 1764745179.231798 mapwrite 0x00c6d0 thru 0x012000 (0x5931 bytes) 000894 1764745179.261065 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12001): Mirror 14: 0x29ba5478 (stale) Mirror 21: 0xb98b3938 (stale) Mirror 22: 0x1e85ceca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12001): Mirror 14: 0x29ba5478 (stale) Mirror 21: 0xb98b3938 (stale) Mirror 22: 0x1e85ceca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12001: skipped 000896 1764745179.369393 read 0x00b750 thru 0x012000 (0x68b1 bytes) 000899 1764745179.372969 write 0x0087c6 thru 0x00d3e4 (0x4c1f bytes) 000902 1764745179.375789 mapwrite 0x02c2bf thru 0x039b59 (0xd89b bytes) 000903 1764745179.401918 mapread 0x034ab4 thru 0x039b59 (0x50a6 bytes) 000906 1764745179.403205 mapwrite 0x03ded4 thru 0x03ffff (0x212c bytes) 000907 1764745179.425496 read 0x004434 thru 0x009bb3 (0x5780 bytes) 000909 1764745179.430266 read 0x003fe2 thru 0x0124cb (0xe4ea bytes) 000911 1764745179.431372 mapread 0x00330f thru 0x0132fc (0xffee bytes) 000912 1764745179.432245 write 0x00ca03 thru 0x017a31 (0xb02f bytes) 000914 1764745179.433627 write 0x00de4e thru 0x01addc (0xcf8f bytes) 000916 1764745179.434883 write 0x03f971 thru 0x03ffff (0x68f bytes) 000918 1764745179.437682 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9bfd37c9 (stale) Mirror 21: 0x28f6b01b (stale) Mirror 22: 0x6ede889 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5cf60bd0 Mirror 23: 0x5cf60bd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000921 1764745179.700258 MIRROR_RESYNC 5 mirrors 000922 1764745179.932060 mapwrite 0x01af2c thru 0x01fb12 (0x4be7 bytes) 000923 1764745179.987359 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x5cf60bd0 (stale) Mirror 21: 0x5cf60bd0 (stale) Mirror 22: 0x5cf60bd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x5cf60bd0 (stale) Mirror 21: 0x5cf60bd0 (stale) Mirror 22: 0x5cf60bd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000925 1764745180.104725 trunc from 0x040000 to 0x02d737 (0x128c8 bytes) 000927 1764745180.117701 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d738): Mirror 14: 0xa7e0fd9a (stale) Mirror 21: 0xa7e0fd9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d738): Mirror 14: 0xa7e0fd9a (stale) Mirror 21: 0xa7e0fd9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d738: skipped 000928 1764745180.213237 write 0x03aa0f thru 0x03c5bc (0x1bae bytes) 000929 1764745180.214252 trunc from 0x03c5bd to 0x00b8ee (0x30cce bytes) 000930 1764745180.219376 write 0x034c55 thru 0x03ffff (0xb3ab bytes) 000933 1764745180.221391 read 0x011756 thru 0x019289 (0x7b34 bytes) 000934 1764745180.224534 mapwrite 0x023048 thru 0x0238e3 (0x89c bytes) 000936 1764745180.239547 write 0x022ec0 thru 0x024165 (0x12a6 bytes) 000939 1764745180.244332 read 0x016635 thru 0x018f1f (0x28eb bytes) 000941 1764745180.245092 write 0x037e32 thru 0x03ffff (0x81ce bytes) 000942 1764745180.245851 trunc from 0x040000 to 0x03a209 (0x5df6 bytes) 000945 1764745180.253024 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a20a): Mirror 14: 0x7fce9b6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a20a): Mirror 14: 0x7fce9b6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a20a: skipped 000946 1764745180.332223 mapread 0x02b350 thru 0x0370c0 (0xbd71 bytes) 000947 1764745180.342343 write 0x024066 thru 0x0321b0 (0xe14b bytes) 000949 1764745180.345871 mapread 0x0108bd thru 0x0195d4 (0x8d18 bytes) 000953 1764745180.361002 write 0x00548b thru 0x0100ad (0xac23 bytes) 000954 1764745180.366428 MIRROR_RESYNC 2 mirrors 000956 1764745180.539696 write 0x02cfd2 thru 0x03c8be (0xf8ed bytes) 000957 1764745180.558421 write 0x00f2dc thru 0x017562 (0x8287 bytes) 000958 1764745180.561699 write 0x008a77 thru 0x00dcb2 (0x523c bytes) 000959 1764745180.565116 read 0x00a14c thru 0x017d0b (0xdbc0 bytes) 000960 1764745180.565710 write 0x02e54f thru 0x033437 (0x4ee9 bytes) 000961 1764745180.568440 MIRROR_RESYNC 2 mirrors 000964 1764745180.750886 mapread 0x006fce thru 0x0137d5 (0xc808 bytes) 000968 1764745180.764732 write 0x0234fa thru 0x02fdc2 (0xc8c9 bytes) 000970 1764745180.784623 read 0x01fede thru 0x02fe2c (0xff4f bytes) 000971 1764745180.786638 mapread 0x029643 thru 0x02cded (0x37ab bytes) 000976 1764745180.788451 read 0x02b378 thru 0x033bfb (0x8884 bytes) 000977 1764745180.792983 MIRROR_RESYNC 2 mirrors 000978 1764745180.973856 read 0x0077f9 thru 0x00c67a (0x4e82 bytes) 000980 1764745180.978199 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) 000983 1764745181.016284 read 0x01bc14 thru 0x01cb6d (0xf5a bytes) 000985 1764745181.018634 mapread 0x003d1a thru 0x00fdb7 (0xc09e bytes) 000986 1764745181.023677 read 0x023e0c thru 0x033bb2 (0xfda7 bytes) 000988 1764745181.027273 mapread 0x033b0f thru 0x03c8be (0x8db0 bytes) 000990 1764745181.028225 read 0x02481c thru 0x02d281 (0x8a66 bytes) 000991 1764745181.028733 read 0x00ef50 thru 0x0131f9 (0x42aa bytes) 000992 1764745181.029117 mapread 0x034605 thru 0x03c8be (0x82ba bytes) 000993 1764745181.029801 trunc from 0x03c8bf to 0x02e358 (0xe566 bytes) 000994 1764745181.043041 MIRROR_EXTEND to 2 mirrors 000995 1764745181.212420 MIRROR_RESYNC 2 mirrors 000996 1764745181.323449 mapread 0x0244b9 thru 0x02d735 (0x927d bytes) 000999 1764745181.331038 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69c:0x0]: Invalid argument (22) 001000 1764745181.369036 mapread 0x02c4e3 thru 0x02e358 (0x1e76 bytes) All operations completed A-OK! PASS 70a (29s) == sanity-flr test 71: check mirror extend parent fid ==== 01:59:43 (1764745183) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 71: check append on FLR file ========== 01:59:44 (1764745184) /mnt/lustre/f71.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0747164 s, 28.1 MB/s /mnt/lustre/f71.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496406 s, 21.1 MB/s /mnt/lustre/f71.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f71.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9f7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 71 (2s) == sanity-flr test 200a: stress test ===================== 01:59:46 (1764745186) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3999904 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8603840 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6214656 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7083328 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7793984 .. Extending file size to 7893440 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9324608 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8210336 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8645920 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8797536 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8891232 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59381 59382 59384 59386 59388 done 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) 192.168.202.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:00:52 (1764745252) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4572800 .. Extending file size to 615584 .. done Extending file size to 938528 .. Extending file size to 7317856 .. Extending file size to 4728032 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7701376 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8555712 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9351104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8422432 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8510848 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8794848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9143584 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9177632 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9250528 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 63114 63115 63117 63119 63121 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) 192.168.202.131@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:01:57 (1764745317) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5120 2203008 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 19456 3749888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 5120 2203008 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 19456 3749888 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 45056 7493632 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.031646 s, 66.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107071 s, 19.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x15869779 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:02:03 (1764745323) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0308919 s, 67.9 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 02:02:05 (1764745325) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6e1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa10:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa10:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa11:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xa0c:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:02:08 (1764745328) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0466217 s, 45.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa12:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:02:10 (1764745330) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0456484 s, 45.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa13:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa14:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa13:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa14:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa0f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa10:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:02:12 (1764745332) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0426323 s, 49.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa15:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa11:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa15:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa11:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa12:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:02:36 (1764745356) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224361 s, 46.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa16:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa14:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa13:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa16:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa14:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa13:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa17:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:03:20 (1764745400) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa15:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa16:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0413218 s, 50.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa15:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa16:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa18:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa15:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa16:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa18:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:04:07 (1764745447) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg231-server mds-ost sync done. Creating new pool test_204f oleg231-server: Pool lustre.test_204f created Adding targets to pool oleg231-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg231-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0387819 s, 54.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa17:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa19:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa17:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa19:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg231-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg231-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg231-server: Pool lustre.test_204f destroyed PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:05:05 (1764745505) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:05:08 (1764745508) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:05:10 (1764745510) oleg231-server: Pool lustre.test_206 created Adding targets to pool oleg231-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg231-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg231-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg231-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg231-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:05:25 (1764745525) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026048 s, 40.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115896 s, 90.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb2341a3f Mirror 3: 0xb2341a3f PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:05:28 (1764745528) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:05:29 (1764745529) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:05:31 (1764745531) Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0217525 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1810000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1280000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1280000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2440000 Read this (4 bytes): BBBB 192.168.202.131@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg231-client.virtnet /mnt/lustre2 (opts:) PASS 209a (15s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:05:46 (1764745546) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00327244 s, 2.5 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:05:49 (1764745549) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162423 s, 64.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x706 lmm_fid: [0x200000404:0x706:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x706 lmm_fid: [0x200000404:0x706:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa23:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835718 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-03 02:05:49.000000000 -0500 Modify: 2025-12-03 02:05:49.000000000 -0500 Change: 2025-12-03 02:05:50.000000000 -0500 Birth: 2025-12-03 02:05:49.000000000 -0500 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:05:51 (1764745551) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:05:52 (1764745552) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.12505 s, 328 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0120163 s, 341 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 3 02:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x708:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 3 02:05 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:05:55 (1764745555) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:05:57 (1764745557) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 842 sec ============ 02:05:59 (1764745559) === sanity-flr: start cleanup 02:06:00 (1764745560) === === sanity-flr: finish cleanup 02:06:01 (1764745561) ===