-----============= acceptance-small: sanity-flr ============----- Mon Jan 12 15:18:28 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg320-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:18:51 (1768249131) === oleg320-client.virtnet: executing check_config_client /mnt/lustre oleg320-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg320-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f11a112d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f11a112d000.idle_timeout=debug disable quota as required oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:19:35 (1768249175) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8381] == sanity-flr test 0a: lfs mirror create with -N option == 15:19:37 (1768249177) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:19:56 (1768249196) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:19:59 (1768249199) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:20:03 (1768249203) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:20:28 (1768249228) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:20:32 (1768249232) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:20:36 (1768249236) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f11a112d000.stats=clear osc.lustre-OST0001-osc-ffff9f11a112d000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:20:46 (1768249246) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:20:58 (1768249258) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:21:08 (1768249268) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:21:12 (1768249272) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 15:21:23 (1768249283) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:21:31 (1768249291) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:21:40 (1768249300) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:21:51 (1768249311) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:22:04 (1768249324) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:22:12 (1768249332) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.15499 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184486 s, 5.7 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:22:25 (1768249345) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.141366 s, 29.7 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:22:34 (1768249354) Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg320-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:23:47 (1768249427) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.915617 s, 12.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:24:39 (1768249479) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg320-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (83s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:26:02 (1768249562) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.58214 s, 11.4 MB/s -rw-r--r-- 1 root root 52428800 Jan 12 15:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.052s user 0m0.000s sys 0m0.157s Stopping /mnt/lustre-ost1 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f11a112d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.530s user 0m0.002s sys 0m0.158s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg320-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg320-server oleg320-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.353s user 0m0.001s sys 0m0.200s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg320-server: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg320-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg320-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f11a112d000.ost_server_uuid in FULL state after 0 sec PASS 33b (87s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:27:30 (1768249650) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:27:34 (1768249654) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:27:38 (1768249658) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 15:27:42 (1768249662) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 15:27:51 (1768249671) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.716833 s, 17.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:28:06 (1768249686) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467761 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330531 s, 31.7 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:28:15 (1768249695) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:28:25 (1768249705) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00400815 s, 255 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00169892 s, 603 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00400937 s, 255 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163729 s, 625 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 15:28:38 (1768249718) create test files with size 15755366 .. d02b5150a6f08d853135129be02b1aac - 69133d0c6a92b6a2f409eb9707c948cf - e575df433b936f2634c6b63ce4251063 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg320-client.virtnet /mnt/lustre (opts:) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 15:29:07 (1768249747) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:29:23 (1768249763) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:29:31 (1768249771) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164557 s, 6.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x35:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822402 s, 12.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 15:29:40 (1768249780) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192627 s, 10.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.347127 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160395 s, 13.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 15:30:01 (1768249801) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 15:30:05 (1768249805) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:30:10 (1768249810) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447147 s, 23.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 15:30:22 (1768249822) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.504691 s, 6.2 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:30:34 (1768249834) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x40:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:30:45 (1768249845) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.178468 s, 58.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg320-client.virtnet /mnt/lustre (opts:) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:30:57 (1768249857) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f1184cee000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.398104 s, 26.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg320-client.virtnet /mnt/lustre (opts:) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f1185191000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:31:14 (1768249874) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:31:54 (1768249914) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:32:36 (1768249956) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:32:40 (1768249960) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.177878 s, 17.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.188676 s, 16.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x92a30f5d (stale) Mirror 2: 0x92a30f5d (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: c5ff13459c6459d299219fa417d03903 - mirror 1 checksum: b1a9f78c6c6354ccc00c652a5f975789 - mirror 2 checksum: c5ff13459c6459d299219fa417d03903 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jan 12 15:32:44 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4b:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jan 12 15:32:44 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c5ff13459c6459d299219fa417d03903 - mirror 1 checksum: b1a9f78c6c6354ccc00c652a5f975789 - mirror 2 checksum: b1a9f78c6c6354ccc00c652a5f975789 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jan 12 15:32:44 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4b:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jan 12 15:32:44 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:32:56 (1768249976) Failing mds1 on oleg320-server Stopping /mnt/lustre-mds1 (opts:) on oleg320-server 15:33:04 (1768249984) shut down facet: mds1 facet_host: oleg320-server facet_failover_host: oleg320-server Failover mds1 to oleg320-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg320-server: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg320-client: oleg320-server: ssh exited with exit code 1 Started lustre-MDT0000 15:33:37 (1768250017) targets are mounted 15:33:37 (1768250017) facet_failover done oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xe:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341799 s, 30.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xe lmm_fid: [0x200000405:0xe:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:33:52 (1768250032) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:33:55 (1768250035) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:33:59 (1768250039) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0222215 s, 737 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:34:07 (1768250047) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:34:11 (1768250051) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:34:15 (1768250055) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x7:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:34:41 (1768250081) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:35:11 (1768250111) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:35:37 (1768250137) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:36:02 (1768250162) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 15:36:11 (1768250171) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 54379 mirror split pid 54380 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7638: 54380 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 54379 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (72s) == sanity-flr test 70a: flr mode fsx test ================ 15:37:23 (1768250243) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 245 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1768250246.102020 mapwrite 0x01af4f thru 0x02563c (0xa6ee bytes) 000003 1768250246.440973 mapwrite 0x01667d thru 0x016aa2 (0x426 bytes) 000004 1768250246.631090 read 0x006243 thru 0x00e875 (0x8633 bytes) 000005 1768250246.648700 mapread 0x020d03 thru 0x02563c (0x493a bytes) 000009 1768250246.655507 mapread 0x00b8b1 thru 0x00e4b7 (0x2c07 bytes) 000010 1768250246.681280 MIRROR_EXTEND to 2 mirrors 000011 1768250247.730867 read 0x0076fa thru 0x00e7d0 (0x70d7 bytes) 000012 1768250247.750266 write 0x006d37 thru 0x00bcb7 (0x4f81 bytes) 000013 1768250247.960019 mapwrite 0x01048a thru 0x019151 (0x8cc8 bytes) 000014 1768250248.118058 mapwrite 0x00cf5c thru 0x00efcc (0x2071 bytes) 000017 1768250248.185742 read 0x023e46 thru 0x02563c (0x17f7 bytes) truncating to largest ever: 0x3c7a 000018 1768250248.196966 trunc from 0x02563d to 0x003c79 (0x219c3 bytes) 000020 1768250248.239260 mapwrite 0x005eed thru 0x012387 (0xc49b bytes) 000021 1768250248.613164 read 0x000d3d thru 0x00f297 (0xe55b bytes) 000023 1768250248.622703 mapread 0x00e759 thru 0x012387 (0x3c2f bytes) 000024 1768250248.641672 MIRROR_RESYNC 2 mirrors 000026 1768250249.998849 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000027 1768250250.293417 write 0x0215a5 thru 0x02dd05 (0xc761 bytes) truncating to largest ever: 0x1e717 000028 1768250250.327525 trunc from 0x02dd06 to 0x01e716 (0xf5ef bytes) 000031 1768250250.357947 mapread 0x008b97 thru 0x0163ab (0xd815 bytes) 000032 1768250250.421982 write 0x02e296 thru 0x03b0ca (0xce35 bytes) 000038 1768250250.447643 read 0x028ae2 thru 0x033dd0 (0xb2ef bytes) 000042 1768250250.473962 read 0x01eb63 thru 0x01f4d3 (0x971 bytes) 000046 1768250250.492594 write 0x0287c8 thru 0x02aaf0 (0x2329 bytes) 000047 1768250250.493937 mapwrite 0x03c1d4 thru 0x03f9c7 (0x37f4 bytes) 000048 1768250250.647029 write 0x0070d3 thru 0x00a666 (0x3594 bytes) truncating to largest ever: 0x25056 000049 1768250250.649992 trunc from 0x03f9c8 to 0x025055 (0x1a972 bytes) 000051 1768250250.688771 trunc from 0x025056 to 0x0116b2 (0x139a3 bytes) 000052 1768250250.722668 write 0x028205 thru 0x02a2f8 (0x20f4 bytes) 000060 1768250250.732296 read 0x00f8a8 thru 0x01408d (0x47e6 bytes) 000062 1768250250.748265 mapwrite 0x0252a9 thru 0x02819c (0x2ef4 bytes) 000063 1768250250.875135 MIRROR_EXTEND to 2 mirrors 000068 1768250252.306600 mapwrite 0x03c849 thru 0x03e57a (0x1d32 bytes) truncating to largest ever: 0x266da 000069 1768250252.580988 trunc from 0x03e57b to 0x0266d9 (0x17ea1 bytes) 000071 1768250252.633578 write 0x03084a thru 0x0326f7 (0x1eae bytes) 000073 1768250252.643450 mapread 0x018ba9 thru 0x028a5d (0xfeb5 bytes) 000074 1768250252.687146 mapwrite 0x032215 thru 0x03ffff (0xddeb bytes) 000077 1768250252.852449 read 0x01141a thru 0x01799f (0x6586 bytes) 000078 1768250252.873321 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000079 1768250253.144742 mapread 0x034f6b thru 0x03f3ba (0xa450 bytes) 000080 1768250253.248052 trunc from 0x040000 to 0x00a8e9 (0x35716 bytes) 000081 1768250253.299819 mapwrite 0x02978a thru 0x032be0 (0x9457 bytes) 000082 1768250253.444494 write 0x027250 thru 0x02b96f (0x4720 bytes) 000084 1768250253.448702 write 0x01906d thru 0x021a90 (0x8a24 bytes) 000085 1768250253.468968 read 0x02e35f thru 0x032be0 (0x4882 bytes) 000086 1768250253.470727 write 0x023e9e thru 0x02adf4 (0x6f57 bytes) 000087 1768250253.473689 mapwrite 0x016efc thru 0x021bbc (0xacc1 bytes) 000088 1768250253.531315 mapwrite 0x02edc3 thru 0x02fb59 (0xd97 bytes) 000089 1768250253.580185 write 0x01bd4c thru 0x028a96 (0xcd4b bytes) 000092 1768250253.582674 mapread 0x017314 thru 0x019339 (0x2026 bytes) 000094 1768250253.590330 trunc from 0x032be1 to 0x01f19c (0x13a44 bytes) 000097 1768250253.611238 mapwrite 0x02e940 thru 0x03b597 (0xcc58 bytes) 000099 1768250253.703211 trunc from 0x03b598 to 0x022465 (0x19132 bytes) 000100 1768250253.717736 mapread 0x00ef00 thru 0x01db7e (0xec7f bytes) 000101 1768250253.727559 mapwrite 0x02b237 thru 0x03652a (0xb2f4 bytes) 000103 1768250253.803627 trunc from 0x03652b to 0x00cb37 (0x299f3 bytes) 000104 1768250253.821241 mapwrite 0x00504b thru 0x012801 (0xd7b7 bytes) 000106 1768250253.901460 read 0x00c49b thru 0x012801 (0x6367 bytes) 000107 1768250253.902239 write 0x01877a thru 0x024555 (0xbddc bytes) 000108 1768250253.905253 write 0x0069bf thru 0x00a5fb (0x3c3d bytes) 000109 1768250253.911166 MIRROR_EXTEND to 2 mirrors 000110 1768250255.031842 write 0x00d4d3 thru 0x00d8d9 (0x407 bytes) truncating to largest ever: 0x35c6f 000112 1768250255.222624 trunc from 0x024556 to 0x035c6e (0x11719 bytes) 000113 1768250255.279524 trunc from 0x035c6f to 0x021fbb (0x13cb3 bytes) 000114 1768250255.320747 mapwrite 0x017202 thru 0x01e552 (0x7351 bytes) 000115 1768250255.510770 write 0x02f5d2 thru 0x03a51f (0xaf4e bytes) 000116 1768250255.514773 trunc from 0x03a520 to 0x020605 (0x19f1a bytes) 000118 1768250255.569245 write 0x03710e thru 0x03a204 (0x30f7 bytes) truncating to largest ever: 0x3d4cb 000119 1768250255.576958 trunc from 0x03a205 to 0x03d4ca (0x32c6 bytes) 000120 1768250255.628003 trunc from 0x03d4cb to 0x024f05 (0x185c5 bytes) 000121 1768250255.673604 mapwrite 0x031b1d thru 0x036bc9 (0x50ad bytes) 000122 1768250255.806296 mapread 0x01a0be thru 0x01b0ef (0x1032 bytes) 000123 1768250255.807166 mapread 0x01bf0f thru 0x02a450 (0xe542 bytes) 000124 1768250255.837412 MIRROR_RESYNC 2 mirrors 000125 1768250256.683666 read 0x00773b thru 0x0112e6 (0x9bac bytes) 000126 1768250256.700575 mapwrite 0x03fb60 thru 0x03ffff (0x4a0 bytes) 000127 1768250256.932165 trunc from 0x040000 to 0x00de8b (0x32174 bytes) 000128 1768250256.970911 mapread 0x00acdb thru 0x00de8b (0x31b1 bytes) 000129 1768250257.000046 mapwrite 0x00b96b thru 0x010d63 (0x53f9 bytes) 000131 1768250257.090674 mapwrite 0x002559 thru 0x00393e (0x13e6 bytes) 000133 1768250257.150762 write 0x011048 thru 0x011a77 (0xa30 bytes) 000134 1768250257.151676 trunc from 0x011a78 to 0x03b633 (0x29bbc bytes) 000135 1768250257.178112 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b634): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b634): Mirror 2: 0x80f62ea (stale) CRC-32 checksum value for chunk [0, 0x3b634): Mirror 1: 0xaf798618 Mirror 3: 0xaf798618 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b634: skipped 000141 1768250258.121869 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b634): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b634): Mirror 2: 0x80f62ea (stale) CRC-32 checksum value for chunk [0, 0x3b634): Mirror 1: 0xaf798618 Mirror 3: 0xaf798618 Mirror 4: 0xaf798618 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b634: skipped 000143 1768250259.465979 write 0x00636c thru 0x00a651 (0x42e6 bytes) 000145 1768250259.569150 mapread 0x00f49c thru 0x015a07 (0x656c bytes) 000146 1768250259.580592 read 0x02fcfd thru 0x036d39 (0x703d bytes) 000148 1768250259.591644 trunc from 0x03b634 to 0x011eda (0x29759 bytes) 000150 1768250259.619708 mapwrite 0x03180d thru 0x03d504 (0xbcf8 bytes) 000151 1768250259.769709 read 0x014197 thru 0x0198a3 (0x570d bytes) 000152 1768250259.786447 mapread 0x0005e3 thru 0x008e6d (0x888b bytes) 000153 1768250259.787613 read 0x025144 thru 0x02bd74 (0x6c31 bytes) 000154 1768250259.790228 mapread 0x013e3a thru 0x02159f (0xd766 bytes) 000155 1768250259.791952 mapread 0x009bdb thru 0x015f0e (0xc334 bytes) 000156 1768250259.798072 write 0x03f0a4 thru 0x03ffff (0xf5c bytes) 000157 1768250259.810648 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9455e708 (stale) Mirror 4: 0x256a1788 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9455e708 (stale) Mirror 4: 0x256a1788 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000158 1768250260.410281 mapwrite 0x008e6c thru 0x00de06 (0x4f9b bytes) 000166 1768250260.562490 MIRROR_RESYNC 3 mirrors 000167 1768250261.642637 read 0x03220c thru 0x034574 (0x2369 bytes) 000169 1768250261.652154 trunc from 0x040000 to 0x012866 (0x2d799 bytes) 000171 1768250261.740851 read 0x008a7a thru 0x010878 (0x7dff bytes) 000173 1768250261.752273 mapwrite 0x039ead thru 0x03ffff (0x6153 bytes) 000174 1768250261.840028 mapwrite 0x0325a4 thru 0x0349da (0x2437 bytes) 000175 1768250261.893446 read 0x02def6 thru 0x036191 (0x829c bytes) 000178 1768250261.895155 mapwrite 0x002414 thru 0x00b1e8 (0x8dd5 bytes) 000179 1768250261.976336 write 0x01bffe thru 0x01f54f (0x3552 bytes) 000180 1768250261.978445 mapwrite 0x01b1df thru 0x01d904 (0x2726 bytes) 000185 1768250262.035575 read 0x035bf3 thru 0x03ffff (0xa40d bytes) 000188 1768250262.057352 read 0x03349a thru 0x0372c7 (0x3e2e bytes) 000193 1768250262.062149 read 0x009e03 thru 0x01817a (0xe378 bytes) 000195 1768250262.064827 mapwrite 0x000daa thru 0x00b446 (0xa69d bytes) 000196 1768250262.135852 mapwrite 0x015c05 thru 0x022530 (0xc92c bytes) 000199 1768250262.269918 trunc from 0x040000 to 0x02e6e2 (0x1191d bytes) 000200 1768250262.292737 write 0x03139e thru 0x03af70 (0x9bd3 bytes) 000201 1768250262.295834 read 0x0286d9 thru 0x0342e2 (0xbc0a bytes) 000203 1768250262.306180 write 0x0108d9 thru 0x0203c9 (0xfaf1 bytes) 000204 1768250262.307569 mapwrite 0x03ed2b thru 0x03ffff (0x12d5 bytes) 000206 1768250262.514087 trunc from 0x040000 to 0x031e8b (0xe174 bytes) 000207 1768250262.567351 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e8c): Mirror 2: 0xcdee8636 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e8c): Mirror 2: 0xcdee8636 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e8c: skipped 000208 1768250262.949024 mapread 0x0116a8 thru 0x012cb5 (0x160e bytes) 000209 1768250262.967516 write 0x03d7d8 thru 0x03ffff (0x2828 bytes) 000210 1768250262.969576 mapwrite 0x0024f6 thru 0x00539e (0x2ea9 bytes) 000211 1768250263.075645 mapwrite 0x0356da thru 0x037e9f (0x27c6 bytes) 000212 1768250263.139598 read 0x01f25f thru 0x02dfff (0xeda1 bytes) 000214 1768250263.149662 mapwrite 0x021c86 thru 0x030746 (0xeac1 bytes) 000215 1768250263.206334 mapwrite 0x03c192 thru 0x03ffff (0x3e6e bytes) 000216 1768250263.254178 trunc from 0x040000 to 0x002f30 (0x3d0cf bytes) 000217 1768250263.270676 read 0x000bbe thru 0x002f30 (0x2373 bytes) 000218 1768250263.284034 read 0x000f78 thru 0x002f30 (0x1fb9 bytes) 000221 1768250263.285310 write 0x00ed0e thru 0x00f438 (0x72b bytes) 000222 1768250263.287885 mapread 0x00045f thru 0x006db4 (0x6956 bytes) 000224 1768250263.301718 write 0x026cf0 thru 0x02d795 (0x6aa6 bytes) 000225 1768250263.317186 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000227 1768250263.554462 read 0x02415f thru 0x02d795 (0x9637 bytes) 000229 1768250263.579159 write 0x03e887 thru 0x03ffff (0x1779 bytes) 000232 1768250263.582556 read 0x028abf thru 0x02f78f (0x6cd1 bytes) 000233 1768250263.601559 trunc from 0x040000 to 0x02cfe5 (0x1301a bytes) 000235 1768250263.640451 read 0x028ecd thru 0x02cfe5 (0x4119 bytes) 000236 1768250263.643520 mapwrite 0x03461e thru 0x03a651 (0x6034 bytes) 000238 1768250263.867935 trunc from 0x03a652 to 0x0354ba (0x5197 bytes) 000240 1768250263.911973 write 0x000180 thru 0x007cb9 (0x7b3a bytes) 000242 1768250263.918802 trunc from 0x0354bb to 0x01e679 (0x16e41 bytes) 000245 1768250263.995639 read 0x001047 thru 0x0075bd (0x6577 bytes) 000246 1768250263.999430 trunc from 0x01e67a to 0x0320fb (0x13a82 bytes) 000248 1768250264.042461 mapwrite 0x01467c thru 0x02123e (0xcbc3 bytes) 000249 1768250264.180797 write 0x01c27c thru 0x02c10a (0xfe8f bytes) 000251 1768250264.189921 write 0x03a176 thru 0x03b99c (0x1827 bytes) 000252 1768250264.191820 read 0x0240c0 thru 0x02dd95 (0x9cd6 bytes) 000253 1768250264.196451 mapwrite 0x03a964 thru 0x03ffff (0x569c bytes) 000256 1768250264.355812 MIRROR_EXTEND to 2 mirrors 000257 1768250264.842005 mapwrite 0x021b29 thru 0x029ce3 (0x81bb bytes) 000258 1768250265.046856 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xabb2bea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeb3afa18 Mirror 3: 0xeb3afa18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000259 1768250266.313646 read 0x00d84f thru 0x0151ba (0x796c bytes) 000260 1768250266.326209 mapread 0x02ce38 thru 0x036289 (0x9452 bytes) 000261 1768250266.375434 mapwrite 0x03d495 thru 0x03ffff (0x2b6b bytes) 000262 1768250266.622566 mapread 0x00ecaf thru 0x0186ac (0x99fe bytes) 000263 1768250266.669026 write 0x03a9c5 thru 0x03ffff (0x563b bytes) 000266 1768250266.677502 mapread 0x01692c thru 0x01ac33 (0x4308 bytes) 000267 1768250266.683443 mapwrite 0x02555f thru 0x03083b (0xb2dd bytes) 000270 1768250266.850968 mapwrite 0x003ed9 thru 0x00536c (0x1494 bytes) 000271 1768250266.958508 mapwrite 0x03b1d8 thru 0x03cc0e (0x1a37 bytes) 000272 1768250267.086917 mapread 0x03cb7e thru 0x03e558 (0x19db bytes) 000275 1768250267.102278 write 0x00b34a thru 0x01795d (0xc614 bytes) 000276 1768250267.105071 read 0x021099 thru 0x02ed0b (0xdc73 bytes) 000278 1768250267.106860 read 0x01d9c3 thru 0x0202bb (0x28f9 bytes) 000279 1768250267.110805 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeb3afa18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeb3afa18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1768250267.469532 mapread 0x02f611 thru 0x03b234 (0xbc24 bytes) 000281 1768250267.519979 read 0x038ea5 thru 0x03bb84 (0x2ce0 bytes) 000284 1768250267.523555 read 0x02cca3 thru 0x032ba4 (0x5f02 bytes) 000285 1768250267.531909 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeb3afa18 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf1247598 Mirror 4: 0xf1247598 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000287 1768250269.064638 mapwrite 0x0037ff thru 0x003f41 (0x743 bytes) 000288 1768250269.252337 read 0x0356b5 thru 0x03ffff (0xa94b bytes) 000290 1768250269.264988 mapwrite 0x0034aa thru 0x006858 (0x33af bytes) 000294 1768250269.342333 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf1247598 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf1247598 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000296 1768250269.985073 write 0x00b9db thru 0x016546 (0xab6c bytes) 000297 1768250270.010703 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000302 1768250270.193236 mapwrite 0x014437 thru 0x014553 (0x11d bytes) 000307 1768250270.248087 mapread 0x030660 thru 0x032d97 (0x2738 bytes) 000308 1768250270.267784 mapread 0x01df07 thru 0x020c82 (0x2d7c bytes) 000311 1768250270.285085 write 0x036a17 thru 0x03ffff (0x95e9 bytes) 000312 1768250270.288847 mapwrite 0x02db9b thru 0x03acb2 (0xd118 bytes) 000315 1768250270.383603 mapread 0x01a26b thru 0x01b756 (0x14ec bytes) 000318 1768250270.387203 read 0x0374b9 thru 0x037fcd (0xb15 bytes) 000319 1768250270.388862 read 0x03edd2 thru 0x03ffff (0x122e bytes) 000320 1768250270.395848 mapwrite 0x0229bd thru 0x031622 (0xec66 bytes) 000321 1768250270.497331 mapwrite 0x035cc6 thru 0x03ffff (0xa33a bytes) 000322 1768250270.556838 mapwrite 0x00d56d thru 0x01a8ca (0xd35e bytes) 000324 1768250270.623192 mapread 0x020453 thru 0x0221b7 (0x1d65 bytes) 000327 1768250270.626555 mapwrite 0x03457b thru 0x038891 (0x4317 bytes) 000329 1768250270.706119 mapwrite 0x028e2f thru 0x02c45c (0x362e bytes) 000330 1768250270.747808 trunc from 0x040000 to 0x005edc (0x3a123 bytes) 000334 1768250270.761508 mapread 0x0025fb thru 0x004568 (0x1f6e bytes) 000336 1768250270.766022 MIRROR_EXTEND to 2 mirrors 000337 1768250271.959445 trunc from 0x005edd to 0x02ef5e (0x29082 bytes) 000338 1768250272.128231 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000339 1768250272.426338 mapread 0x012608 thru 0x01920b (0x6c04 bytes) 000342 1768250272.492538 trunc from 0x02ef5f to 0x01e7a7 (0x107b7 bytes) 000345 1768250272.544337 mapwrite 0x00922b thru 0x0128a5 (0x967b bytes) 000346 1768250272.724736 write 0x03bed0 thru 0x03e547 (0x2678 bytes) 000347 1768250272.729899 mapwrite 0x03d05a thru 0x03ffff (0x2fa6 bytes) 000350 1768250272.891082 read 0x02a745 thru 0x03540d (0xacc9 bytes) 000351 1768250272.916085 read 0x001406 thru 0x0089af (0x75aa bytes) 000353 1768250272.940489 read 0x037563 thru 0x03ffff (0x8a9d bytes) 000355 1768250272.949595 trunc from 0x040000 to 0x0093a2 (0x36c5d bytes) 000357 1768250272.995021 mapwrite 0x028948 thru 0x035749 (0xce02 bytes) 000358 1768250273.375787 mapread 0x000567 thru 0x006831 (0x62cb bytes) 000359 1768250273.388851 mapread 0x02f251 thru 0x035749 (0x64f9 bytes) 000361 1768250273.402411 write 0x022914 thru 0x025192 (0x287f bytes) 000362 1768250273.407761 write 0x033dfb thru 0x03ffff (0xc205 bytes) 000363 1768250273.417570 write 0x00da6c thru 0x01977c (0xbd11 bytes) 000364 1768250273.424563 write 0x00a9b2 thru 0x00efd9 (0x4628 bytes) 000369 1768250273.436113 read 0x03d357 thru 0x03ffff (0x2ca9 bytes) 000370 1768250273.436761 mapwrite 0x036ffe thru 0x03ffff (0x9002 bytes) 000372 1768250273.675261 write 0x00cbc4 thru 0x01998b (0xcdc8 bytes) 000373 1768250273.689951 MIRROR_EXTEND to 2 mirrors 000374 1768250274.792911 mapwrite 0x03f4e0 thru 0x03ffff (0xb20 bytes) 000375 1768250274.994339 write 0x00744f thru 0x00c646 (0x51f8 bytes) 000376 1768250275.016137 read 0x0188ad thru 0x024290 (0xb9e4 bytes) 000378 1768250275.031383 trunc from 0x040000 to 0x00594b (0x3a6b4 bytes) 000380 1768250275.070541 read 0x001803 thru 0x00594b (0x4149 bytes) 000382 1768250275.077176 trunc from 0x00594c to 0x000a95 (0x4eb6 bytes) 000383 1768250275.092293 trunc from 0x000a96 to 0x011110 (0x1067b bytes) 000384 1768250275.116234 MIRROR_RESYNC 2 mirrors 000385 1768250276.294432 mapwrite 0x011a15 thru 0x01af08 (0x94f4 bytes) 000386 1768250276.711270 write 0x008cf8 thru 0x00b96a (0x2c73 bytes) 000389 1768250276.733583 mapread 0x012859 thru 0x01420a (0x19b2 bytes) 000390 1768250276.744652 mapwrite 0x038d6c thru 0x03ffff (0x7294 bytes) 000391 1768250276.967785 mapread 0x032a27 thru 0x03ffff (0xd5d9 bytes) 000393 1768250276.974743 mapread 0x0340bb thru 0x03e7cc (0xa712 bytes) 000394 1768250276.980262 mapread 0x01f453 thru 0x023971 (0x451f bytes) 000395 1768250276.990482 trunc from 0x040000 to 0x01d5a7 (0x22a58 bytes) 000397 1768250277.013945 read 0x013f55 thru 0x01c9fc (0x8aa8 bytes) 000398 1768250277.014651 trunc from 0x01d5a8 to 0x03b8d5 (0x1e32e bytes) 000399 1768250277.047531 MIRROR_RESYNC 2 mirrors 000400 1768250278.376818 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000401 1768250278.695101 trunc from 0x03b8d6 to 0x00cf1e (0x2e9b7 bytes) 000403 1768250278.799952 MIRROR_EXTEND to 2 mirrors 000404 1768250280.508343 trunc from 0x00cf1f to 0x01604b (0x912d bytes) 000406 1768250280.706647 read 0x00d422 thru 0x01176d (0x434c bytes) 000408 1768250280.736805 MIRROR_RESYNC 2 mirrors 000411 1768250282.311679 trunc from 0x01604c to 0x01ab9b (0x4b50 bytes) 000412 1768250282.465867 mapread 0x00f335 thru 0x01733d (0x8009 bytes) 000413 1768250282.501518 read 0x014e55 thru 0x01696b (0x1b17 bytes) 000414 1768250282.503275 trunc from 0x01ab9c to 0x020b83 (0x5fe8 bytes) 000415 1768250282.557674 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000419 1768250282.774516 write 0x00eae0 thru 0x01c17b (0xd69c bytes) 000420 1768250282.788961 read 0x002201 thru 0x0114f5 (0xf2f5 bytes) 000422 1768250282.802909 write 0x03416c thru 0x039f5d (0x5df2 bytes) 000425 1768250282.809728 mapwrite 0x01672f thru 0x01f770 (0x9042 bytes) 000426 1768250282.877558 write 0x034d01 thru 0x03798f (0x2c8f bytes) 000430 1768250282.883129 trunc from 0x039f5e to 0x03cd0f (0x2db2 bytes) 000431 1768250282.902228 mapread 0x0038d5 thru 0x0121aa (0xe8d6 bytes) 000433 1768250282.903598 trunc from 0x03cd10 to 0x028715 (0x145fa bytes) 000435 1768250282.915913 mapwrite 0x01c230 thru 0x025988 (0x9759 bytes) 000439 1768250282.971745 mapread 0x0281bc thru 0x028715 (0x55a bytes) 000441 1768250282.979014 mapwrite 0x03594d thru 0x03d773 (0x7e27 bytes) 000443 1768250283.059412 trunc from 0x03d774 to 0x00bac0 (0x31cb3 bytes) 000444 1768250283.081324 mapwrite 0x008faa thru 0x017e2a (0xee81 bytes) truncating to largest ever: 0x3e46a 000446 1768250283.186345 trunc from 0x017e2b to 0x03e469 (0x2663f bytes) 000447 1768250283.206893 trunc from 0x03e46a to 0x0097eb (0x34c7e bytes) 000448 1768250283.222865 read 0x006116 thru 0x0097eb (0x36d6 bytes) 000453 1768250283.225399 write 0x01304d thru 0x01aa19 (0x79cd bytes) 000461 1768250283.236878 mapread 0x00794e thru 0x00efb6 (0x7669 bytes) 000462 1768250283.246396 trunc from 0x01aa1a to 0x00c739 (0xe2e0 bytes) 000471 1768250283.270431 write 0x0042a4 thru 0x00b9bc (0x7719 bytes) 000472 1768250283.272878 read 0x00bc79 thru 0x00c739 (0xac1 bytes) 000473 1768250283.274103 read 0x0072d4 thru 0x00c739 (0x5466 bytes) 000475 1768250283.276841 mapwrite 0x011c41 thru 0x012c3c (0xffc bytes) 000476 1768250283.360642 mapwrite 0x026dac thru 0x02bb3d (0x4d92 bytes) 000480 1768250283.476780 write 0x01987e thru 0x01cf38 (0x36bb bytes) 000481 1768250283.478070 mapwrite 0x00a6f8 thru 0x014af0 (0xa3f9 bytes) 000482 1768250283.569737 write 0x01da47 thru 0x022d11 (0x52cb bytes) 000484 1768250283.572306 read 0x00435d thru 0x00f95c (0xb600 bytes) 000486 1768250283.575329 mapread 0x016862 thru 0x0188f0 (0x208f bytes) 000487 1768250283.577106 mapread 0x0044fb thru 0x011af8 (0xd5fe bytes) 000488 1768250283.579790 read 0x00b904 thru 0x019acb (0xe1c8 bytes) 000489 1768250283.581727 mapread 0x013726 thru 0x0222a8 (0xeb83 bytes) 000492 1768250283.591431 read 0x01fe4b thru 0x029637 (0x97ed bytes) 000493 1768250283.594041 write 0x003492 thru 0x0131e8 (0xfd57 bytes) 000495 1768250283.600006 mapwrite 0x0108c4 thru 0x01e356 (0xda93 bytes) 000499 1768250283.772081 mapread 0x0276b0 thru 0x029a82 (0x23d3 bytes) 000500 1768250283.775092 mapread 0x01a17c thru 0x029f94 (0xfe19 bytes) 000501 1768250283.785616 mapwrite 0x0201b0 thru 0x022d6c (0x2bbd bytes) 000504 1768250283.950900 read 0x00eda2 thru 0x01d71f (0xe97e bytes) 000505 1768250283.956705 mapread 0x006da3 thru 0x0090f2 (0x2350 bytes) 000507 1768250283.973567 read 0x005928 thru 0x009c83 (0x435c bytes) 000508 1768250283.976313 trunc from 0x02bb3e to 0x01c37d (0xf7c0 bytes) 000509 1768250284.027097 read 0x000f3f thru 0x00579f (0x4861 bytes) 000510 1768250284.028612 trunc from 0x01c37e to 0x025c57 (0x98da bytes) 000512 1768250284.062271 write 0x0237f5 thru 0x03171c (0xdf28 bytes) 000513 1768250284.111505 MIRROR_EXTEND to 2 mirrors 000514 1768250285.439612 trunc from 0x03171d to 0x017660 (0x1a0bc bytes) 000515 1768250285.547548 write 0x003457 thru 0x011dd0 (0xe97a bytes) 000516 1768250285.556109 trunc from 0x017661 to 0x029915 (0x122b5 bytes) 000517 1768250285.578061 read 0x015973 thru 0x016849 (0xed7 bytes) 000518 1768250285.590214 MIRROR_RESYNC 2 mirrors 000519 1768250286.487487 mapread 0x009ea6 thru 0x0194aa (0xf605 bytes) 000521 1768250286.571286 trunc from 0x029916 to 0x034e6f (0xb55a bytes) 000522 1768250286.714982 read 0x00b30d thru 0x0137df (0x84d3 bytes) 000523 1768250286.735135 mapwrite 0x032ad0 thru 0x03744d (0x497e bytes) 000524 1768250286.893401 trunc from 0x03744e to 0x025a55 (0x119f8 bytes) 000526 1768250286.937418 MIRROR_RESYNC 2 mirrors 000528 1768250287.665118 read 0x00cda3 thru 0x016d8a (0x9fe8 bytes) 000531 1768250287.673740 trunc from 0x025a56 to 0x031c6a (0xc215 bytes) 000534 1768250287.774495 read 0x03079b thru 0x031c6a (0x14d0 bytes) 000536 1768250287.794252 mapwrite 0x00f543 thru 0x0179f0 (0x84ae bytes) 000537 1768250287.873978 MIRROR_RESYNC 2 mirrors 000538 1768250288.606170 write 0x03ebb1 thru 0x03ffff (0x144f bytes) 000540 1768250288.750289 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe2a3fca3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd03bb85d Mirror 3: 0xd03bb85d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000541 1768250289.566251 mapwrite 0x026e64 thru 0x02adfe (0x3f9b bytes) 000543 1768250289.735079 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe2a3fca3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe2a3fca3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000545 1768250290.313812 mapwrite 0x0150d2 thru 0x01cea3 (0x7dd2 bytes) 000546 1768250290.400668 read 0x03a1ed thru 0x03ffff (0x5e13 bytes) 000548 1768250290.412856 write 0x011fdf thru 0x0168d3 (0x48f5 bytes) 000550 1768250290.414467 write 0x008e19 thru 0x00f095 (0x627d bytes) 000555 1768250290.450214 write 0x019d22 thru 0x024dcc (0xb0ab bytes) 000557 1768250290.480752 mapwrite 0x031f18 thru 0x03cb2e (0xac17 bytes) 000558 1768250290.590604 read 0x0229df thru 0x029454 (0x6a76 bytes) 000560 1768250290.600210 read 0x01ab40 thru 0x020542 (0x5a03 bytes) 000561 1768250290.601085 read 0x0324d5 thru 0x034c1c (0x2748 bytes) 000562 1768250290.601790 trunc from 0x040000 to 0x01fe94 (0x2016b bytes) 000563 1768250290.615782 read 0x013b35 thru 0x01d35b (0x9827 bytes) 000565 1768250290.619619 trunc from 0x01fe95 to 0x006ae4 (0x193b0 bytes) 000566 1768250290.644029 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000567 1768250290.962188 read 0x0065cc thru 0x006ae4 (0x519 bytes) 000568 1768250290.977056 read 0x0033cc thru 0x006ae4 (0x3719 bytes) 000571 1768250290.987320 read 0x0045c6 thru 0x006ae4 (0x251f bytes) 000576 1768250290.993936 trunc from 0x006ae5 to 0x005030 (0x1ab4 bytes) 000577 1768250291.009711 read 0x00159f thru 0x005030 (0x3a92 bytes) 000578 1768250291.010585 write 0x035d90 thru 0x035f9e (0x20f bytes) 000579 1768250291.012104 mapread 0x025598 thru 0x035223 (0xfc8c bytes) 000584 1768250291.021028 mapread 0x0295fc thru 0x02af6d (0x1972 bytes) 000587 1768250291.023413 mapwrite 0x029ac9 thru 0x033ff7 (0xa52f bytes) 000593 1768250291.068638 mapread 0x02d539 thru 0x03118d (0x3c55 bytes) 000594 1768250291.069502 trunc from 0x035f9f to 0x031ac2 (0x44dc bytes) 000598 1768250291.095225 trunc from 0x031ac3 to 0x03bb28 (0xa066 bytes) 000600 1768250291.117546 trunc from 0x03bb29 to 0x017bbf (0x23f69 bytes) 000603 1768250291.139026 read 0x00a1da thru 0x00f0cd (0x4ef4 bytes) 000604 1768250291.139763 mapread 0x010c1e thru 0x01600b (0x53ee bytes) 000605 1768250291.140669 read 0x012b86 thru 0x017bbf (0x503a bytes) 000606 1768250291.142594 trunc from 0x017bc0 to 0x03db83 (0x25fc4 bytes) 000607 1768250291.156085 mapwrite 0x01fb0f thru 0x022ccf (0x31c1 bytes) 000608 1768250291.241624 write 0x02022e thru 0x0208e8 (0x6bb bytes) 000609 1768250291.243278 mapread 0x0086ff thru 0x0167b7 (0xe0b9 bytes) 000610 1768250291.245177 write 0x01ac4a thru 0x02485a (0x9c11 bytes) 000611 1768250291.247138 write 0x0193a2 thru 0x01f809 (0x6468 bytes) 000614 1768250291.250208 mapwrite 0x01a9ab thru 0x02725c (0xc8b2 bytes) 000615 1768250291.310944 write 0x02058f thru 0x021464 (0xed6 bytes) 000616 1768250291.312773 mapread 0x020510 thru 0x023887 (0x3378 bytes) 000618 1768250291.318253 MIRROR_EXTEND to 2 mirrors 000620 1768250292.146350 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000621 1768250292.286743 MIRROR_EXTEND to 2 mirrors 000622 1768250293.120981 trunc from 0x03db84 to 0x017ece (0x25cb5 bytes) 000623 1768250293.238434 mapread 0x0053f5 thru 0x00ece2 (0x98ee bytes) 000624 1768250293.278122 mapread 0x006976 thru 0x0117c4 (0xae4f bytes) 000625 1768250293.293415 read 0x015385 thru 0x017ece (0x2b4a bytes) 000626 1768250293.294820 mapread 0x0152a7 thru 0x017ece (0x2c28 bytes) 000629 1768250293.303429 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x17ecf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ecf): Mirror 2: 0xb50b5674 (stale) CRC-32 checksum value for chunk [0, 0x17ecf): Mirror 1: 0xb50b5674 Mirror 3: 0xb50b5674 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ecf: skipped 000632 1768250294.604651 MIRROR_RESYNC 3 mirrors 000636 1768250295.690286 mapwrite 0x00bd28 thru 0x01846f (0xc748 bytes) 000641 1768250295.911596 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18470): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18470): Mirror 2: 0x7e4615ce (stale) Mirror 3: 0x7e4615ce (stale) CRC-32 checksum value for chunk [0, 0x18470): Mirror 1: 0x6d606734 Mirror 4: 0x6d606734 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18470: skipped 000643 1768250297.367616 mapread 0x0156e7 thru 0x016d13 (0x162d bytes) 000647 1768250297.490067 mapwrite 0x03ffbd thru 0x03ffff (0x43 bytes) 000652 1768250298.005141 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xecb41abb (stale) Mirror 4: 0xda4a5882 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xecb41abb (stale) Mirror 4: 0xda4a5882 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000655 1768250298.673158 mapread 0x0210cd thru 0x02ea42 (0xd976 bytes) 000656 1768250298.771905 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xecb41abb (stale) Mirror 4: 0xda4a5882 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f8530b4 Mirror 5: 0x6f8530b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000657 1768250300.716992 MIRROR_RESYNC 4 mirrors 000658 1768250302.026409 trunc from 0x040000 to 0x017b05 (0x284fa bytes) 000660 1768250302.163610 mapread 0x00aee6 thru 0x01370a (0x8825 bytes) 000664 1768250302.200321 read 0x0143a8 thru 0x017b05 (0x375e bytes) 000665 1768250302.208952 read 0x011557 thru 0x017201 (0x5cab bytes) 000666 1768250302.218504 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17b06): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b06): Mirror 2: 0x1ac3c7a0 (stale) Mirror 4: 0x1ac3c7a0 (stale) Mirror 5: 0x1ac3c7a0 (stale) CRC-32 checksum value for chunk [0, 0x17b06): Mirror 1: 0x1ac3c7a0 Mirror 6: 0x1ac3c7a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b06: skipped 000667 1768250303.623342 trunc from 0x017b06 to 0x001681 (0x16484 bytes) 000669 1768250303.755834 read 0x001315 thru 0x001681 (0x36d bytes) 000670 1768250303.763971 read 0x00000f thru 0x001681 (0x1673 bytes) 000671 1768250303.791999 read 0x0013cb thru 0x001681 (0x2b7 bytes) 000672 1768250303.806881 mapread 0x00015c thru 0x001681 (0x1526 bytes) 000674 1768250303.819331 MIRROR_RESYNC 5 mirrors 000676 1768250305.402116 read 0x000a8d thru 0x001681 (0xbf5 bytes) 000678 1768250305.413085 trunc from 0x001682 to 0x03cf77 (0x3b8f6 bytes) 000679 1768250305.511177 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf78): Mirror 2: 0xcf46ffae (stale) Mirror 4: 0xcf46ffae (stale) Mirror 6: 0xcf46ffae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf78): Mirror 2: 0xcf46ffae (stale) Mirror 4: 0xcf46ffae (stale) Mirror 6: 0xcf46ffae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf78: skipped 000680 1768250305.886782 write 0x02f0c9 thru 0x031d6d (0x2ca5 bytes) 000682 1768250305.914892 mapwrite 0x005681 thru 0x0106f0 (0xb070 bytes) 000684 1768250306.062177 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf78): Mirror 2: 0xcf46ffae (stale) Mirror 4: 0xcf46ffae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf78): Mirror 2: 0xcf46ffae (stale) Mirror 4: 0xcf46ffae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf78: skipped 000685 1768250306.456034 write 0x036f4b thru 0x03ffff (0x90b5 bytes) 000687 1768250306.467862 mapread 0x02133d thru 0x021923 (0x5e7 bytes) 000690 1768250306.476224 read 0x0364c4 thru 0x03a181 (0x3cbe bytes) 000691 1768250306.482282 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7f1c8feb (stale) Mirror 4: 0x7f1c8feb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5926541 Mirror 5: 0xf5926541 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000692 1768250307.371094 trunc from 0x040000 to 0x014ed5 (0x2b12a bytes) 000694 1768250307.500157 mapwrite 0x020739 thru 0x020777 (0x3f bytes) 000696 1768250307.625892 write 0x030b91 thru 0x03a291 (0x9701 bytes) 000697 1768250307.639588 mapread 0x00b482 thru 0x00f8af (0x442e bytes) 000701 1768250307.712240 mapread 0x024cf7 thru 0x0291ae (0x44b8 bytes) 000702 1768250307.730647 mapwrite 0x00b054 thru 0x015f6d (0xaf1a bytes) 000706 1768250307.873419 MIRROR_RESYNC 4 mirrors 000707 1768250308.838490 read 0x002bee thru 0x00f88b (0xcc9e bytes) 000709 1768250308.848644 mapread 0x01cd36 thru 0x0204e8 (0x37b3 bytes) 000710 1768250308.877060 mapread 0x00af76 thru 0x016990 (0xba1b bytes) 000714 1768250308.898209 mapread 0x008a75 thru 0x00d8c6 (0x4e52 bytes) 000715 1768250308.898870 trunc from 0x03a292 to 0x012ecd (0x273c4 bytes) 000720 1768250309.074647 mapread 0x007ecc thru 0x00edce (0x6f03 bytes) 000722 1768250309.091164 write 0x0031cc thru 0x00b68c (0x84c1 bytes) 000725 1768250309.112246 trunc from 0x012ece to 0x01a216 (0x7349 bytes) 000727 1768250309.157312 read 0x003795 thru 0x004af3 (0x135f bytes) 000728 1768250309.158010 mapread 0x005fc6 thru 0x00c778 (0x67b3 bytes) 000729 1768250309.160677 write 0x00bb95 thru 0x011f8a (0x63f6 bytes) 000731 1768250309.179231 trunc from 0x01a217 to 0x009368 (0x10eae bytes) 000734 1768250309.207867 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9369): Mirror 2: 0xead95a7f (stale) Mirror 5: 0xead95a7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9369): Mirror 2: 0xead95a7f (stale) Mirror 5: 0xead95a7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9369: skipped 000735 1768250309.599272 mapread 0x00263e thru 0x009368 (0x6d2b bytes) 000739 1768250309.625162 write 0x01ec2d thru 0x02ce6e (0xe242 bytes) 000740 1768250309.627297 read 0x01b5e8 thru 0x025f44 (0xa95d bytes) 000741 1768250309.639887 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ce6f): Mirror 5: 0xa953c8e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce6f): Mirror 5: 0xa953c8e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce6f: skipped 000744 1768250310.068308 MIRROR_RESYNC 2 mirrors 000745 1768250311.392105 write 0x008e10 thru 0x00c514 (0x3705 bytes) 000747 1768250311.574623 mapwrite 0x021b17 thru 0x02f33b (0xd825 bytes) 000748 1768250311.816150 mapwrite 0x003ff1 thru 0x00be07 (0x7e17 bytes) 000749 1768250311.909233 mapread 0x02e092 thru 0x02f33b (0x12aa bytes) 000750 1768250311.923135 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f33c): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f33c): Mirror 5: 0x7059fb2e (stale) CRC-32 checksum value for chunk [0, 0x2f33c): Mirror 1: 0x479d44aa Mirror 6: 0x479d44aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f33c: skipped 000751 1768250312.999298 read 0x02256e thru 0x029a44 (0x74d7 bytes) 000752 1768250313.011764 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000755 1768250313.416022 trunc from 0x02f33c to 0x00f0a7 (0x20294 bytes) 000757 1768250313.545551 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf0a8): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf0a8): Mirror 6: 0x2de4024d (stale) CRC-32 checksum value for chunk [0, 0xf0a8): Mirror 1: 0x2de4024d Mirror 7: 0x2de4024d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf0a8: skipped 000759 1768250314.727954 mapread 0x0078cf thru 0x00f0a7 (0x77d9 bytes) 000760 1768250314.776687 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) 000761 1768250315.165046 read 0x00c8d1 thru 0x00dabc (0x11ec bytes) 000763 1768250315.231635 MIRROR_EXTEND to 3 mirrors 000764 1768250315.959192 read 0x00c527 thru 0x00f0a7 (0x2b81 bytes) 000765 1768250315.965018 trunc from 0x00f0a8 to 0x02d8c0 (0x1e819 bytes) 000766 1768250316.218460 read 0x01b395 thru 0x02a2af (0xef1b bytes) 000768 1768250316.249625 mapwrite 0x03633e thru 0x03ae67 (0x4b2a bytes) 000769 1768250316.349035 mapwrite 0x021ac8 thru 0x028459 (0x6992 bytes) 000771 1768250316.454885 MIRROR_RESYNC 3 mirrors 000772 1768250317.256863 read 0x018000 thru 0x0195a7 (0x15a8 bytes) 000774 1768250317.269225 read 0x032ed7 thru 0x03ae67 (0x7f91 bytes) 000776 1768250317.298913 mapread 0x021c92 thru 0x02c18e (0xa4fd bytes) 000778 1768250317.300080 trunc from 0x03ae68 to 0x029c7b (0x111ec bytes) 000779 1768250317.560908 write 0x0223b7 thru 0x026b29 (0x4773 bytes) 000783 1768250317.584711 mapread 0x022b84 thru 0x024742 (0x1bbf bytes) 000785 1768250317.586225 mapwrite 0x030751 thru 0x03ffff (0xf8af bytes) 000786 1768250317.938399 mapwrite 0x033427 thru 0x0391e2 (0x5dbc bytes) 000787 1768250318.048966 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbfdd3d86 (stale) Mirror 8: 0xbfdd3d86 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbb457a3a Mirror 9: 0xbb457a3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000788 1768250318.933919 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbfdd3d86 (stale) Mirror 8: 0xbfdd3d86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbfdd3d86 (stale) Mirror 8: 0xbfdd3d86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000790 1768250319.261416 trunc from 0x040000 to 0x00d616 (0x329e9 bytes) 000791 1768250319.293947 mapwrite 0x00c91b thru 0x00e203 (0x18e9 bytes) 000793 1768250319.440934 trunc from 0x00e204 to 0x023739 (0x15536 bytes) 000794 1768250319.476650 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2373a): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2373a): Mirror 7: 0x5aa08e0c (stale) Mirror 8: 0x5aa08e0c (stale) CRC-32 checksum value for chunk [0, 0x2373a): Mirror 1: 0x91bd75f4 Mirror 9: 0x91bd75f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2373a: skipped 000799 1768250320.472135 read 0x012e73 thru 0x01d55f (0xa6ed bytes) 000802 1768250320.513975 write 0x010179 thru 0x01bef4 (0xbd7c bytes) 000807 1768250320.663831 read 0x001753 thru 0x00ba9d (0xa34b bytes) 000809 1768250320.683778 trunc from 0x02373a to 0x022d91 (0x9a8 bytes) 000811 1768250320.741204 write 0x03182e thru 0x033aed (0x22c0 bytes) 000812 1768250320.754186 read 0x00d556 thru 0x01bd93 (0xe83e bytes) 000813 1768250320.762451 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33aee): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33aee): Mirror 7: 0xc526d8ed (stale) Mirror 8: 0xc526d8ed (stale) Mirror 9: 0x2c2239dd (stale) CRC-32 checksum value for chunk [0, 0x33aee): Mirror 1: 0x2c4fcf0a Mirror 10: 0x2c4fcf0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33aee: skipped 000815 1768250322.024308 mapread 0x012902 thru 0x01a28f (0x798e bytes) 000818 1768250322.082847 mapread 0x031e11 thru 0x033aed (0x1cdd bytes) 000820 1768250322.118861 write 0x02fa1d thru 0x033309 (0x38ed bytes) 000821 1768250322.216269 read 0x00ef5c thru 0x0110ae (0x2153 bytes) 000823 1768250322.226007 write 0x021a5b thru 0x025d7b (0x4321 bytes) 000824 1768250322.236233 read 0x01488f thru 0x0151ac (0x91e bytes) 000825 1768250322.246146 trunc from 0x033aee to 0x01905c (0x1aa91 bytes) 000826 1768250322.296997 mapread 0x00a7e1 thru 0x0106b3 (0x5ed3 bytes) 000830 1768250322.304427 mapwrite 0x01081b thru 0x01b50f (0xacf5 bytes) 000832 1768250322.469952 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1b510): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b510): Mirror 7: 0x4a4ec456 (stale) Mirror 8: 0x4a4ec456 (stale) Mirror 9: 0x1e106837 (stale) Mirror 10: 0x203f5a6d (stale) CRC-32 checksum value for chunk [0, 0x1b510): Mirror 1: 0xa20efa5e Mirror 11: 0xa20efa5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b510: skipped 000833 1768250323.385285 mapwrite 0x006ed8 thru 0x012bd0 (0xbcf9 bytes) 000836 1768250323.952146 mapwrite 0x000a77 thru 0x008af4 (0x807e bytes) 000840 1768250324.081196 trunc from 0x01b510 to 0x002b6f (0x189a0 bytes) 000841 1768250324.189031 mapread 0x001a7e thru 0x002b6f (0x10f2 bytes) 000845 1768250324.191445 read 0x000f2a thru 0x002b6f (0x1c46 bytes) 000846 1768250324.199086 read 0x001df0 thru 0x002b6f (0xd80 bytes) 000848 1768250324.201025 write 0x021aee thru 0x029a10 (0x7f23 bytes) 000850 1768250324.212330 mapread 0x019c43 thru 0x0252e1 (0xb69f bytes) 000853 1768250324.246893 read 0x01970c thru 0x022a00 (0x92f5 bytes) 000856 1768250324.248715 mapread 0x00e999 thru 0x013e26 (0x548e bytes) 000858 1768250324.257083 write 0x019ca6 thru 0x023e0e (0xa169 bytes) 000859 1768250324.259228 mapwrite 0x01b328 thru 0x022b56 (0x782f bytes) 000860 1768250324.404966 write 0x01eaff thru 0x026940 (0x7e42 bytes) 000862 1768250324.410444 mapread 0x006bed thru 0x014b1b (0xdf2f bytes) 000863 1768250324.412098 trunc from 0x029a11 to 0x005c1e (0x23df2 bytes) 000865 1768250324.434052 mapwrite 0x01b98c thru 0x022581 (0x6bf6 bytes) 000866 1768250324.575782 read 0x013047 thru 0x022001 (0xefbb bytes) 000867 1768250324.576599 write 0x03d22d thru 0x03ffff (0x2dd3 bytes) 000869 1768250324.578147 mapread 0x03e2c6 thru 0x03f730 (0x146b bytes) 000870 1768250324.581040 mapwrite 0x014f54 thru 0x01b725 (0x67d2 bytes) 000871 1768250324.663426 mapwrite 0x00fea0 thru 0x0103dd (0x53e bytes) 000872 1768250324.715256 mapwrite 0x000a47 thru 0x000eae (0x468 bytes) 000874 1768250324.761061 read 0x02dde7 thru 0x02f943 (0x1b5d bytes) 000875 1768250324.774065 mapread 0x000e15 thru 0x0010c7 (0x2b3 bytes) 000876 1768250324.775733 read 0x012988 thru 0x01dbd9 (0xb252 bytes) 000877 1768250324.777769 trunc from 0x040000 to 0x00fe96 (0x30169 bytes) 000878 1768250324.793940 mapwrite 0x03d089 thru 0x03ffff (0x2f77 bytes) 000879 1768250324.866006 trunc from 0x040000 to 0x01ba6d (0x24592 bytes) 000880 1768250324.882162 mapwrite 0x01c9f7 thru 0x02626d (0x9877 bytes) 000883 1768250325.032870 trunc from 0x02626e to 0x013fe3 (0x1228a bytes) 000884 1768250325.063836 read 0x000299 thru 0x00fd60 (0xfac8 bytes) 000885 1768250325.064916 trunc from 0x013fe4 to 0x0305be (0x1c5db bytes) 000887 1768250325.096148 mapwrite 0x014c90 thru 0x01b993 (0x6d04 bytes) 000888 1768250325.189311 mapwrite 0x01e12f thru 0x02a01d (0xbeef bytes) 000889 1768250325.266325 trunc from 0x0305bf to 0x017435 (0x19189 bytes) 000890 1768250325.288697 write 0x00dfbc thru 0x0164ce (0x8513 bytes) 000894 1768250325.294815 trunc from 0x017436 to 0x00476a (0x12ccb bytes) 000895 1768250325.318012 mapread 0x003941 thru 0x00476a (0xe2a bytes) 000898 1768250325.323001 mapread 0x0021d2 thru 0x00476a (0x2599 bytes) 000899 1768250325.324154 mapread 0x001125 thru 0x0042c5 (0x31a1 bytes) 000900 1768250325.326586 mapwrite 0x03eed0 thru 0x03ffff (0x1130 bytes) 000901 1768250325.539125 write 0x009e18 thru 0x00a658 (0x841 bytes) 000904 1768250325.576621 read 0x03f1ce thru 0x03ffff (0xe32 bytes) 000905 1768250325.577262 mapwrite 0x02982b thru 0x033075 (0x984b bytes) 000906 1768250325.696834 mapread 0x00cb1d thru 0x017464 (0xa948 bytes) 000908 1768250325.697890 mapread 0x01823d thru 0x01deca (0x5c8e bytes) 000909 1768250325.703999 MIRROR_RESYNC 6 mirrors 000910 1768250327.486271 MIRROR_EXTEND to 7 mirrors 000911 1768250328.623180 mapread 0x01b71a thru 0x028594 (0xce7b bytes) 000914 1768250328.649519 read 0x037b18 thru 0x03b690 (0x3b79 bytes) 000916 1768250328.656790 trunc from 0x040000 to 0x03b5f0 (0x4a0f bytes) 000918 1768250328.749189 mapwrite 0x02265b thru 0x022c82 (0x628 bytes) 000919 1768250328.909507 write 0x019e5b thru 0x029949 (0xfaef bytes) 000920 1768250328.929031 trunc from 0x03b5f1 to 0x0344fa (0x70f6 bytes) 000921 1768250328.967127 mapwrite 0x0059d0 thru 0x012fe1 (0xd612 bytes) 000922 1768250329.148826 read 0x00ac2c thru 0x010547 (0x591c bytes) 000923 1768250329.152283 read 0x013aea thru 0x01ab84 (0x709b bytes) 000924 1768250329.154710 mapwrite 0x032d84 thru 0x038c9e (0x5f1b bytes) 000925 1768250329.260466 trunc from 0x038c9f to 0x034d63 (0x3f3b bytes) 000926 1768250329.288520 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34d64): Mirror 8: 0xb40da8f3 (stale) Mirror 9: 0xb40da8f3 (stale) Mirror 10: 0xb40da8f3 (stale) Mirror 11: 0xb40da8f3 (stale) Mirror 12: 0xb40da8f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d64): Mirror 8: 0xb40da8f3 (stale) Mirror 9: 0xb40da8f3 (stale) Mirror 10: 0xb40da8f3 (stale) Mirror 11: 0xb40da8f3 (stale) Mirror 12: 0xb40da8f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d64: skipped 000927 1768250329.986508 trunc from 0x034d64 to 0x0092db (0x2ba88 bytes) 000929 1768250330.026400 MIRROR_RESYNC 6 mirrors 000930 1768250331.552337 mapwrite 0x021276 thru 0x02f666 (0xe3f1 bytes) 000932 1768250331.748799 mapwrite 0x03a30b thru 0x03ffff (0x5cf5 bytes) 000933 1768250331.851880 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb9138de1 (stale) Mirror 10: 0xb9138de1 (stale) Mirror 11: 0xb9138de1 (stale) Mirror 12: 0xb9138de1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb9138de1 (stale) Mirror 10: 0xb9138de1 (stale) Mirror 11: 0xb9138de1 (stale) Mirror 12: 0xb9138de1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000934 1768250332.492297 trunc from 0x040000 to 0x034024 (0xbfdb bytes) 000935 1768250332.566708 mapread 0x0137b7 thru 0x015cae (0x24f8 bytes) 000936 1768250332.614656 read 0x018e8d thru 0x026c6f (0xdde3 bytes) 000937 1768250332.626733 mapwrite 0x01ef43 thru 0x02b0d6 (0xc194 bytes) 000939 1768250332.741235 read 0x03133b thru 0x034024 (0x2cea bytes) 000940 1768250332.757457 trunc from 0x034025 to 0x013162 (0x20ec2 bytes) 000943 1768250332.810019 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x138:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13163): Mirror 9: 0x55e91693 (stale) Mirror 10: 0x55e91693 (stale) Mirror 11: 0x55e91693 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13163): Mirror 9: 0x55e91693 (stale) Mirror 10: 0x55e91693 (stale) Mirror 11: 0x55e91693 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13163: skipped 000944 1768250333.233848 write 0x0287cf thru 0x037270 (0xeaa2 bytes) 000945 1768250333.242394 mapwrite 0x016a8c thru 0x01bd86 (0x52fb bytes) 000947 1768250333.389938 write 0x02b7e0 thru 0x02d0e7 (0x1908 bytes) 000948 1768250333.391978 read 0x01ca39 thru 0x02957a (0xcb42 bytes) 000951 1768250333.408014 mapwrite 0x02fc48 thru 0x03ac56 (0xb00f bytes) 000952 1768250333.532873 mapread 0x00acd5 thru 0x00ddad (0x30d9 bytes) 000953 1768250333.564739 trunc from 0x03ac57 to 0x0039f6 (0x37260 bytes) 000955 1768250333.609574 read 0x000279 thru 0x000af0 (0x878 bytes) 000956 1768250333.616728 mapwrite 0x01c5ea thru 0x028155 (0xbb6c bytes) 000957 1768250333.791851 trunc from 0x028156 to 0x0251dc (0x2f79 bytes) 000958 1768250333.822625 trunc from 0x0251dd to 0x024877 (0x965 bytes) 000959 1768250333.856971 trunc from 0x024878 to 0x005caf (0x1ebc8 bytes) 000960 1768250333.936764 write 0x02d581 thru 0x0302bf (0x2d3f bytes) 000961 1768250333.937845 write 0x02cb60 thru 0x03044d (0x38ee bytes) 000963 1768250333.945845 mapread 0x01aba4 thru 0x01b475 (0x8d2 bytes) 000967 1768250333.967125 mapwrite 0x010337 thru 0x01a230 (0x9efa bytes) 000968 1768250334.241124 read 0x029dbe thru 0x02baae (0x1cf1 bytes) 000970 1768250334.244998 mapwrite 0x012919 thru 0x014992 (0x207a bytes) 000971 1768250334.408447 mapread 0x026ea3 thru 0x02c323 (0x5481 bytes) 000973 1768250334.467698 MIRROR_RESYNC 4 mirrors 000974 1768250335.637187 trunc from 0x03044e to 0x03280f (0x23c2 bytes) 000975 1768250335.753838 write 0x01d291 thru 0x01f080 (0x1df0 bytes) 000976 1768250335.770064 mapread 0x031571 thru 0x03280f (0x129f bytes) 000977 1768250335.778274 mapread 0x02e0d3 thru 0x030780 (0x26ae bytes) 000979 1768250335.779842 mapwrite 0x02801b thru 0x031aa6 (0x9a8c bytes) 000985 1768250335.833780 read 0x028dac thru 0x02f8e5 (0x6b3a bytes) 000988 1768250335.836123 write 0x013a69 thru 0x019de2 (0x637a bytes) 000989 1768250335.857666 mapread 0x0195b6 thru 0x01e2f6 (0x4d41 bytes) 000990 1768250335.867255 mapread 0x0209d6 thru 0x02c8ee (0xbf19 bytes) 000991 1768250335.868680 mapread 0x01ade4 thru 0x022e42 (0x805f bytes) 000992 1768250335.870284 mapwrite 0x036749 thru 0x03ffff (0x98b7 bytes) 000993 1768250335.972610 mapread 0x031f57 thru 0x0387cb (0x6875 bytes) 000994 1768250335.973774 mapread 0x004d67 thru 0x00d3d1 (0x866b bytes) 000995 1768250335.990836 mapread 0x037d8f thru 0x03ec9b (0x6f0d bytes) 000996 1768250335.999142 mapwrite 0x00d7e2 thru 0x014462 (0x6c81 bytes) 000998 1768250336.071740 trunc from 0x040000 to 0x03158b (0xea74 bytes) 001000 1768250336.095589 mapwrite 0x02002c thru 0x02b286 (0xb25b bytes) All operations completed A-OK! PASS 70a (100s) == sanity-flr test 71: check mirror extend parent fid ==== 15:39:03 (1768250343) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 71: check append on FLR file ========== 15:39:07 (1768250347) /mnt/lustre/f71.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x87:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x69:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203009 s, 10.3 MB/s /mnt/lustre/f71.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x87:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x88:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x69:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153525 s, 6.8 MB/s /mnt/lustre/f71.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x87:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x88:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x69:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f71.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x87:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x88:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x6a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x69:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x89:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 71 (11s) == sanity-flr test 200a: stress test ===================== 15:39:19 (1768250359) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5586464 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3163008 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7577600 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7270272 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7881856 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8847328 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8248992 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9053696 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8848480 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 57389 57391 57392 57393 57396 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) 192.168.203.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:40:40 (1768250440) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2734496 .. Extending file size to 5564928 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4916288 .. Extending file size to 6187008 .. failed Extending file size to 7070368 .. Extending file size to 7789760 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8058496 .. Extending file size to 8110624 .. Extending file size to 8287328 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8227488 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8658880 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 59079 59080 59081 59082 59084 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) 192.168.203.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:42:00 (1768250520) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 26624 3742720 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 19456 3749888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 46080 7492608 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 3584 2204928 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 26624 3742720 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 19456 3749888 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 46080 7492608 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170984 s, 12.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.840212 s, 2.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5267798 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) PASS 200c (27s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:42:30 (1768250550) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241491 s, 8.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 15:42:39 (1768250559) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x30:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x80:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x80:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x81:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x9e:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:42:48 (1768250568) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.320819 s, 6.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x197:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x196:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x197:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:42:59 (1768250579) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.428886 s, 4.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa0:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:43:10 (1768250590) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.71947 s, 2.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:43:44 (1768250624) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg320-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250843 s, 4.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x198:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x199:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x198:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x198:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x199:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x198:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x199:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:44:46 (1768250686) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg320-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342864 s, 6.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x85:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x85:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (69s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:45:55 (1768250755) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg320-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg320-server: Pool lustre.test_204f created Adding targets to pool oleg320-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg320-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.450056 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x86:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x86:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg320-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg320-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg320-server: Pool lustre.test_204f destroyed PASS 204f (93s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:47:28 (1768250848) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:47:36 (1768250856) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:47:46 (1768250866) oleg320-server: Pool lustre.test_206 created Adding targets to pool oleg320-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg320-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg320-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg320-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg320-server: Pool lustre.test_206 destroyed PASS 206 (33s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:48:19 (1768250899) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112726 s, 9.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107786 s, 9.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3c71d35f Mirror 3: 0x3c71d35f PASS 207 (18s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:48:37 (1768250917) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:48:42 (1768250922) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:48:47 (1768250927) Starting client: oleg320-client.virtnet: -o user_xattr,flock 192.168.203.120@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.196281 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1000000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x990000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x990000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x990000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2170000 Read this (4 bytes): BBBB 192.168.203.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg320-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:49:32 (1768250972) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0291815 s, 281 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:49:41 (1768250981) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533679 s, 19.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x51 lmm_fid: [0x200000405:0x51:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x51 lmm_fid: [0x200000405:0x51:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611217 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-12 15:49:42.000000000 -0500 Modify: 2026-01-12 15:49:42.000000000 -0500 Change: 2026-01-12 15:49:44.000000000 -0500 Birth: 2026-01-12 15:49:43.000000000 -0500 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:49:50 (1768250990) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:49:54 (1768250994) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.860899 s, 47.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0678753 s, 60.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 12 15:49 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 12 15:49 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:50:06 (1768251006) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:50:14 (1768251014) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1912 sec =========== 15:50:23 (1768251023) === sanity-flr: start cleanup 15:50:25 (1768251025) === === sanity-flr: finish cleanup 15:50:33 (1768251033) ===