-----============= acceptance-small: sanity-flr ============----- Thu Mar 5 19:44:00 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg140-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:44:07 (1772757847) === oleg140-client.virtnet: executing check_config_client /mnt/lustre oleg140-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg140-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff931d85513800.idle_timeout=debug osc.lustre-OST0001-osc-ffff931d85513800.idle_timeout=debug disable quota as required oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:44:17 (1772757857) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7540] == sanity-flr test 0a: lfs mirror create with -N option == 19:44:17 (1772757857) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:44:22 (1772757862) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:44:23 (1772757863) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:44:24 (1772757864) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:44:30 (1772757870) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:44:31 (1772757871) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 19:44:32 (1772757872) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff931d85513800.stats=clear osc.lustre-OST0001-osc-ffff931d85513800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:44:35 (1772757875) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:44:38 (1772757878) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:44:40 (1772757880) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:44:41 (1772757881) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 19:44:44 (1772757884) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:44:46 (1772757886) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:44:47 (1772757887) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:44:49 (1772757889) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:44:52 (1772757892) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:44:54 (1772757894) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.277161 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386009 s, 27.2 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:44:57 (1772757897) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116866 s, 89.7 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:44:59 (1772757899) Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (24s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:45:23 (1772757923) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.163452 s, 57.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:45:35 (1772757935) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (21s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:45:56 (1772757956) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.20297 s, 43.6 MB/s -rw-r--r-- 1 root root 52428800 Mar 5 19:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.292s user 0m0.000s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff931d85513800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.215s user 0m0.001s sys 0m0.120s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg140-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg140-server oleg140-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.268s user 0m0.001s sys 0m0.142s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg140-server: oleg140-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg140-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg140-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff931d85513800.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:46:23 (1772757983) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:46:24 (1772757984) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:46:25 (1772757985) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 19:46:27 (1772757987) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 19:46:29 (1772757989) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.18818 s, 66.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:46:34 (1772757994) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140273 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117712 s, 89.1 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:46:36 (1772757996) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:46:40 (1772758000) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00089187 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000582038 s, 1.8 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00079349 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000519731 s, 2.0 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 19:46:43 (1772758003) create test files with size 15734264 .. 400cb205e51765251bfc33dd79ee6a06 - 96683bedeffb93c95a3a09a32ffc381c - b979e6784411c06e2ddfbc1104961807 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 19:46:49 (1772758009) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:46:55 (1772758015) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:46:57 (1772758017) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455566 s, 23.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011851 s, 88.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 19:47:00 (1772758020) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0465369 s, 45.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.132682 s, 31.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0808182 s, 25.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 19:47:06 (1772758026) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 19:47:07 (1772758027) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:47:09 (1772758029) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118673 s, 88.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 19:47:11 (1772758031) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:47:13 (1772758033) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:47:17 (1772758037) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.120004 s, 87.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:47:35 (1772758055) llite.lustre-ffff931d82fbd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.134279 s, 78.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg140-client.virtnet /mnt/lustre (opts:) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff931d89d7c800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:47:40 (1772758060) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (16s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:47:56 (1772758076) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:48:10 (1772758090) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 19:48:11 (1772758091) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0968575 s, 32.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.063476 s, 49.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x68cfc96d (stale) Mirror 2: 0x68cfc96d (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 48556040b46340224314f1c3c13cd6d1 - mirror 1 checksum: c9076643bac0f6101a292b81d0ce61bf - mirror 2 checksum: 48556040b46340224314f1c3c13cd6d1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Mar 5 19:48:12 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Mar 5 19:48:12 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 48556040b46340224314f1c3c13cd6d1 - mirror 1 checksum: c9076643bac0f6101a292b81d0ce61bf - mirror 2 checksum: c9076643bac0f6101a292b81d0ce61bf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Mar 5 19:48:12 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Mar 5 19:48:12 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:48:15 (1772758095) Failing mds1 on oleg140-server Stopping /mnt/lustre-mds1 (opts:) on oleg140-server 19:48:18 (1772758098) shut down facet: mds1 facet_host: oleg140-server facet_failover_host: oleg140-server Failover mds1 to oleg140-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 19:48:32 (1772758112) targets are mounted 19:48:32 (1772758112) facet_failover done oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139293 s, 75.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:48:39 (1772758119) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:48:40 (1772758120) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:48:41 (1772758121) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00485525 s, 3.4 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:48:44 (1772758124) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:48:45 (1772758125) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:48:47 (1772758127) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:48:53 (1772758133) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:49:14 (1772758154) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:49:32 (1772758172) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:49:50 (1772758190) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 19:49:53 (1772758193) mirror create pid 52196 mirror split pid 52197 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52196 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52197 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 19:50:55 (1772758255) Seed set to 8256 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1772758256.463348 mapwrite 0x029d61 thru 0x034e84 (0xb124 bytes) 000002 1772758256.507638 MIRROR_EXTEND to 2 mirrors 000003 1772758256.692017 mapread 0x008e8d thru 0x00c0d3 (0x3247 bytes) truncating to largest ever: 0x5e41 000008 1772758256.705861 trunc from 0x034e85 to 0x005e40 (0x2f044 bytes) 000010 1772758256.738633 mapwrite 0x01f8c6 thru 0x02ddd2 (0xe50d bytes) 000011 1772758256.773701 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ddd3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddd3): Mirror 2: 0x8ee39c52 (stale) CRC-32 checksum value for chunk [0, 0x2ddd3): Mirror 1: 0x60db26cf Mirror 3: 0x60db26cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddd3: skipped 000012 1772758257.023456 mapwrite 0x0119e0 thru 0x01ec4d (0xd26e bytes) 000014 1772758257.089800 write 0x0023ef thru 0x00938d (0x6f9f bytes) 000015 1772758257.094000 mapread 0x00defe thru 0x01b697 (0xd79a bytes) 000016 1772758257.095210 mapwrite 0x01f186 thru 0x021648 (0x24c3 bytes) 000017 1772758257.110030 write 0x029eaa thru 0x02c816 (0x296d bytes) 000019 1772758257.113825 mapread 0x02a124 thru 0x02ddd2 (0x3caf bytes) 000020 1772758257.114843 write 0x01ee7e thru 0x025336 (0x64b9 bytes) 000021 1772758257.115960 mapwrite 0x0236ac thru 0x025540 (0x1e95 bytes) truncating to largest ever: 0x346ba 000025 1772758257.134074 trunc from 0x02ddd3 to 0x0346b9 (0x68e7 bytes) 000028 1772758257.149978 mapwrite 0x00c757 thru 0x015904 (0x91ae bytes) 000029 1772758257.167449 read 0x02b801 thru 0x0346b9 (0x8eb9 bytes) 000030 1772758257.171782 trunc from 0x0346ba to 0x02ca4e (0x7c6b bytes) 000031 1772758257.180984 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ca4f): Mirror 3: 0x16396174 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca4f): Mirror 3: 0x16396174 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca4f: skipped 000032 1772758257.296702 MIRROR_RESYNC 2 mirrors 000033 1772758257.500125 trunc from 0x02ca4f to 0x0022fd (0x2a751 bytes) 000035 1772758257.523197 mapread 0x000152 thru 0x0022fd (0x21ac bytes) 000037 1772758257.529736 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22fe): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22fe): Mirror 3: 0xe70233a9 (stale) CRC-32 checksum value for chunk [0, 0x22fe): Mirror 1: 0xe70233a9 Mirror 4: 0xe70233a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22fe: skipped 000040 1772758257.742729 mapread 0x000bdb thru 0x0022fd (0x1723 bytes) 000041 1772758257.753701 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000042 1772758257.853762 MIRROR_EXTEND to 3 mirrors 000043 1772758258.029530 MIRROR_RESYNC 3 mirrors 000044 1772758258.144006 mapread 0x00017e thru 0x0022fd (0x2180 bytes) 000048 1772758258.155384 write 0x0216b8 thru 0x028f9a (0x78e3 bytes) 000050 1772758258.174914 mapread 0x011d1f thru 0x01f1ec (0xd4ce bytes) 000051 1772758258.180423 mapread 0x024e57 thru 0x028f9a (0x4144 bytes) 000052 1772758258.181191 trunc from 0x028f9b to 0x0116ee (0x178ac bytes) 000053 1772758258.192416 mapwrite 0x00cb82 thru 0x00eee3 (0x2362 bytes) 000054 1772758258.206438 mapwrite 0x008f61 thru 0x00a8f9 (0x1999 bytes) 000055 1772758258.224716 mapwrite 0x02926f thru 0x02c721 (0x34b3 bytes) 000056 1772758258.251447 MIRROR_RESYNC 3 mirrors 000058 1772758258.464653 mapread 0x00457d thru 0x007822 (0x32a6 bytes) 000060 1772758258.470722 write 0x017948 thru 0x023750 (0xbe09 bytes) 000065 1772758258.495686 mapread 0x02385a thru 0x02828c (0x4a33 bytes) 000066 1772758258.499789 mapwrite 0x029306 thru 0x03126a (0x7f65 bytes) 000067 1772758258.527213 mapread 0x00d1d3 thru 0x0125b3 (0x53e1 bytes) 000068 1772758258.531367 read 0x02b7bb thru 0x030788 (0x4fce bytes) 000069 1772758258.532341 read 0x01090f thru 0x01941d (0x8b0f bytes) 000071 1772758258.533395 trunc from 0x03126b to 0x028249 (0x9021 bytes) 000072 1772758258.540816 MIRROR_RESYNC 3 mirrors 000073 1772758258.735569 trunc from 0x02824a to 0x02fb0a (0x78c1 bytes) 000074 1772758258.761150 read 0x023c51 thru 0x02fb0a (0xbeba bytes) 000076 1772758258.767981 trunc from 0x02fb0b to 0x00efa9 (0x20b61 bytes) 000078 1772758258.776500 trunc from 0x00efaa to 0x011ce4 (0x2d3b bytes) 000081 1772758258.785290 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11ce5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ce5): Mirror 4: 0xd6f39348 (stale) Mirror 5: 0xd6f39348 (stale) CRC-32 checksum value for chunk [0, 0x11ce5): Mirror 1: 0xd6f39348 Mirror 6: 0xd6f39348 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ce5: skipped 000083 1772758259.013313 write 0x00c495 thru 0x01376f (0x72db bytes) 000084 1772758259.034980 mapread 0x007dbd thru 0x01204f (0xa293 bytes) 000085 1772758259.040414 read 0x008646 thru 0x01376f (0xb12a bytes) 000086 1772758259.043542 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13770): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13770): Mirror 4: 0xc797fcc4 (stale) Mirror 5: 0xc797fcc4 (stale) Mirror 6: 0xc797fcc4 (stale) CRC-32 checksum value for chunk [0, 0x13770): Mirror 1: 0xf3f6fa01 Mirror 7: 0xf3f6fa01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13770: skipped 000087 1772758259.288063 trunc from 0x013770 to 0x02836c (0x14bfd bytes) 000089 1772758259.315119 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2836d): Mirror 5: 0x8e6316b4 (stale) Mirror 6: 0x863f982a (stale) Mirror 7: 0x5d045460 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2836d): Mirror 5: 0x8e6316b4 (stale) Mirror 6: 0x863f982a (stale) Mirror 7: 0x5d045460 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2836d: skipped 000090 1772758259.421007 MIRROR_RESYNC 4 mirrors 000091 1772758259.661688 write 0x00aae2 thru 0x0125b0 (0x7acf bytes) 000092 1772758259.681429 mapwrite 0x0014af thru 0x00ca80 (0xb5d2 bytes) 000093 1772758259.699142 write 0x0201fd thru 0x029437 (0x923b bytes) 000094 1772758259.702862 write 0x03dda4 thru 0x03ffff (0x225c bytes) 000095 1772758259.703450 mapwrite 0x02ce9d thru 0x030cd8 (0x3e3c bytes) 000099 1772758259.719935 mapwrite 0x01cf82 thru 0x02778a (0xa809 bytes) 000100 1772758259.735937 mapread 0x03b464 thru 0x03ffff (0x4b9c bytes) 000101 1772758259.740514 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x4ffd0e5d (stale) Mirror 6: 0x4ffd0e5d (stale) Mirror 7: 0x4ffd0e5d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf228c1f3 Mirror 8: 0xf228c1f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000103 1772758259.968323 mapread 0x00b805 thru 0x00d6a0 (0x1e9c bytes) 000104 1772758259.976806 mapread 0x02a300 thru 0x037db1 (0xdab2 bytes) 000105 1772758259.983452 write 0x03739e thru 0x03ffff (0x8c62 bytes) 000107 1772758260.004652 mapwrite 0x0234f8 thru 0x02598b (0x2494 bytes) 000108 1772758260.019962 mapread 0x023204 thru 0x02514d (0x1f4a bytes) 000110 1772758260.021795 write 0x03d79a thru 0x03f0c0 (0x1927 bytes) 000111 1772758260.022723 read 0x00b251 thru 0x0199f3 (0xe7a3 bytes) 000112 1772758260.027748 MIRROR_RESYNC 5 mirrors 000114 1772758260.292225 read 0x011110 thru 0x0180c3 (0x6fb4 bytes) 000121 1772758260.296145 read 0x0033d9 thru 0x00e47b (0xb0a3 bytes) 000122 1772758260.300121 write 0x00273c thru 0x00e532 (0xbdf7 bytes) 000124 1772758260.320817 read 0x0222e2 thru 0x0267d4 (0x44f3 bytes) 000125 1772758260.324142 read 0x000aae thru 0x003abb (0x300e bytes) 000127 1772758260.326666 MIRROR_RESYNC 5 mirrors truncating to largest ever: 0x3ea29 000128 1772758260.584980 trunc from 0x040000 to 0x03ea28 (0x15d7 bytes) 000130 1772758260.612675 write 0x027ed7 thru 0x033f8e (0xc0b8 bytes) 000131 1772758260.618782 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea29): Mirror 5: 0x2f7afb3a (stale) Mirror 6: 0x2f7afb3a (stale) Mirror 8: 0x2f7afb3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea29): Mirror 5: 0x2f7afb3a (stale) Mirror 6: 0x2f7afb3a (stale) Mirror 8: 0x2f7afb3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea29: skipped 000132 1772758260.725063 mapread 0x024618 thru 0x0296e8 (0x50d1 bytes) 000133 1772758260.731470 write 0x026c4b thru 0x035806 (0xebbc bytes) 000134 1772758260.733818 write 0x0265cb thru 0x02b692 (0x50c8 bytes) 000135 1772758260.734802 read 0x0366ad thru 0x03ea28 (0x837c bytes) 000137 1772758260.738549 write 0x000014 thru 0x006256 (0x6243 bytes) 000138 1772758260.742015 mapwrite 0x00ba8b thru 0x00e624 (0x2b9a bytes) 000139 1772758260.757654 MIRROR_RESYNC 4 mirrors 000143 1772758260.995046 mapread 0x00646c thru 0x010c5c (0xa7f1 bytes) 000146 1772758261.001124 write 0x02e2b4 thru 0x0326e0 (0x442d bytes) 000147 1772758261.026986 write 0x00dd9d thru 0x017912 (0x9b76 bytes) 000148 1772758261.032209 write 0x035a54 thru 0x038361 (0x290e bytes) 000149 1772758261.035532 MIRROR_RESYNC 4 mirrors 000150 1772758261.288571 trunc from 0x03ea29 to 0x02f737 (0xf2f1 bytes) 000151 1772758261.309945 trunc from 0x02f738 to 0x021ac0 (0xdc77 bytes) 000153 1772758261.317197 write 0x01db2b thru 0x01e33d (0x813 bytes) 000154 1772758261.320358 read 0x00e789 thru 0x01ddc8 (0xf640 bytes) 000155 1772758261.324635 MIRROR_RESYNC 4 mirrors 000158 1772758261.529935 read 0x015787 thru 0x021ac0 (0xc33a bytes) 000163 1772758261.533358 mapread 0x01be08 thru 0x021ac0 (0x5cb9 bytes) 000164 1772758261.533894 write 0x00b63d thru 0x01b3ed (0xfdb1 bytes) 000165 1772758261.554038 read 0x0206db thru 0x021ac0 (0x13e6 bytes) 000168 1772758261.557955 write 0x0033d3 thru 0x00cf19 (0x9b47 bytes) 000169 1772758261.559131 trunc from 0x021ac1 to 0x013178 (0xe948 bytes) 000170 1772758261.569260 mapread 0x000c22 thru 0x0033da (0x27b9 bytes) 000172 1772758261.570288 read 0x005b0d thru 0x00b664 (0x5b58 bytes) 000175 1772758261.574402 mapwrite 0x01e84c thru 0x01f4be (0xc73 bytes) 000176 1772758261.594845 mapread 0x01ea34 thru 0x01f4be (0xa8b bytes) 000179 1772758261.596804 mapread 0x01cf7d thru 0x01f4be (0x2542 bytes) 000181 1772758261.597950 trunc from 0x01f4bf to 0x00ada3 (0x1471b bytes) 000183 1772758261.603854 write 0x02a38a thru 0x037e23 (0xda9a bytes) 000184 1772758261.607497 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37e24): Mirror 5: 0x6c69ce4c (stale) Mirror 8: 0x6c69ce4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e24): Mirror 5: 0x6c69ce4c (stale) Mirror 8: 0x6c69ce4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e24: skipped 000185 1772758261.707351 MIRROR_RESYNC 3 mirrors 000188 1772758261.915393 write 0x029f5f thru 0x038b82 (0xec24 bytes) 000189 1772758261.937720 mapread 0x019da3 thru 0x01bf5c (0x21ba bytes) 000190 1772758261.943881 write 0x01cad7 thru 0x024a73 (0x7f9d bytes) 000191 1772758261.946472 mapread 0x02fbb3 thru 0x030453 (0x8a1 bytes) 000192 1772758261.947335 mapread 0x01b43b thru 0x02ac9c (0xf862 bytes) 000193 1772758261.949874 mapwrite 0x0079fd thru 0x00f529 (0x7b2d bytes) 000194 1772758261.970988 write 0x024e92 thru 0x0252c7 (0x436 bytes) 000196 1772758261.972645 write 0x018018 thru 0x026258 (0xe241 bytes) 000197 1772758261.974032 mapread 0x01d66f thru 0x028182 (0xab14 bytes) 000199 1772758261.975276 trunc from 0x038b83 to 0x003921 (0x35261 bytes) 000200 1772758261.988160 mapwrite 0x027d80 thru 0x037789 (0xfa0a bytes) 000203 1772758262.021948 MIRROR_RESYNC 3 mirrors 000205 1772758262.259896 trunc from 0x03778a to 0x0362a3 (0x14e6 bytes) 000208 1772758262.279022 write 0x031fec thru 0x0343a3 (0x23b8 bytes) 000209 1772758262.283035 mapwrite 0x011d24 thru 0x01848c (0x6769 bytes) 000210 1772758262.306749 mapread 0x02c0bf thru 0x03237d (0x62bf bytes) 000214 1772758262.315388 MIRROR_RESYNC 3 mirrors 000215 1772758262.536974 write 0x034766 thru 0x038499 (0x3d34 bytes) 000219 1772758262.558787 mapwrite 0x0315f2 thru 0x035c96 (0x46a5 bytes) 000220 1772758262.581330 mapread 0x02efae thru 0x038499 (0x94ec bytes) 000221 1772758262.582521 trunc from 0x03849a to 0x03b33b (0x2ea2 bytes) 000222 1772758262.594400 trunc from 0x03b33c to 0x02589c (0x15a9f bytes) 000224 1772758262.602905 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2589d): Mirror 8: 0xb10bed88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2589d): Mirror 8: 0xb10bed88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2589d: skipped 000226 1772758262.708942 trunc from 0x02589d to 0x029adf (0x4243 bytes) 000228 1772758262.715264 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000229 1772758262.759043 read 0x005ef6 thru 0x015099 (0xf1a4 bytes) 000230 1772758262.763393 trunc from 0x029ae0 to 0x01be96 (0xdc49 bytes) 000237 1772758262.772920 mapread 0x006edf thru 0x0077d1 (0x8f3 bytes) 000238 1772758262.773926 mapwrite 0x02bd45 thru 0x031143 (0x53ff bytes) 000243 1772758262.801089 read 0x0279e2 thru 0x030670 (0x8c8f bytes) 000244 1772758262.801874 trunc from 0x031144 to 0x034a73 (0x3930 bytes) 000245 1772758262.808712 read 0x02ae30 thru 0x034a73 (0x9c44 bytes) 000247 1772758262.813276 read 0x0089d2 thru 0x016dc1 (0xe3f0 bytes) 000250 1772758262.814604 write 0x0314fb thru 0x035cfd (0x4803 bytes) 000251 1772758262.815367 read 0x01780b thru 0x01de05 (0x65fb bytes) 000254 1772758262.816740 write 0x00e836 thru 0x01806b (0x9836 bytes) 000255 1772758262.817902 trunc from 0x035cfe to 0x02dc47 (0x80b6 bytes) 000256 1772758262.825023 write 0x0243ad thru 0x02c5ab (0x81ff bytes) 000258 1772758262.827079 read 0x00f127 thru 0x01c9af (0xd889 bytes) 000259 1772758262.828078 mapread 0x0299bb thru 0x02dc47 (0x428d bytes) 000261 1772758262.831884 MIRROR_EXTEND to 2 mirrors 000266 1772758263.037002 trunc from 0x02dc48 to 0x01fd79 (0xdece bytes) 000268 1772758263.076021 trunc from 0x01fd7a to 0x0262f8 (0x657f bytes) truncating to largest ever: 0x3fef3 000270 1772758263.082796 trunc from 0x0262f9 to 0x03fef2 (0x19bfa bytes) 000271 1772758263.088951 write 0x0379d3 thru 0x03f1bf (0x77ed bytes) 000273 1772758263.094396 trunc from 0x03fef3 to 0x02a16b (0x15d87 bytes) 000274 1772758263.100030 mapread 0x01917b thru 0x023f66 (0xadec bytes) 000275 1772758263.109799 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000276 1772758263.157429 write 0x032114 thru 0x03d452 (0xb33f bytes) 000277 1772758263.159015 trunc from 0x03d453 to 0x03d843 (0x3f1 bytes) 000280 1772758263.170427 trunc from 0x03d844 to 0x00b27b (0x325c8 bytes) 000287 1772758263.179764 mapread 0x0043e5 thru 0x0069be (0x25da bytes) 000288 1772758263.187246 trunc from 0x00b27c to 0x02ce6f (0x21bf4 bytes) 000289 1772758263.193092 mapread 0x0241d7 thru 0x02ce6f (0x8c99 bytes) 000290 1772758263.196789 mapread 0x0176e4 thru 0x01e3a9 (0x6cc6 bytes) 000292 1772758263.201166 trunc from 0x02ce70 to 0x013b77 (0x192f8 bytes) 000295 1772758263.207985 mapread 0x01034d thru 0x013b77 (0x382b bytes) 000299 1772758263.210091 mapread 0x002bbf thru 0x00dc2a (0xb06c bytes) 000300 1772758263.214206 write 0x034630 thru 0x03d524 (0x8ef5 bytes) 000301 1772758263.215336 write 0x01cf45 thru 0x01e994 (0x1a50 bytes) 000302 1772758263.217963 trunc from 0x03d525 to 0x024e9a (0x1868a bytes) 000305 1772758263.224441 read 0x0221aa thru 0x024e9a (0x2cf1 bytes) 000306 1772758263.226670 MIRROR_EXTEND to 2 mirrors 000308 1772758263.423924 read 0x01a7c6 thru 0x022239 (0x7a74 bytes) 000309 1772758263.428903 mapwrite 0x03f683 thru 0x03ffff (0x97d bytes) 000310 1772758263.471305 mapread 0x008ae0 thru 0x00b1a3 (0x26c4 bytes) 000311 1772758263.477788 trunc from 0x040000 to 0x035dd1 (0xa22e bytes) 000312 1772758263.483882 read 0x028be6 thru 0x029336 (0x751 bytes) 000316 1772758263.489112 read 0x008eed thru 0x0106f4 (0x7808 bytes) 000319 1772758263.490548 mapread 0x025b3f thru 0x0288e5 (0x2da7 bytes) 000321 1772758263.491979 read 0x01bdf3 thru 0x02a239 (0xe447 bytes) 000322 1772758263.492459 write 0x02ece1 thru 0x03590c (0x6c2c bytes) 000324 1772758263.494112 trunc from 0x035dd2 to 0x030745 (0x568c bytes) 000327 1772758263.501339 mapread 0x022ac2 thru 0x029910 (0x6e4f bytes) 000329 1772758263.502661 trunc from 0x030746 to 0x01806d (0x186d8 bytes) 000330 1772758263.508341 mapread 0x00bcc6 thru 0x0111b0 (0x54eb bytes) 000332 1772758263.510345 write 0x01c45b thru 0x027adb (0xb681 bytes) 000334 1772758263.513094 read 0x008ce7 thru 0x00f328 (0x6642 bytes) 000336 1772758263.514268 write 0x0142e0 thru 0x024100 (0xfe21 bytes) 000338 1772758263.516705 mapwrite 0x0140a7 thru 0x01ab56 (0x6ab0 bytes) 000339 1772758263.536630 read 0x003ed6 thru 0x00a5b9 (0x66e4 bytes) 000340 1772758263.537874 write 0x03ffb9 thru 0x03ffff (0x47 bytes) 000342 1772758263.539026 read 0x001a00 thru 0x00c622 (0xac23 bytes) 000343 1772758263.539681 mapread 0x00c4f2 thru 0x012717 (0x6226 bytes) 000345 1772758263.540805 mapwrite 0x0076ed thru 0x00a6dc (0x2ff0 bytes) 000346 1772758263.560191 mapwrite 0x012893 thru 0x015df4 (0x3562 bytes) 000349 1772758263.576202 write 0x00ae44 thru 0x0138d1 (0x8a8e bytes) 000353 1772758263.579079 write 0x01b2cc thru 0x01e6aa (0x33df bytes) 000354 1772758263.579695 mapread 0x02087e thru 0x0208f3 (0x76 bytes) 000357 1772758263.580964 trunc from 0x040000 to 0x0284c4 (0x17b3b bytes) 000358 1772758263.586329 read 0x015bbb thru 0x021083 (0xb4c9 bytes) 000359 1772758263.587076 read 0x01e27a thru 0x0284c4 (0xa24b bytes) 000361 1772758263.591076 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000363 1772758263.632038 mapread 0x02307a thru 0x0248e3 (0x186a bytes) 000364 1772758263.635593 mapwrite 0x03e185 thru 0x03ffff (0x1e7b bytes) 000365 1772758263.658292 write 0x0289f6 thru 0x032ace (0xa0d9 bytes) 000366 1772758263.662988 trunc from 0x040000 to 0x0369c8 (0x9637 bytes) 000368 1772758263.669988 read 0x02f073 thru 0x02fc30 (0xbbe bytes) 000370 1772758263.670974 trunc from 0x0369c9 to 0x006eab (0x2fb1d bytes) 000372 1772758263.677353 mapwrite 0x01701a thru 0x025335 (0xe31c bytes) 000373 1772758263.708734 mapwrite 0x024899 thru 0x0286ce (0x3e36 bytes) 000375 1772758263.731838 mapread 0x00fdf4 thru 0x0109bc (0xbc9 bytes) 000376 1772758263.737157 trunc from 0x0286cf to 0x020851 (0x7e7d bytes) 000377 1772758263.742406 mapwrite 0x02409b thru 0x027d57 (0x3cbd bytes) 000378 1772758263.766658 mapread 0x018f98 thru 0x023ae8 (0xab51 bytes) 000379 1772758263.768259 read 0x012ead thru 0x017f38 (0x508c bytes) 000382 1772758263.770996 write 0x025671 thru 0x034e67 (0xf7f7 bytes) 000383 1772758263.773146 mapwrite 0x024e8b thru 0x025ca4 (0xe1a bytes) 000384 1772758263.786994 read 0x007d08 thru 0x011910 (0x9c09 bytes) 000385 1772758263.790691 mapwrite 0x007564 thru 0x0113fb (0x9e98 bytes) 000387 1772758263.809291 mapread 0x009445 thru 0x00b80e (0x23ca bytes) 000388 1772758263.810819 write 0x0234a2 thru 0x024828 (0x1387 bytes) 000389 1772758263.812113 mapwrite 0x0165ac thru 0x01a49c (0x3ef1 bytes) 000390 1772758263.831861 MIRROR_EXTEND to 2 mirrors 000391 1772758264.007828 read 0x031838 thru 0x034e67 (0x3630 bytes) 000392 1772758264.011203 write 0x016ed3 thru 0x024049 (0xd177 bytes) 000397 1772758264.036056 trunc from 0x034e68 to 0x03af98 (0x6131 bytes) 000400 1772758264.049223 write 0x014252 thru 0x0196b4 (0x5463 bytes) 000402 1772758264.052933 read 0x0201cd thru 0x02a600 (0xa434 bytes) 000404 1772758264.057005 mapread 0x003bbf thru 0x005e96 (0x22d8 bytes) 000409 1772758264.059654 write 0x01f72b thru 0x02e297 (0xeb6d bytes) 000414 1772758264.062858 trunc from 0x03af99 to 0x032376 (0x8c22 bytes) 000417 1772758264.071675 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32377): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32377): Mirror 2: 0x79cae329 (stale) CRC-32 checksum value for chunk [0, 0x32377): Mirror 1: 0xc774a7c4 Mirror 3: 0xc774a7c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32377: skipped 000418 1772758264.308529 mapread 0x019561 thru 0x02791b (0xe3bb bytes) 000419 1772758264.320805 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32377): Mirror 2: 0x79cae329 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32377): Mirror 2: 0x79cae329 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32377: skipped 000420 1772758264.404153 mapwrite 0x01848c thru 0x023516 (0xb08b bytes) 000421 1772758264.433036 mapread 0x00a965 thru 0x00f3eb (0x4a87 bytes) 000424 1772758264.439488 trunc from 0x032377 to 0x021cca (0x106ac bytes) 000425 1772758264.453774 mapwrite 0x01bc13 thru 0x0231c4 (0x75b2 bytes) 000426 1772758264.484168 write 0x009ddc thru 0x011021 (0x7246 bytes) 000429 1772758264.486452 mapwrite 0x02caea thru 0x02eb3c (0x2053 bytes) 000430 1772758264.513827 read 0x025d70 thru 0x02eb3c (0x8dcd bytes) 000432 1772758264.516477 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000433 1772758264.557870 read 0x00e181 thru 0x01c4a8 (0xe328 bytes) 000434 1772758264.561568 trunc from 0x02eb3d to 0x02d282 (0x18ba bytes) 000437 1772758264.567978 MIRROR_EXTEND to 2 mirrors 000438 1772758264.745386 MIRROR_RESYNC 2 mirrors 000439 1772758264.873439 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000440 1772758264.920099 read 0x008af3 thru 0x012541 (0x9a4f bytes) 000441 1772758264.923381 trunc from 0x02d283 to 0x02140c (0xbe76 bytes) 000445 1772758264.937698 trunc from 0x02140d to 0x02a3ad (0x8fa1 bytes) 000446 1772758264.942713 write 0x0144b8 thru 0x01dd2a (0x9873 bytes) 000447 1772758264.946092 write 0x00be59 thru 0x019f2a (0xe0d2 bytes) 000453 1772758264.954673 MIRROR_EXTEND to 2 mirrors 000455 1772758265.164671 read 0x0249a6 thru 0x026595 (0x1bf0 bytes) 000456 1772758265.168340 read 0x020de7 thru 0x0219df (0xbf9 bytes) 000457 1772758265.173842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000458 1772758265.216827 MIRROR_EXTEND to 2 mirrors 000460 1772758265.449893 read 0x01c691 thru 0x02171a (0x508a bytes) 000461 1772758265.460149 mapread 0x006716 thru 0x00e176 (0x7a61 bytes) 000462 1772758265.474212 read 0x025925 thru 0x02a3ad (0x4a89 bytes) 000464 1772758265.482535 mapread 0x00b070 thru 0x019823 (0xe7b4 bytes) 000465 1772758265.488555 MIRROR_RESYNC 2 mirrors 000466 1772758265.607711 read 0x003c65 thru 0x00c642 (0x89de bytes) 000471 1772758265.617950 write 0x036d15 thru 0x03ffff (0x92eb bytes) 000472 1772758265.637737 read 0x02733f thru 0x0319af (0xa671 bytes) 000473 1772758265.643692 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000474 1772758265.687778 mapread 0x02eaea thru 0x035068 (0x657f bytes) 000475 1772758265.695879 read 0x01fcf5 thru 0x02d56a (0xd876 bytes) 000477 1772758265.700190 mapread 0x0287cb thru 0x031a0d (0x9243 bytes) 000478 1772758265.701255 write 0x00f5a6 thru 0x015e31 (0x688c bytes) 000479 1772758265.705726 mapwrite 0x01b507 thru 0x029c43 (0xe73d bytes) 000481 1772758265.731814 mapread 0x0183f0 thru 0x0262ab (0xdebc bytes) 000482 1772758265.733052 mapwrite 0x023d1b thru 0x02587c (0x1b62 bytes) 000485 1772758265.749525 read 0x03b9a0 thru 0x03f8cd (0x3f2e bytes) 000486 1772758265.753479 read 0x02625e thru 0x02984e (0x35f1 bytes) 000487 1772758265.753982 mapwrite 0x005040 thru 0x005941 (0x902 bytes) 000489 1772758265.767208 read 0x02117d thru 0x029c36 (0x8aba bytes) 000490 1772758265.767753 write 0x000baf thru 0x007530 (0x6982 bytes) 000491 1772758265.768802 read 0x02f767 thru 0x0317cd (0x2067 bytes) 000493 1772758265.769999 mapread 0x00c90e thru 0x012003 (0x56f6 bytes) 000494 1772758265.770915 read 0x007863 thru 0x0112bc (0x9a5a bytes) 000497 1772758265.772573 read 0x031f23 thru 0x034dfe (0x2edc bytes) 000499 1772758265.773641 mapwrite 0x0289d1 thru 0x035fd5 (0xd605 bytes) 000501 1772758265.790750 mapwrite 0x01d639 thru 0x01f647 (0x200f bytes) 000504 1772758265.805069 read 0x030a32 thru 0x0368c7 (0x5e96 bytes) 000507 1772758265.806915 trunc from 0x040000 to 0x025a15 (0x1a5ea bytes) 000508 1772758265.813400 read 0x016b17 thru 0x0244fc (0xd9e6 bytes) 000509 1772758265.814449 mapwrite 0x0161ea thru 0x01681d (0x634 bytes) 000510 1772758265.830389 MIRROR_EXTEND to 2 mirrors 000511 1772758265.993886 MIRROR_RESYNC 2 mirrors 000513 1772758266.130899 mapwrite 0x002d41 thru 0x00ca90 (0x9d50 bytes) 000515 1772758266.194608 read 0x01072c thru 0x01a4fa (0x9dcf bytes) 000516 1772758266.198122 mapwrite 0x002e58 thru 0x0086d3 (0x587c bytes) 000519 1772758266.214014 MIRROR_RESYNC 2 mirrors 000520 1772758266.394455 read 0x004b2e thru 0x00f831 (0xad04 bytes) 000521 1772758266.397626 mapwrite 0x00f4d6 thru 0x018344 (0x8e6f bytes) 000524 1772758266.457202 write 0x00a493 thru 0x00c3f9 (0x1f67 bytes) 000525 1772758266.458420 read 0x019f55 thru 0x01abce (0xc7a bytes) 000526 1772758266.468527 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) 000527 1772758266.516706 write 0x013a35 thru 0x01dace (0xa09a bytes) 000528 1772758266.521378 mapread 0x0067fd thru 0x0143db (0xdbdf bytes) 000530 1772758266.528819 mapwrite 0x027453 thru 0x0288cd (0x147b bytes) 000534 1772758266.563925 mapwrite 0x01fd07 thru 0x029434 (0x972e bytes) 000535 1772758266.589273 mapread 0x01fdac thru 0x02713c (0x7391 bytes) 000536 1772758266.590500 write 0x020f69 thru 0x030ed0 (0xff68 bytes) 000538 1772758266.592853 read 0x028ea8 thru 0x030ed0 (0x8029 bytes) 000541 1772758266.593971 write 0x02069e thru 0x028ad3 (0x8436 bytes) 000547 1772758266.596994 mapwrite 0x038d5b thru 0x03ffff (0x72a5 bytes) 000549 1772758266.627332 read 0x013082 thru 0x0227d5 (0xf754 bytes) 000551 1772758266.629008 mapread 0x020b9f thru 0x02a9c7 (0x9e29 bytes) 000553 1772758266.630750 mapread 0x007d2b thru 0x00f39d (0x7673 bytes) 000554 1772758266.631800 mapwrite 0x01be03 thru 0x022d81 (0x6f7f bytes) 000557 1772758266.652969 mapread 0x00602b thru 0x00c7a4 (0x677a bytes) 000558 1772758266.653967 write 0x015267 thru 0x0223fb (0xd195 bytes) 000562 1772758266.656812 mapread 0x003723 thru 0x00e868 (0xb146 bytes) 000563 1772758266.657752 trunc from 0x040000 to 0x027954 (0x186ab bytes) 000564 1772758266.666813 MIRROR_EXTEND to 2 mirrors 000566 1772758266.854145 MIRROR_EXTEND to 3 mirrors 000567 1772758267.019055 mapread 0x017350 thru 0x0241f9 (0xceaa bytes) 000569 1772758267.031892 mapwrite 0x003f37 thru 0x00e156 (0xa220 bytes) 000570 1772758267.087887 mapread 0x026375 thru 0x027954 (0x15e0 bytes) 000572 1772758267.091870 read 0x001dda thru 0x002893 (0xaba bytes) 000573 1772758267.092504 write 0x00bc23 thru 0x00d62d (0x1a0b bytes) 000575 1772758267.093558 read 0x01c19d thru 0x021293 (0x50f7 bytes) 000577 1772758267.097633 write 0x00fdfc thru 0x01bae4 (0xbce9 bytes) 000578 1772758267.098402 trunc from 0x027955 to 0x015173 (0x127e1 bytes) 000579 1772758267.110846 write 0x03fdc0 thru 0x03ffff (0x240 bytes) 000580 1772758267.113841 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3087474f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3087474f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000581 1772758267.202741 mapwrite 0x002a63 thru 0x003c82 (0x1220 bytes) 000583 1772758267.219375 mapread 0x0289a5 thru 0x034d2c (0xc388 bytes) 000584 1772758267.226060 write 0x0136f4 thru 0x01a68e (0x6f9b bytes) 000586 1772758267.231260 trunc from 0x040000 to 0x00656f (0x39a90 bytes) 000587 1772758267.237377 mapwrite 0x021237 thru 0x02634b (0x5115 bytes) 000588 1772758267.259964 mapread 0x00a61f thru 0x00dd04 (0x36e6 bytes) 000590 1772758267.265496 trunc from 0x02634c to 0x0155f7 (0x10d54 bytes) 000591 1772758267.270373 mapwrite 0x01ffa4 thru 0x0228bb (0x2918 bytes) 000592 1772758267.292257 mapwrite 0x02799d thru 0x0362a2 (0xe906 bytes) 000594 1772758267.323892 read 0x01b7f8 thru 0x028a01 (0xd20a bytes) 000595 1772758267.324517 read 0x00fb22 thru 0x0193c2 (0x98a1 bytes) 000597 1772758267.328380 trunc from 0x0362a3 to 0x025321 (0x10f81 bytes) 000602 1772758267.334650 mapread 0x00a8be thru 0x01070b (0x5e4e bytes) 000603 1772758267.337799 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25322): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25322): Mirror 3: 0x1c6156a5 (stale) CRC-32 checksum value for chunk [0, 0x25322): Mirror 1: 0x27f45ab1 Mirror 4: 0x27f45ab1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25322: skipped 000604 1772758267.567979 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25322): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25322): Mirror 3: 0x1c6156a5 (stale) CRC-32 checksum value for chunk [0, 0x25322): Mirror 1: 0x27f45ab1 Mirror 4: 0x27f45ab1 Mirror 5: 0x27f45ab1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25322: skipped 000609 1772758267.812650 read 0x01f3ca thru 0x025321 (0x5f58 bytes) 000611 1772758267.818946 read 0x00c17c thru 0x011260 (0x50e5 bytes) 000613 1772758267.823810 read 0x00b31b thru 0x00c3ca (0x10b0 bytes) 000616 1772758267.827636 read 0x0222e9 thru 0x025321 (0x3039 bytes) 000617 1772758267.828963 mapwrite 0x036a1a thru 0x03ffff (0x95e6 bytes) 000618 1772758267.870950 mapwrite 0x02fb9b thru 0x0391e2 (0x9648 bytes) 000620 1772758267.885165 read 0x02c7a2 thru 0x02fcaa (0x3509 bytes) 000622 1772758267.886009 write 0x038b87 thru 0x03ffff (0x7479 bytes) 000623 1772758267.886735 read 0x039796 thru 0x03ffff (0x686a bytes) 000625 1772758267.887508 trunc from 0x040000 to 0x01012e (0x2fed1 bytes) 000626 1772758267.893781 write 0x02d2cc thru 0x0305c9 (0x32fe bytes) 000630 1772758267.895644 trunc from 0x0305ca to 0x025b7f (0xaa4a bytes) 000633 1772758267.900544 mapread 0x0089d0 thru 0x0105a4 (0x7bd5 bytes) 000634 1772758267.901246 trunc from 0x025b80 to 0x008c02 (0x1cf7d bytes) 000635 1772758267.905381 mapread 0x00413e thru 0x004cda (0xb9d bytes) 000640 1772758267.908093 mapread 0x0048dc thru 0x0066d2 (0x1df7 bytes) 000642 1772758267.909017 mapread 0x0032dc thru 0x008c02 (0x5927 bytes) 000643 1772758267.909914 write 0x00765b thru 0x013def (0xc795 bytes) 000645 1772758267.911146 write 0x021185 thru 0x02bab8 (0xa934 bytes) 000646 1772758267.913913 MIRROR_RESYNC 4 mirrors 000647 1772758268.178795 mapwrite 0x01e3e0 thru 0x02d57e (0xf19f bytes) 000649 1772758268.228478 write 0x0003f3 thru 0x00ddb9 (0xd9c7 bytes) 000650 1772758268.231902 write 0x016a76 thru 0x01efcd (0x8558 bytes) 000651 1772758268.234484 mapread 0x0144d3 thru 0x01bc37 (0x7765 bytes) 000652 1772758268.236897 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d57f): Mirror 4: 0x3de05714 (stale) Mirror 5: 0x3de05714 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d57f): Mirror 4: 0x3de05714 (stale) Mirror 5: 0x3de05714 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d57f: skipped 000654 1772758268.336961 write 0x02e64e thru 0x033068 (0x4a1b bytes) 000656 1772758268.340498 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33069): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33069): Mirror 4: 0xafff6868 (stale) Mirror 5: 0xafff6868 (stale) CRC-32 checksum value for chunk [0, 0x33069): Mirror 1: 0x9a00ffe1 Mirror 6: 0x9a00ffe1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33069: skipped 000657 1772758268.625738 write 0x005d9f thru 0x010972 (0xabd4 bytes) 000658 1772758268.650643 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33069): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33069): Mirror 4: 0xafff6868 (stale) Mirror 5: 0xafff6868 (stale) Mirror 6: 0x9a00ffe1 (stale) CRC-32 checksum value for chunk [0, 0x33069): Mirror 1: 0x94031fb Mirror 7: 0x94031fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33069: skipped 000659 1772758268.964099 read 0x004976 thru 0x00c340 (0x79cb bytes) 000661 1772758268.974062 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33069): Mirror 5: 0xafff6868 (stale) Mirror 6: 0x9a00ffe1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33069): Mirror 5: 0xafff6868 (stale) Mirror 6: 0x9a00ffe1 (stale) CRC-32 checksum value for chunk [0, 0x33069): Mirror 1: 0x94031fb Mirror 7: 0x94031fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33069: skipped 000663 1772758269.237268 mapread 0x00207d thru 0x0077ab (0x572f bytes) 000665 1772758269.266410 mapwrite 0x0182a0 thru 0x01a0be (0x1e1f bytes) 000668 1772758269.335058 mapwrite 0x00415c thru 0x0074d2 (0x3377 bytes) 000669 1772758269.354089 mapwrite 0x00db91 thru 0x010e90 (0x3300 bytes) 000671 1772758269.369084 mapwrite 0x017c67 thru 0x01d3ba (0x5754 bytes) 000673 1772758269.389415 trunc from 0x033069 to 0x029fa2 (0x90c6 bytes) 000674 1772758269.400554 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29fa3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29fa3): Mirror 5: 0xa77447c6 (stale) Mirror 6: 0xd001adfa (stale) Mirror 7: 0x1342f782 (stale) CRC-32 checksum value for chunk [0, 0x29fa3): Mirror 1: 0x70f3f3a2 Mirror 8: 0x70f3f3a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29fa3: skipped 000678 1772758269.722074 MIRROR_RESYNC 5 mirrors 000679 1772758269.988197 read 0x00af90 thru 0x00f88d (0x48fe bytes) 000680 1772758269.991938 read 0x00bb52 thru 0x0124a2 (0x6951 bytes) 000681 1772758269.995767 mapread 0x008329 thru 0x013da7 (0xba7f bytes) 000682 1772758269.996399 write 0x0156c6 thru 0x01de1a (0x8755 bytes) 000683 1772758270.020240 write 0x038d52 thru 0x03ffff (0x72ae bytes) 000684 1772758270.021496 mapread 0x0118b6 thru 0x0178d6 (0x6021 bytes) 000685 1772758270.029925 read 0x018aed thru 0x027333 (0xe847 bytes) 000688 1772758270.035570 write 0x038047 thru 0x03ffff (0x7fb9 bytes) 000689 1772758270.036291 write 0x015cdc thru 0x01be73 (0x6198 bytes) 000691 1772758270.037368 mapwrite 0x03a9aa thru 0x03dd33 (0x338a bytes) 000692 1772758270.056012 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadd9f696 (stale) Mirror 6: 0xadd9f696 (stale) Mirror 7: 0xadd9f696 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadd9f696 (stale) Mirror 6: 0xadd9f696 (stale) Mirror 7: 0xadd9f696 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000695 1772758270.194012 write 0x03f98c thru 0x03ffff (0x674 bytes) 000696 1772758270.197267 read 0x01fb67 thru 0x020932 (0xdcc bytes) 000697 1772758270.199942 mapwrite 0x00a36d thru 0x00ace1 (0x975 bytes) 000700 1772758270.221222 write 0x03e873 thru 0x03ffff (0x178d bytes) 000704 1772758270.229611 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xadd9f696 (stale) Mirror 6: 0xadd9f696 (stale) Mirror 7: 0xadd9f696 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x43bb8630 Mirror 8: 0x43bb8630 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000706 1772758270.500471 write 0x001f3b thru 0x0108b4 (0xe97a bytes) 000708 1772758270.531157 mapwrite 0x023950 thru 0x030819 (0xceca bytes) 000710 1772758270.562025 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xadd9f696 (stale) Mirror 7: 0xadd9f696 (stale) Mirror 8: 0x43bb8630 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xadd9f696 (stale) Mirror 7: 0xadd9f696 (stale) Mirror 8: 0x43bb8630 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000711 1772758270.706225 MIRROR_RESYNC 4 mirrors 000712 1772758270.997277 write 0x0220e3 thru 0x02e067 (0xbf85 bytes) 000713 1772758271.023504 mapwrite 0x03644f thru 0x03ffff (0x9bb1 bytes) 000714 1772758271.047064 mapread 0x03dbf5 thru 0x03ffff (0x240b bytes) 000715 1772758271.047803 trunc from 0x040000 to 0x00963e (0x369c1 bytes) 000716 1772758271.061313 read 0x003a61 thru 0x00963e (0x5bde bytes) 000718 1772758271.065514 read 0x0093cc thru 0x00963e (0x273 bytes) 000721 1772758271.067306 mapwrite 0x000b67 thru 0x001073 (0x50d bytes) 000722 1772758271.086472 mapwrite 0x00313f thru 0x011a5d (0xe91f bytes) 000724 1772758271.119195 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x11a5e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11a5e): Mirror 6: 0x9c5f4906 (stale) Mirror 7: 0x9c5f4906 (stale) Mirror 8: 0x9c5f4906 (stale) CRC-32 checksum value for chunk [0, 0x11a5e): Mirror 1: 0x9bb03456 Mirror 9: 0x9bb03456 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11a5e: skipped 000727 1772758271.407952 mapwrite 0x01d17c thru 0x0275f9 (0xa47e bytes) 000729 1772758271.465861 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x275fa): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x275fa): Mirror 6: 0x613f0221 (stale) Mirror 7: 0x613f0221 (stale) Mirror 8: 0x613f0221 (stale) Mirror 9: 0x9f976a74 (stale) CRC-32 checksum value for chunk [0, 0x275fa): Mirror 1: 0xad6294f0 Mirror 10: 0xad6294f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x275fa: skipped 000730 1772758271.770426 write 0x03b30b thru 0x03ffff (0x4cf5 bytes) 000731 1772758271.798670 mapread 0x011ba7 thru 0x015b46 (0x3fa0 bytes) 000732 1772758271.805446 read 0x02e13d thru 0x030ee4 (0x2da8 bytes) 000733 1772758271.808356 trunc from 0x040000 to 0x02e216 (0x11de9 bytes) 000735 1772758271.821965 MIRROR_RESYNC 6 mirrors 000736 1772758272.160876 mapwrite 0x035faf thru 0x03c8ed (0x693f bytes) 000737 1772758272.211016 write 0x022083 thru 0x026038 (0x3fb6 bytes) 000738 1772758272.217772 MIRROR_RESYNC 6 mirrors 000743 1772758272.568677 mapread 0x019331 thru 0x027fa5 (0xec75 bytes) 000745 1772758272.583995 read 0x02ebfb thru 0x0349ce (0x5dd4 bytes) 000747 1772758272.594812 mapread 0x01d761 thru 0x0217d2 (0x4072 bytes) 000748 1772758272.595573 mapread 0x017635 thru 0x025c1f (0xe5eb bytes) 000749 1772758272.598540 MIRROR_RESYNC 6 mirrors 000753 1772758272.797230 trunc from 0x03c8ee to 0x03069c (0xc251 bytes) 000754 1772758272.829357 mapwrite 0x0383d0 thru 0x03a2f2 (0x1f23 bytes) 000755 1772758272.860950 trunc from 0x03a2f3 to 0x015af9 (0x247f9 bytes) 000757 1772758272.866893 read 0x01320e thru 0x013b61 (0x954 bytes) 000758 1772758272.869166 mapread 0x008242 thru 0x00e15a (0x5f19 bytes) 000759 1772758272.873550 trunc from 0x015afa to 0x02bf75 (0x1647c bytes) 000761 1772758272.879972 write 0x0086a8 thru 0x00ef22 (0x687b bytes) 000762 1772758272.883237 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bf76): Mirror 6: 0x4cb6a7e9 (stale) Mirror 7: 0x4cb6a7e9 (stale) Mirror 8: 0x4cb6a7e9 (stale) Mirror 10: 0x4cb6a7e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf76): Mirror 6: 0x4cb6a7e9 (stale) Mirror 7: 0x4cb6a7e9 (stale) Mirror 8: 0x4cb6a7e9 (stale) Mirror 10: 0x4cb6a7e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf76: skipped 000763 1772758273.041370 mapread 0x01d2ab thru 0x02661f (0x9375 bytes) 000764 1772758273.049905 mapwrite 0x026f58 thru 0x033951 (0xc9fa bytes) 000766 1772758273.081313 mapwrite 0x03856c thru 0x03ffff (0x7a94 bytes) 000770 1772758273.106870 mapread 0x010510 thru 0x0134fe (0x2fef bytes) 000777 1772758273.116435 mapread 0x015a24 thru 0x01a722 (0x4cff bytes) 000781 1772758273.119372 trunc from 0x040000 to 0x00cd4d (0x332b2 bytes) 000783 1772758273.127372 mapread 0x001688 thru 0x00bb28 (0xa4a1 bytes) 000784 1772758273.132794 mapwrite 0x015f40 thru 0x020bb4 (0xac75 bytes) 000786 1772758273.166261 mapwrite 0x023e48 thru 0x02404e (0x207 bytes) 000788 1772758273.194382 mapwrite 0x00c86f thru 0x0147e2 (0x7f74 bytes) 000789 1772758273.216803 MIRROR_RESYNC 5 mirrors 000790 1772758273.531507 mapwrite 0x030f9b thru 0x03ffe3 (0xf049 bytes) 000792 1772758273.586123 write 0x03a06e thru 0x03ffff (0x5f92 bytes) 000794 1772758273.588541 write 0x0263ef thru 0x031150 (0xad62 bytes) 000795 1772758273.593214 read 0x039c41 thru 0x03af55 (0x1315 bytes) 000796 1772758273.593748 read 0x006e5c thru 0x00f658 (0x87fd bytes) 000797 1772758273.601061 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x117a1470 (stale) Mirror 7: 0x117a1470 (stale) Mirror 8: 0x117a1470 (stale) Mirror 10: 0x117a1470 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x611986c6 Mirror 11: 0x611986c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000798 1772758273.893164 write 0x03e407 thru 0x03ffff (0x1bf9 bytes) 000799 1772758273.920206 mapwrite 0x0056d3 thru 0x00a774 (0x50a2 bytes) 000801 1772758273.947891 mapwrite 0x02a0a4 thru 0x02cae8 (0x2a45 bytes) 000802 1772758273.978332 trunc from 0x040000 to 0x009fda (0x36025 bytes) 000803 1772758274.001804 trunc from 0x009fdb to 0x012b01 (0x8b27 bytes) 000809 1772758274.018379 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12b02): Mirror 6: 0x922fc1d2 (stale) Mirror 8: 0x922fc1d2 (stale) Mirror 10: 0x922fc1d2 (stale) Mirror 11: 0x922fc1d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12b02): Mirror 6: 0x922fc1d2 (stale) Mirror 8: 0x922fc1d2 (stale) Mirror 10: 0x922fc1d2 (stale) Mirror 11: 0x922fc1d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12b02: skipped 000810 1772758274.166206 read 0x00b582 thru 0x0111ca (0x5c49 bytes) 000811 1772758274.169854 trunc from 0x012b02 to 0x01ffe3 (0xd4e2 bytes) 000814 1772758274.176381 trunc from 0x01ffe4 to 0x00e291 (0x11d52 bytes) 000815 1772758274.182724 trunc from 0x00e292 to 0x00c8d6 (0x19bb bytes) 000817 1772758274.191741 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc8d7): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc8d7): Mirror 6: 0xdd7b6235 (stale) Mirror 8: 0xdd7b6235 (stale) Mirror 10: 0xdd7b6235 (stale) Mirror 11: 0xdd7b6235 (stale) CRC-32 checksum value for chunk [0, 0xc8d7): Mirror 1: 0x51d0541f Mirror 12: 0x51d0541f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc8d7: skipped 000818 1772758274.463069 write 0x012465 thru 0x014b6a (0x2706 bytes) 000819 1772758274.491090 MIRROR_RESYNC 6 mirrors 000820 1772758274.828949 mapwrite 0x01559a thru 0x024f5c (0xf9c3 bytes) 000824 1772758274.884841 write 0x008347 thru 0x00fb88 (0x7842 bytes) 000825 1772758274.888966 mapwrite 0x01872f thru 0x01985f (0x1131 bytes) 000826 1772758274.902695 read 0x016ea9 thru 0x020832 (0x998a bytes) 000828 1772758274.903820 mapwrite 0x01f253 thru 0x0256a3 (0x6451 bytes) 000829 1772758274.926654 write 0x02842b thru 0x037bbb (0xf791 bytes) 000830 1772758274.928906 read 0x02bab0 thru 0x02cd77 (0x12c8 bytes) 000832 1772758274.930157 read 0x008992 thru 0x01528f (0xc8fe bytes) 000837 1772758274.933803 read 0x01ad67 thru 0x0229b3 (0x7c4d bytes) 000839 1772758274.934888 mapread 0x020638 thru 0x029d12 (0x96db bytes) 000843 1772758274.939568 mapread 0x0171b6 thru 0x0226c1 (0xb50c bytes) 000844 1772758274.940236 trunc from 0x037bbc to 0x0359e8 (0x21d3 bytes) 000845 1772758274.945431 read 0x009d39 thru 0x011bc2 (0x7e8a bytes) 000849 1772758274.947866 write 0x033517 thru 0x03d7cc (0xa2b6 bytes) 000850 1772758274.949764 trunc from 0x03d7cd to 0x026f80 (0x1684c bytes) 000853 1772758274.956165 mapread 0x023a9d thru 0x024f22 (0x1486 bytes) 000854 1772758274.956823 read 0x011b28 thru 0x01532d (0x3806 bytes) 000856 1772758274.957616 mapread 0x00b3d9 thru 0x017b24 (0xc74c bytes) 000857 1772758274.960601 MIRROR_RESYNC 6 mirrors 000862 1772758275.291091 read 0x0173bf thru 0x01daff (0x6741 bytes) 000864 1772758275.295969 write 0x00b1ab thru 0x014b6a (0x99c0 bytes) 000865 1772758275.325941 mapwrite 0x01f1e0 thru 0x02a3a8 (0xb1c9 bytes) 000870 1772758275.364998 trunc from 0x02a3a9 to 0x0255bc (0x4dec bytes) 000871 1772758275.371672 trunc from 0x0255bd to 0x00941e (0x1c19e bytes) 000873 1772758275.378844 read 0x0017fe thru 0x0081cb (0x69ce bytes) 000874 1772758275.383500 mapread 0x003d01 thru 0x00941e (0x571e bytes) 000876 1772758275.384700 mapread 0x0045dc thru 0x00941e (0x4e43 bytes) 000878 1772758275.386135 read 0x0063f2 thru 0x00941e (0x302d bytes) 000879 1772758275.387039 write 0x024f13 thru 0x03338c (0xe47a bytes) 000880 1772758275.390793 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3338d): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3338d): Mirror 6: 0xad96afc4 (stale) Mirror 8: 0xad96afc4 (stale) Mirror 10: 0xad96afc4 (stale) Mirror 11: 0xad96afc4 (stale) Mirror 12: 0xad96afc4 (stale) CRC-32 checksum value for chunk [0, 0x3338d): Mirror 1: 0x48d48af6 Mirror 13: 0x48d48af6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3338d: skipped 000883 1772758275.665612 mapread 0x019e50 thru 0x024e1d (0xafce bytes) 000885 1772758275.680832 mapread 0x006976 thru 0x016421 (0xfaac bytes) 000889 1772758275.696151 mapread 0x0323e7 thru 0x03338c (0xfa6 bytes) 000890 1772758275.702562 mapwrite 0x00d01b thru 0x00e479 (0x145f bytes) 000891 1772758275.775032 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3338d): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3338d): Mirror 6: 0xad96afc4 (stale) Mirror 8: 0xad96afc4 (stale) Mirror 10: 0xad96afc4 (stale) Mirror 11: 0xad96afc4 (stale) Mirror 12: 0xad96afc4 (stale) Mirror 13: 0x48d48af6 (stale) CRC-32 checksum value for chunk [0, 0x3338d): Mirror 1: 0x64db6a90 Mirror 14: 0x64db6a90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3338d: skipped 000892 1772758276.050922 mapwrite 0x027eb1 thru 0x02d7d1 (0x5921 bytes) 000893 1772758276.120942 mapwrite 0x024081 thru 0x02778f (0x370f bytes) 000894 1772758276.138110 trunc from 0x03338d to 0x02cdac (0x65e0 bytes) 000896 1772758276.152952 mapread 0x01517b thru 0x020d6b (0xbbf1 bytes) 000897 1772758276.160503 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2cdad): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdad): Mirror 6: 0x521a3879 (stale) Mirror 8: 0x521a3879 (stale) Mirror 10: 0x521a3879 (stale) Mirror 11: 0x521a3879 (stale) Mirror 12: 0x521a3879 (stale) Mirror 13: 0x35786b5c (stale) Mirror 14: 0x8d8bb9ad (stale) CRC-32 checksum value for chunk [0, 0x2cdad): Mirror 1: 0x324bb53b Mirror 15: 0x324bb53b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdad: skipped 000898 1772758276.430368 read 0x0043eb thru 0x012402 (0xe018 bytes) 000903 1772758276.441033 mapwrite 0x003fd1 thru 0x0049cb (0x9fb bytes) 000906 1772758276.513926 trunc from 0x02cdad to 0x00d98c (0x1f420 bytes) 000907 1772758276.527081 write 0x031db1 thru 0x03eab2 (0xcd02 bytes) 000908 1772758276.528382 read 0x0045fa thru 0x00c4d6 (0x7edd bytes) 000909 1772758276.529087 mapwrite 0x026e92 thru 0x02f9e6 (0x8b55 bytes) 000911 1772758276.549853 mapread 0x020baa thru 0x0232c2 (0x2719 bytes) 000913 1772758276.551760 write 0x01f4b6 thru 0x02024c (0xd97 bytes) 000914 1772758276.552665 trunc from 0x03eab3 to 0x00fc4c (0x2ee66 bytes) 000915 1772758276.559064 write 0x018a41 thru 0x0230e0 (0xa6a0 bytes) 000920 1772758276.565906 MIRROR_RESYNC 9 mirrors 000921 1772758276.978397 mapread 0x002668 thru 0x003420 (0xdb9 bytes) 000929 1772758276.984808 mapwrite 0x003694 thru 0x00cad7 (0x9444 bytes) 000931 1772758277.041527 trunc from 0x0230e1 to 0x03c574 (0x19494 bytes) 000932 1772758277.057971 mapread 0x022673 thru 0x026a8d (0x441b bytes) 000933 1772758277.065065 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3c575): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c575): Mirror 6: 0xbf41896d (stale) Mirror 8: 0xbf41896d (stale) Mirror 10: 0xbf41896d (stale) Mirror 11: 0xbf41896d (stale) Mirror 12: 0xbf41896d (stale) Mirror 13: 0xbf41896d (stale) Mirror 14: 0xbf41896d (stale) Mirror 15: 0xbf41896d (stale) CRC-32 checksum value for chunk [0, 0x3c575): Mirror 1: 0xec6b2ca6 Mirror 16: 0xec6b2ca6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c575: skipped 000936 1772758277.390116 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c575): Mirror 8: 0xbf41896d (stale) Mirror 10: 0xbf41896d (stale) Mirror 11: 0xbf41896d (stale) Mirror 12: 0xbf41896d (stale) Mirror 13: 0xbf41896d (stale) Mirror 14: 0xbf41896d (stale) Mirror 15: 0xbf41896d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c575): Mirror 8: 0xbf41896d (stale) Mirror 10: 0xbf41896d (stale) Mirror 11: 0xbf41896d (stale) Mirror 12: 0xbf41896d (stale) Mirror 13: 0xbf41896d (stale) Mirror 14: 0xbf41896d (stale) Mirror 15: 0xbf41896d (stale) CRC-32 checksum value for chunk [0, 0x3c575): Mirror 1: 0xec6b2ca6 Mirror 16: 0xec6b2ca6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c575: skipped 000939 1772758277.709874 trunc from 0x03c575 to 0x02bd32 (0x10842 bytes) 000941 1772758277.744894 mapread 0x003ce8 thru 0x00c6f8 (0x8a11 bytes) 000943 1772758277.752697 mapwrite 0x02c8fa thru 0x02e344 (0x1a4b bytes) 000944 1772758277.784629 mapread 0x00c6c3 thru 0x0120a7 (0x59e5 bytes) 000945 1772758277.791472 read 0x000291 thru 0x00cc6b (0xc9db bytes) 000946 1772758277.795069 mapread 0x006c78 thru 0x015e93 (0xf21c bytes) 000949 1772758277.797765 mapread 0x00b8c7 thru 0x00f433 (0x3b6d bytes) 000950 1772758277.799499 mapread 0x026913 thru 0x02e11c (0x780a bytes) 000951 1772758277.801000 read 0x02676a thru 0x02c34f (0x5be6 bytes) 000952 1772758277.804404 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e345): Mirror 8: 0x41ac62e1 (stale) Mirror 10: 0x41ac62e1 (stale) Mirror 11: 0x41ac62e1 (stale) Mirror 13: 0x41ac62e1 (stale) Mirror 14: 0x41ac62e1 (stale) Mirror 15: 0x41ac62e1 (stale) Mirror 16: 0xac22c616 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e345): Mirror 8: 0x41ac62e1 (stale) Mirror 10: 0x41ac62e1 (stale) Mirror 11: 0x41ac62e1 (stale) Mirror 13: 0x41ac62e1 (stale) Mirror 14: 0x41ac62e1 (stale) Mirror 15: 0x41ac62e1 (stale) Mirror 16: 0xac22c616 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e345: skipped 000953 1772758277.980941 write 0x01f71e thru 0x023d0e (0x45f1 bytes) 000956 1772758277.987696 mapread 0x006c0d thru 0x0142c0 (0xd6b4 bytes) 000958 1772758277.997392 write 0x027688 thru 0x03062e (0x8fa7 bytes) 000959 1772758278.005631 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3062f): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3062f): Mirror 8: 0x64614b1f (stale) Mirror 10: 0x64614b1f (stale) Mirror 11: 0x64614b1f (stale) Mirror 13: 0x64614b1f (stale) Mirror 14: 0x64614b1f (stale) Mirror 15: 0x64614b1f (stale) Mirror 16: 0x6ceabe06 (stale) CRC-32 checksum value for chunk [0, 0x3062f): Mirror 1: 0xe19428ab Mirror 17: 0xe19428ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3062f: skipped 000960 1772758278.431334 MIRROR_RESYNC 9 mirrors 000963 1772758278.888089 trunc from 0x03062f to 0x00ac96 (0x25998 bytes) 000964 1772758278.911389 mapwrite 0x032be7 thru 0x03bb71 (0x8f8b bytes) 000965 1772758278.941277 write 0x0189f5 thru 0x0205cf (0x7bdb bytes) 000966 1772758278.947741 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 8: 0x612ac9e (stale) Mirror 10: 0x612ac9e (stale) Mirror 11: 0x612ac9e (stale) Mirror 13: 0x612ac9e (stale) Mirror 14: 0x612ac9e (stale) Mirror 15: 0x612ac9e (stale) Mirror 16: 0x612ac9e (stale) Mirror 17: 0x612ac9e (stale) CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 1: 0x692d1b22 Mirror 18: 0x692d1b22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb72: skipped 000967 1772758279.306021 write 0x013bdc thru 0x01bcf0 (0x8115 bytes) 000968 1772758279.334717 mapwrite 0x03836b thru 0x038aa6 (0x73c bytes) 000971 1772758279.357667 read 0x0286d4 thru 0x031bbd (0x94ea bytes) 000972 1772758279.363942 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4ea:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 10: 0x612ac9e (stale) Mirror 11: 0x612ac9e (stale) Mirror 13: 0x612ac9e (stale) Mirror 14: 0x612ac9e (stale) Mirror 15: 0x612ac9e (stale) Mirror 16: 0x612ac9e (stale) Mirror 17: 0x612ac9e (stale) Mirror 18: 0x692d1b22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb72): Mirror 10: 0x612ac9e (stale) Mirror 11: 0x612ac9e (stale) Mirror 13: 0x612ac9e (stale) Mirror 14: 0x612ac9e (stale) Mirror 15: 0x612ac9e (stale) Mirror 16: 0x612ac9e (stale) Mirror 17: 0x612ac9e (stale) Mirror 18: 0x692d1b22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb72: skipped 000974 1772758279.580619 write 0x03aea1 thru 0x03ffff (0x515f bytes) 000975 1772758279.584388 trunc from 0x040000 to 0x011291 (0x2ed6e bytes) 000976 1772758279.592953 mapread 0x009367 thru 0x011291 (0x7f2b bytes) 000980 1772758279.604938 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x11292): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11292): Mirror 10: 0x3748a969 (stale) Mirror 11: 0x3748a969 (stale) Mirror 13: 0x3748a969 (stale) Mirror 14: 0x3748a969 (stale) Mirror 15: 0x3748a969 (stale) Mirror 16: 0x3748a969 (stale) Mirror 17: 0x3748a969 (stale) Mirror 18: 0x1543f990 (stale) CRC-32 checksum value for chunk [0, 0x11292): Mirror 1: 0x1543f990 Mirror 19: 0x1543f990 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11292: skipped 000984 1772758279.922790 mapwrite 0x01900c thru 0x01a086 (0x107b bytes) 000986 1772758279.982532 MIRROR_RESYNC 10 mirrors 000987 1772758280.452192 write 0x02e576 thru 0x02fde0 (0x186b bytes) 000991 1772758280.473896 mapread 0x023ce7 thru 0x02fde0 (0xc0fa bytes) 000994 1772758280.486901 mapwrite 0x020a56 thru 0x02c9dc (0xbf87 bytes) 000995 1772758280.512882 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2fde1): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fde1): Mirror 10: 0x330a7121 (stale) Mirror 11: 0x330a7121 (stale) Mirror 13: 0x330a7121 (stale) Mirror 14: 0x330a7121 (stale) Mirror 15: 0x330a7121 (stale) Mirror 16: 0x330a7121 (stale) Mirror 17: 0x330a7121 (stale) Mirror 18: 0x330a7121 (stale) Mirror 19: 0x330a7121 (stale) CRC-32 checksum value for chunk [0, 0x2fde1): Mirror 1: 0x4cf98564 Mirror 20: 0x4cf98564 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fde1: skipped 000996 1772758280.843329 mapread 0x02c159 thru 0x02fde0 (0x3c88 bytes) 000998 1772758280.854213 mapread 0x019477 thru 0x024a79 (0xb603 bytes) 000999 1772758280.867200 mapwrite 0x0342ec thru 0x035dc0 (0x1ad5 bytes) 001000 1772758280.929668 trunc from 0x035dc1 to 0x006a85 (0x2f33b bytes) All operations completed A-OK! PASS 70a (28s) == sanity-flr test 71: check mirror extend parent fid ==== 19:51:23 (1772758283) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 19:51:24 (1772758284) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106584 s, 19.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679066 s, 15.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x71e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x71c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 19:51:28 (1772758288) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3453472 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8130112 .. Extending file size to 7874912 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8352896 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8693248 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8352256 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8876768 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9121888 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9153472 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8752320 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8773152 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8833312 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 57673 57674 57676 57678 57680 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) 192.168.201.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:52:35 (1772758355) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2341440 .. Extending file size to 1400640 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5459776 .. Extending file size to 7593184 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6482656 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7529024 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8399552 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7693728 .. Extending file size to 8273184 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8927456 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8316896 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9367680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61004 61005 61006 61009 61011 done 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) 192.168.201.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:53:41 (1772758421) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4736 2203520 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0460805 s, 45.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0979605 s, 21.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa046a2ee (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:53:48 (1772758428) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0421588 s, 49.7 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 19:53:51 (1772758431) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x537:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x735:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x735:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x736:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x731:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:53:54 (1772758434) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0651401 s, 32.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x737:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x732:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x733:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:53:57 (1772758437) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0630582 s, 33.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x738:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x739:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x734:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x738:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x739:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x734:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x735:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:54:00 (1772758440) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0700686 s, 29.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x736:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x736:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x737:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:54:25 (1772758465) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246882 s, 42.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x739:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x738:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x739:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x738:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73c:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:55:12 (1772758512) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0435663 s, 48.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:55:58 (1772758558) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg140-server mds-ost sync done. Creating new pool test_204f oleg140-server: Pool lustre.test_204f created Adding targets to pool oleg140-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg140-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0499064 s, 42.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x73c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x73e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg140-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg140-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg140-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:56:58 (1772758618) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:57:02 (1772758622) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:57:05 (1772758625) oleg140-server: Pool lustre.test_206 created Adding targets to pool oleg140-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg140-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg140-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg140-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg140-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:57:19 (1772758639) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376 s, 27.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191819 s, 54.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xebb43495 Mirror 3: 0xebb43495 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:57:24 (1772758644) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:57:26 (1772758646) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:57:27 (1772758647) Starting client: oleg140-client.virtnet: -o user_xattr,flock 192.168.201.140@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0348978 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x920000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1800000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1800000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1800000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9a0000 Read this (4 bytes): BBBB 192.168.201.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg140-client.virtnet /mnt/lustre2 (opts:) PASS 209a (20s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:57:47 (1772758667) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00395318 s, 2.1 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:57:50 (1772758670) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208505 s, 50.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x55c lmm_fid: [0x200000404:0x55c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x55c lmm_fid: [0x200000404:0x55c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x748:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835292 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-05 19:57:51.000000000 -0500 Modify: 2026-03-05 19:57:51.000000000 -0500 Change: 2026-03-05 19:57:52.000000000 -0500 Birth: 2026-03-05 19:57:51.000000000 -0500 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:57:54 (1772758674) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:57:56 (1772758676) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.211438 s, 194 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0293337 s, 140 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 5 19:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x55e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 5 19:57 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:57:59 (1772758679) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:58:02 (1772758682) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 845 sec ============ 19:58:06 (1772758686) === sanity-flr: start cleanup 19:58:07 (1772758687) === === sanity-flr: finish cleanup 19:58:09 (1772758689) ===