-----============= acceptance-small: sanity-flr ============----- Tue Jun 16 09:32:00 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg118-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:32:14 (1781616734) === oleg118-client.virtnet: executing check_config_client /mnt/lustre oleg118-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg118-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f870abcc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f870abcc800.idle_timeout=debug disable quota as required oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:32:41 (1781616761) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7633] == sanity-flr test 0a: lfs mirror create with -N option == 09:32:42 (1781616762) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:32:55 (1781616775) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:32:58 (1781616778) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:33:00 (1781616780) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:33:19 (1781616799) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:33:21 (1781616801) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 09:33:24 (1781616804) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f870abcc800.stats=clear osc.lustre-OST0001-osc-ffff9f870abcc800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:33:29 (1781616809) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:33:37 (1781616817) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:33:43 (1781616823) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:33:46 (1781616826) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:33:51 (1781616831) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 09:33:58 (1781616838) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:34:03 (1781616843) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:34:05 (1781616845) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:34:11 (1781616851) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:34:18 (1781616858) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:34:23 (1781616863) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.379535 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142075 s, 7.4 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:34:31 (1781616871) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0981124 s, 64.1 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:34:37 (1781616877) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:34:45 (1781616885) Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (47s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:35:32 (1781616932) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.147735 s, 28.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (26s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:35:58 (1781616958) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (46s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:36:44 (1781617004) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.77086 s, 18.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 16 09:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.731s user 0m0.001s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f870abcc800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.609s user 0m0.000s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.702s user 0m0.001s sys 0m0.178s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f870abcc800.ost_server_uuid in FULL state after 0 sec PASS 33b (54s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:37:38 (1781617058) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:37:41 (1781617061) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:37:43 (1781617063) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:37:45 (1781617065) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 09:37:50 (1781617070) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.351426 s, 35.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:37:58 (1781617078) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179309 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234103 s, 44.8 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:38:01 (1781617081) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:38:07 (1781617087) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0046917 s, 218 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108 s, 948 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000869838 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000988901 s, 1.0 MB/s flv=10 olv=10 PASS 36d (7s) == sanity-flr test 37: mirror I/O API verification ======= 09:38:13 (1781617093) create test files with size 15729833 .. bc3f6a04252b428eafc9bae3b899d8c0 - ce1397b6e6cfc13c6c54d911b43a0e7c - 997d615a7494c1080c73aaeb2a8a954f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre PASS 37 (12s) == sanity-flr test 38: resync ============================ 09:38:25 (1781617105) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:38:33 (1781617113) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:38:36 (1781617116) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627863 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190434 s, 55.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 09:38:41 (1781617121) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0546948 s, 38.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.193298 s, 21.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0776396 s, 27.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 09:38:48 (1781617128) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 09:38:50 (1781617130) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:38:52 (1781617132) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176972 s, 59.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 09:38:55 (1781617135) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:38:57 (1781617137) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:39:02 (1781617142) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.115446 s, 90.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:39:17 (1781617157) llite.lustre-ffff9f870572a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.161246 s, 65.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f8705706800.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:39:24 (1781617164) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (20s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:39:44 (1781617184) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (18s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:40:02 (1781617202) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 09:40:04 (1781617204) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.107221 s, 29.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.078823 s, 39.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x84dc83d3 (nosync,stale) Mirror 2: 0x84dc83d3 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 24fade584d9e629128dff99243d6c4ae - mirror 1 checksum: 38f58c9641dac578a9e4550c639e6177 - mirror 2 checksum: 24fade584d9e629128dff99243d6c4ae - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-16T09:40:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-16T09:40:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 24fade584d9e629128dff99243d6c4ae - mirror 1 checksum: 38f58c9641dac578a9e4550c639e6177 - mirror 2 checksum: 38f58c9641dac578a9e4550c639e6177 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-16T09:40:05-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-16T09:40:05-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:40:09 (1781617209) Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 09:40:13 (1781617213) shut down facet: mds1 facet_host: oleg118-server facet_failover_host: oleg118-server Failover mds1 to oleg118-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 09:40:28 (1781617228) targets are mounted 09:40:28 (1781617228) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166721 s, 62.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:40:38 (1781617238) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:40:40 (1781617240) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:40:42 (1781617242) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00588231 s, 2.8 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:40:47 (1781617247) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:40:49 (1781617249) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:40:50 (1781617250) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (12s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:41:02 (1781617262) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:41:23 (1781617283) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:41:41 (1781617301) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:42:00 (1781617320) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 09:42:03 (1781617323) mirror create pid 53720 mirror split pid 53722 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53720 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 53722 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 09:43:06 (1781617386) Seed set to 7387 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1781617387.291993 write 0x00ffcf thru 0x0132e6 (0x3318 bytes) 000002 1781617387.296110 mapwrite 0x02cbd2 thru 0x03193f (0x4d6e bytes) 000003 1781617387.333002 mapwrite 0x034e8e thru 0x03ffff (0xb172 bytes) 000004 1781617387.376666 read 0x008c62 thru 0x0164ce (0xd86d bytes) 000005 1781617387.385980 read 0x03bdbe thru 0x03e6ca (0x290d bytes) 000006 1781617387.386673 read 0x01aef1 thru 0x01e775 (0x3885 bytes) 000007 1781617387.392919 mapread 0x0324a6 thru 0x036572 (0x40cd bytes) 000009 1781617387.394373 mapwrite 0x017b9f thru 0x019642 (0x1aa4 bytes) truncating to largest ever: 0xe884 000011 1781617387.421666 trunc from 0x040000 to 0x00e883 (0x3177c bytes) 000013 1781617387.431754 mapwrite 0x01109c thru 0x01f915 (0xe87a bytes) 000015 1781617387.513821 read 0x01212d thru 0x015d54 (0x3c28 bytes) 000017 1781617387.515280 mapread 0x015197 thru 0x019eb6 (0x4d20 bytes) 000018 1781617387.516237 write 0x027581 thru 0x032bcc (0xb64c bytes) 000019 1781617387.517738 mapread 0x009aa6 thru 0x00ca88 (0x2fe3 bytes) 000020 1781617387.518584 mapwrite 0x01081d thru 0x01874b (0x7f2f bytes) 000021 1781617387.545574 mapread 0x01581e thru 0x019c72 (0x4455 bytes) truncating to largest ever: 0x2580b 000024 1781617387.548065 trunc from 0x032bcd to 0x02580a (0xd3c2 bytes) 000025 1781617387.555567 read 0x018d0a thru 0x02580a (0xcb01 bytes) 000027 1781617387.556672 mapwrite 0x013c8b thru 0x02012e (0xc4a4 bytes) 000028 1781617387.586355 read 0x00269a thru 0x007c2e (0x5595 bytes) 000032 1781617387.588284 read 0x012d89 thru 0x01eeab (0xc123 bytes) 000034 1781617387.591476 mapread 0x00b678 thru 0x00c10b (0xa94 bytes) 000035 1781617387.592126 mapread 0x017f30 thru 0x0221d5 (0xa2a6 bytes) truncating to largest ever: 0x3d54d 000036 1781617387.593142 trunc from 0x02580b to 0x03d54c (0x17d42 bytes) 000037 1781617387.599850 trunc from 0x03d54d to 0x0214b2 (0x1c09a bytes) 000038 1781617387.606114 trunc from 0x0214b3 to 0x007cc2 (0x197f0 bytes) 000039 1781617387.612159 mapwrite 0x005bd1 thru 0x0084bd (0x28ed bytes) 000040 1781617387.639578 write 0x037a67 thru 0x03ffff (0x8599 bytes) 000043 1781617387.642513 write 0x005aba thru 0x014c14 (0xf15b bytes) 000044 1781617387.646720 read 0x037ee7 thru 0x03ffff (0x8119 bytes) 000046 1781617387.647960 mapread 0x0242f9 thru 0x033015 (0xed1d bytes) 000048 1781617387.653777 write 0x012fb2 thru 0x0223da (0xf429 bytes) 000051 1781617387.657335 read 0x033a25 thru 0x03ffff (0xc5db bytes) 000053 1781617387.658527 mapread 0x027783 thru 0x02ab5a (0x33d8 bytes) 000055 1781617387.660181 read 0x0145e8 thru 0x01dcff (0x9718 bytes) 000057 1781617387.663719 MIRROR_EXTEND to 2 mirrors 000058 1781617387.883687 mapwrite 0x03a461 thru 0x03ffff (0x5b9f bytes) 000059 1781617387.946553 mapwrite 0x012a83 thru 0x014229 (0x17a7 bytes) 000060 1781617387.963347 read 0x01f9a9 thru 0x020058 (0x6b0 bytes) 000061 1781617387.966111 mapread 0x007fc9 thru 0x00d8b9 (0x58f1 bytes) 000062 1781617387.970255 mapread 0x00154c thru 0x00a518 (0x8fcd bytes) 000063 1781617387.973729 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcef45ecb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcd9461d8 Mirror 3: 0xcd9461d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000066 1781617388.324578 read 0x035f05 thru 0x03ffff (0xa0fb bytes) 000067 1781617388.331501 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcef45ecb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcd9461d8 Mirror 3: 0xcd9461d8 Mirror 4: 0xcd9461d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1781617388.652599 mapread 0x0357cd thru 0x03b94a (0x617e bytes) 000069 1781617388.665995 mapread 0x02d038 thru 0x03caa1 (0xfa6a bytes) 000070 1781617388.677819 mapread 0x025cbc thru 0x02e192 (0x84d7 bytes) 000074 1781617388.699904 mapread 0x03c496 thru 0x03e19c (0x1d07 bytes) 000075 1781617388.706536 trunc from 0x040000 to 0x0328ab (0xd754 bytes) 000076 1781617388.736920 mapwrite 0x03c7b4 thru 0x03ffff (0x384c bytes) 000077 1781617388.770997 mapread 0x02f54b thru 0x03cd8a (0xd840 bytes) 000080 1781617388.776706 mapwrite 0x033d15 thru 0x03dece (0xa1ba bytes) 000081 1781617388.796001 read 0x01d1c6 thru 0x01f04e (0x1e89 bytes) 000082 1781617388.801299 mapwrite 0x029aaf thru 0x02d587 (0x3ad9 bytes) 000083 1781617388.827561 mapwrite 0x001f6f thru 0x008db0 (0x6e42 bytes) 000084 1781617388.856180 trunc from 0x040000 to 0x00f488 (0x30b77 bytes) 000086 1781617388.865043 read 0x00c7d1 thru 0x00f488 (0x2cb8 bytes) 000088 1781617388.866362 mapread 0x00d776 thru 0x00f488 (0x1d13 bytes) 000090 1781617388.868581 read 0x00c71e thru 0x00f488 (0x2d6b bytes) 000091 1781617388.869241 mapwrite 0x009d74 thru 0x018af0 (0xed7d bytes) 000092 1781617388.914868 mapwrite 0x011626 thru 0x016dc0 (0x579b bytes) 000093 1781617388.933497 write 0x01136f thru 0x020d48 (0xf9da bytes) 000094 1781617388.934887 read 0x00431b thru 0x00c8d1 (0x85b7 bytes) 000096 1781617388.935852 mapwrite 0x035157 thru 0x03f77e (0xa628 bytes) 000099 1781617388.970608 mapwrite 0x0012ae thru 0x0068fe (0x5651 bytes) 000101 1781617388.987934 read 0x03bdfb thru 0x03f77e (0x3984 bytes) 000103 1781617388.989324 write 0x015257 thru 0x01ca3f (0x77e9 bytes) truncating to largest ever: 0x3d804 000106 1781617388.991849 trunc from 0x03f77f to 0x03d803 (0x1f7b bytes) 000107 1781617388.999774 mapwrite 0x0226b9 thru 0x02c9a8 (0xa2f0 bytes) 000108 1781617389.018344 read 0x02f405 thru 0x03d1f9 (0xddf5 bytes) 000109 1781617389.019153 mapwrite 0x0073d9 thru 0x00c4a2 (0x50ca bytes) 000112 1781617389.050394 mapwrite 0x02f120 thru 0x03d7fb (0xe6dc bytes) 000114 1781617389.075826 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d804): Mirror 2: 0xd7b75b5f (stale) Mirror 4: 0xe23c4a2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d804): Mirror 2: 0xd7b75b5f (stale) Mirror 4: 0xe23c4a2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d804: skipped 000115 1781617389.214025 MIRROR_RESYNC 3 mirrors 000116 1781617389.475594 read 0x00dcb9 thru 0x01585e (0x7ba6 bytes) truncating to largest ever: 0x3e313 000117 1781617389.479809 trunc from 0x03d804 to 0x03e312 (0xb0f bytes) 000118 1781617389.513008 mapwrite 0x011b8c thru 0x01dda1 (0xc216 bytes) 000120 1781617389.556922 mapwrite 0x03eff5 thru 0x03ffff (0x100b bytes) 000121 1781617389.590010 MIRROR_RESYNC 3 mirrors 000122 1781617389.829637 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) 000124 1781617389.954541 mapread 0x02d9ec thru 0x035bab (0x81c0 bytes) 000126 1781617389.962648 mapwrite 0x01b8eb thru 0x02360d (0x7d23 bytes) 000129 1781617390.031608 mapread 0x03595b thru 0x0388f6 (0x2f9c bytes) 000131 1781617390.037007 read 0x010acc thru 0x0166f3 (0x5c28 bytes) 000132 1781617390.040866 read 0x02231f thru 0x026659 (0x433b bytes) 000134 1781617390.045465 write 0x02ae36 thru 0x03234f (0x751a bytes) 000135 1781617390.049640 MIRROR_RESYNC 2 mirrors 000136 1781617390.283978 trunc from 0x040000 to 0x0323b3 (0xdc4c bytes) 000137 1781617390.346550 write 0x0295c2 thru 0x033bf9 (0xa638 bytes) 000138 1781617390.361083 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) 000140 1781617390.423832 mapread 0x01063e thru 0x01faca (0xf48d bytes) 000143 1781617390.439448 read 0x02b65d thru 0x033bf9 (0x859d bytes) 000145 1781617390.445044 mapread 0x001d50 thru 0x0023e3 (0x694 bytes) 000146 1781617390.449601 mapread 0x02eebf thru 0x033bf9 (0x4d3b bytes) truncating to largest ever: 0x3ec92 000148 1781617390.450913 trunc from 0x033bfa to 0x03ec91 (0xb098 bytes) 000149 1781617390.467061 trunc from 0x03ec92 to 0x0028cb (0x3c3c6 bytes) 000150 1781617390.478219 mapwrite 0x03d813 thru 0x03d825 (0x13 bytes) 000153 1781617390.526137 MIRROR_EXTEND to 2 mirrors 000154 1781617390.807639 read 0x030bf4 thru 0x03b868 (0xac75 bytes) 000155 1781617390.813723 trunc from 0x03d826 to 0x01f45c (0x1e3c9 bytes) 000156 1781617390.857908 trunc from 0x01f45d to 0x02e3bb (0xef5f bytes) 000157 1781617390.867257 mapread 0x00f745 thru 0x019560 (0x9e1c bytes) 000158 1781617390.878736 mapread 0x02b85e thru 0x02e3bb (0x2b5e bytes) 000161 1781617390.884735 trunc from 0x02e3bc to 0x037172 (0x8db7 bytes) 000163 1781617390.895645 mapwrite 0x02574c thru 0x0322cc (0xcb81 bytes) 000165 1781617390.929495 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37173): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37173): Mirror 2: 0x3c1d25f7 (stale) CRC-32 checksum value for chunk [0, 0x37173): Mirror 1: 0xe3d7e7b3 Mirror 3: 0xe3d7e7b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37173: skipped 000166 1781617391.298857 mapwrite 0x01b8af thru 0x02592e (0xa080 bytes) 000167 1781617391.380268 read 0x012ebf thru 0x017761 (0x48a3 bytes) 000168 1781617391.384367 mapwrite 0x03caf6 thru 0x03ffff (0x350a bytes) 000169 1781617391.430140 mapwrite 0x02991c thru 0x035748 (0xbe2d bytes) 000171 1781617391.463547 MIRROR_RESYNC 3 mirrors 000173 1781617391.754630 write 0x027887 thru 0x027fa1 (0x71b bytes) 000175 1781617391.785252 write 0x0197f5 thru 0x025420 (0xbc2c bytes) 000178 1781617391.792812 mapwrite 0x004417 thru 0x010827 (0xc411 bytes) 000179 1781617391.829400 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbdc9bb53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbdc9bb53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000180 1781617391.959537 mapwrite 0x009d0f thru 0x0155e2 (0xb8d4 bytes) 000181 1781617391.995139 mapread 0x02d38b thru 0x0308c0 (0x3536 bytes) 000182 1781617392.006539 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) 000184 1781617392.071487 trunc from 0x040000 to 0x02205e (0x1dfa1 bytes) 000189 1781617392.102973 MIRROR_EXTEND to 2 mirrors 000190 1781617392.376382 read 0x001619 thru 0x00cba1 (0xb589 bytes) 000192 1781617392.386549 write 0x0260de thru 0x028c44 (0x2b67 bytes) 000193 1781617392.407187 trunc from 0x028c45 to 0x031107 (0x84c3 bytes) 000194 1781617392.416033 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) 000195 1781617392.484312 mapread 0x0176bd thru 0x01b6e4 (0x4028 bytes) 000196 1781617392.494754 trunc from 0x031108 to 0x01f3fe (0x11d09 bytes) 000197 1781617392.501534 read 0x012d22 thru 0x01f3fe (0xc6dd bytes) 000198 1781617392.502060 write 0x03699d thru 0x039d09 (0x336d bytes) 000200 1781617392.503498 mapwrite 0x010ff7 thru 0x019857 (0x8861 bytes) 000201 1781617392.528721 mapwrite 0x038c23 thru 0x038e0c (0x1ea bytes) 000204 1781617392.545918 MIRROR_EXTEND to 2 mirrors 000205 1781617392.739965 read 0x00d882 thru 0x0184f0 (0xac6f bytes) 000207 1781617392.749262 MIRROR_RESYNC 2 mirrors 000209 1781617392.881352 mapread 0x00f070 thru 0x01158f (0x2520 bytes) 000212 1781617392.888836 read 0x02ac3a thru 0x0334f9 (0x88c0 bytes) 000214 1781617392.897863 write 0x0237be thru 0x030c78 (0xd4bb bytes) 000215 1781617392.932194 mapread 0x00c1a7 thru 0x019a47 (0xd8a1 bytes) 000216 1781617392.951646 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39d0a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d0a): Mirror 2: 0xdecbce46 (stale) CRC-32 checksum value for chunk [0, 0x39d0a): Mirror 1: 0xc2bb5ef1 Mirror 3: 0xc2bb5ef1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d0a: skipped 000217 1781617393.286518 MIRROR_RESYNC 3 mirrors 000218 1781617393.546236 mapwrite 0x03b078 thru 0x03e4d2 (0x345b bytes) 000219 1781617393.611187 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 3: 0x5f327739 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 3: 0x5f327739 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d3: skipped 000220 1781617393.742682 mapread 0x02261f thru 0x03243e (0xfe20 bytes) 000221 1781617393.755891 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 3: 0x5f327739 (stale) CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 1: 0xeaa59c63 Mirror 4: 0xeaa59c63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d3: skipped 000224 1781617394.090529 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 3: 0x5f327739 (stale) CRC-32 checksum value for chunk [0, 0x3e4d3): Mirror 1: 0xeaa59c63 Mirror 4: 0xeaa59c63 Mirror 5: 0xeaa59c63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4d3: skipped 000226 1781617394.480442 mapwrite 0x03e249 thru 0x03ffff (0x1db7 bytes) 000227 1781617394.542716 mapread 0x02c3a7 thru 0x0374a0 (0xb0fa bytes) 000229 1781617394.556267 trunc from 0x040000 to 0x010c90 (0x2f36f bytes) 000233 1781617394.572960 read 0x0032b0 thru 0x008fea (0x5d3b bytes) 000234 1781617394.583989 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10c91): Mirror 4: 0xd03c5b56 (stale) Mirror 5: 0xd03c5b56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10c91): Mirror 4: 0xd03c5b56 (stale) Mirror 5: 0xd03c5b56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10c91: skipped 000235 1781617394.737578 read 0x00436d thru 0x00fcfd (0xb991 bytes) 000236 1781617394.744709 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10c91): Mirror 4: 0xd03c5b56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10c91): Mirror 4: 0xd03c5b56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10c91: skipped 000237 1781617394.882701 trunc from 0x010c91 to 0x0236b7 (0x12a27 bytes) 000238 1781617394.893418 MIRROR_RESYNC 2 mirrors 000244 1781617395.207027 write 0x01d801 thru 0x029486 (0xbc86 bytes) 000245 1781617395.232421 trunc from 0x029487 to 0x015556 (0x13f30 bytes) 000249 1781617395.249141 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) 000254 1781617395.323355 read 0x00eccc thru 0x0128a5 (0x3bda bytes) 000257 1781617395.329950 write 0x016ed8 thru 0x01ba83 (0x4bac bytes) 000258 1781617395.331102 mapwrite 0x038493 thru 0x03ffff (0x7b6d bytes) 000259 1781617395.377342 mapread 0x0155b8 thru 0x01f68a (0xa0d3 bytes) 000261 1781617395.391445 MIRROR_EXTEND to 2 mirrors 000262 1781617395.647366 mapwrite 0x032264 thru 0x03e24f (0xbfec bytes) 000263 1781617395.747063 write 0x03d0a6 thru 0x03ffff (0x2f5a bytes) 000265 1781617395.748784 read 0x02b812 thru 0x0315e3 (0x5dd2 bytes) 000268 1781617395.753746 write 0x03c389 thru 0x03ce8c (0xb04 bytes) 000270 1781617395.755220 write 0x018d40 thru 0x025b02 (0xcdc3 bytes) 000271 1781617395.762273 read 0x0338fe thru 0x03ffff (0xc702 bytes) 000275 1781617395.764531 write 0x023dfc thru 0x02b5ae (0x77b3 bytes) 000280 1781617395.767693 read 0x0149f3 thru 0x016403 (0x1a11 bytes) 000282 1781617395.774352 read 0x00f05d thru 0x0166db (0x767f bytes) 000283 1781617395.774954 mapread 0x03cc57 thru 0x03ffff (0x33a9 bytes) 000284 1781617395.775790 mapwrite 0x03827c thru 0x03b29b (0x3020 bytes) 000286 1781617395.797013 mapread 0x021c90 thru 0x0288eb (0x6c5c bytes) 000287 1781617395.798094 mapwrite 0x02a2da thru 0x036ee2 (0xcc09 bytes) 000288 1781617395.833428 write 0x02a548 thru 0x02e55d (0x4016 bytes) 000289 1781617395.834503 mapwrite 0x005ea2 thru 0x0065e3 (0x742 bytes) 000290 1781617395.858484 MIRROR_RESYNC 2 mirrors 000292 1781617396.182311 write 0x00d784 thru 0x0122c4 (0x4b41 bytes) 000293 1781617396.219396 MIRROR_RESYNC 2 mirrors 000294 1781617396.538720 write 0x02d622 thru 0x03d5f1 (0xffd0 bytes) 000295 1781617396.569923 mapread 0x02097e thru 0x02f178 (0xe7fb bytes) 000296 1781617396.585728 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x78a31827 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a509cdf Mirror 3: 0x9a509cdf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000297 1781617396.943433 mapread 0x01ff0c thru 0x02d368 (0xd45d bytes) 000299 1781617396.960696 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) 000300 1781617397.154748 mapread 0x034889 thru 0x03ce3d (0x85b5 bytes) 000301 1781617397.176561 mapread 0x00e548 thru 0x0163a4 (0x7e5d bytes) 000302 1781617397.192654 mapwrite 0x031618 thru 0x03b1ec (0x9bd5 bytes) 000303 1781617397.281353 mapwrite 0x004d75 thru 0x00a94e (0x5bda bytes) 000305 1781617397.312127 write 0x03494f thru 0x03ffff (0xb6b1 bytes) 000306 1781617397.313424 mapwrite 0x030316 thru 0x0392a4 (0x8f8f bytes) 000310 1781617397.347331 trunc from 0x040000 to 0x02b547 (0x14ab8 bytes) 000312 1781617397.379602 read 0x01366f thru 0x020b73 (0xd505 bytes) 000313 1781617397.388768 write 0x019a88 thru 0x0267da (0xcd53 bytes) 000315 1781617397.399446 read 0x02ae2b thru 0x02b4ce (0x6a4 bytes) 000316 1781617397.400123 read 0x009cdc thru 0x00baa2 (0x1dc7 bytes) 000317 1781617397.400591 trunc from 0x02b548 to 0x032039 (0x6af2 bytes) 000321 1781617397.415095 read 0x024842 thru 0x03029d (0xba5c bytes) 000322 1781617397.422052 read 0x00f0b8 thru 0x01d228 (0xe171 bytes) 000323 1781617397.430078 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3203a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3203a): Mirror 3: 0xbc7f5993 (stale) CRC-32 checksum value for chunk [0, 0x3203a): Mirror 1: 0x6c40b910 Mirror 4: 0x6c40b910 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3203a: skipped 000324 1781617397.762589 write 0x03f3d9 thru 0x03ffff (0xc27 bytes) 000325 1781617397.801015 write 0x0353f8 thru 0x03ad2e (0x5937 bytes) 000328 1781617397.817678 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x801b1989 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x801b1989 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000329 1781617397.969407 mapwrite 0x021dc5 thru 0x022596 (0x7d2 bytes) 000330 1781617397.989989 mapread 0x0205c8 thru 0x02add4 (0xa80d bytes) 000331 1781617397.998690 trunc from 0x040000 to 0x0301ee (0xfe11 bytes) 000332 1781617398.012360 write 0x022edf thru 0x027caa (0x4dcc bytes) 000333 1781617398.013397 trunc from 0x0301ef to 0x03e58d (0xe39f bytes) 000334 1781617398.023874 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e58e): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e58e): Mirror 4: 0x544045db (stale) CRC-32 checksum value for chunk [0, 0x3e58e): Mirror 1: 0xcaeb3973 Mirror 5: 0xcaeb3973 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e58e: skipped 000335 1781617398.415420 MIRROR_RESYNC 3 mirrors 000336 1781617398.720792 read 0x03ca11 thru 0x03e58d (0x1b7d bytes) 000337 1781617398.727110 trunc from 0x03e58e to 0x00273c (0x3be51 bytes) 000341 1781617398.767452 trunc from 0x00273d to 0x03c58c (0x39e50 bytes) 000342 1781617398.778245 mapwrite 0x00a690 thru 0x0119c4 (0x7335 bytes) 000344 1781617398.810637 trunc from 0x03c58d to 0x01bec5 (0x206c7 bytes) 000345 1781617398.820060 MIRROR_RESYNC 3 mirrors 000346 1781617399.118482 write 0x006811 thru 0x00f03f (0x882f bytes) 000347 1781617399.154906 write 0x031c6e thru 0x031f1f (0x2b2 bytes) 000348 1781617399.156100 write 0x02e4b7 thru 0x038cad (0xa7f7 bytes) 000349 1781617399.163757 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38cae): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38cae): Mirror 4: 0xafd24150 (stale) Mirror 5: 0xafd24150 (stale) CRC-32 checksum value for chunk [0, 0x38cae): Mirror 1: 0xdb5d9b04 Mirror 6: 0xdb5d9b04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38cae: skipped 000351 1781617399.517859 trunc from 0x038cae to 0x010813 (0x2849a bytes) 000352 1781617399.552565 mapwrite 0x038df2 thru 0x03b934 (0x2b43 bytes) truncating to largest ever: 0x3f091 000353 1781617399.588833 trunc from 0x03b935 to 0x03f090 (0x375c bytes) 000355 1781617399.595921 trunc from 0x03f091 to 0x015830 (0x29860 bytes) 000356 1781617399.603317 mapwrite 0x02b732 thru 0x03b269 (0xfb38 bytes) 000360 1781617399.645580 trunc from 0x03b26a to 0x037493 (0x3dd6 bytes) 000365 1781617399.655787 trunc from 0x037494 to 0x02117a (0x16319 bytes) 000366 1781617399.664410 mapread 0x018605 thru 0x01ae53 (0x284f bytes) 000367 1781617399.671550 mapread 0x010769 thru 0x01d629 (0xcec1 bytes) 000368 1781617399.677018 mapread 0x010157 thru 0x012e9b (0x2d45 bytes) 000374 1781617399.680879 mapread 0x0188e1 thru 0x01c860 (0x3f80 bytes) 000378 1781617399.683381 read 0x01a0f1 thru 0x02117a (0x708a bytes) 000380 1781617399.689899 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2117b): Mirror 5: 0xa3f030e (stale) Mirror 6: 0x6ed21db3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2117b): Mirror 5: 0xa3f030e (stale) Mirror 6: 0x6ed21db3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2117b: skipped 000381 1781617399.824999 trunc from 0x02117b to 0x0081a2 (0x18fd8 bytes) 000382 1781617399.832526 trunc from 0x0081a3 to 0x010f72 (0x8dd0 bytes) 000383 1781617399.840721 write 0x00db7b thru 0x01bc93 (0xe119 bytes) 000385 1781617399.844729 write 0x00ba6a thru 0x0133a6 (0x793d bytes) 000386 1781617399.848111 mapwrite 0x008bad thru 0x0105a7 (0x79fb bytes) 000387 1781617399.881577 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1bc94): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc94): Mirror 5: 0x909717d2 (stale) Mirror 6: 0x37b0a4f2 (stale) CRC-32 checksum value for chunk [0, 0x1bc94): Mirror 1: 0x491a0643 Mirror 7: 0x491a0643 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc94: skipped 000388 1781617400.209783 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bc94): Mirror 5: 0x909717d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc94): Mirror 5: 0x909717d2 (stale) CRC-32 checksum value for chunk [0, 0x1bc94): Mirror 1: 0x491a0643 Mirror 7: 0x491a0643 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc94: skipped 000389 1781617400.396737 mapwrite 0x010f20 thru 0x015e57 (0x4f38 bytes) 000391 1781617400.478759 mapwrite 0x0237e9 thru 0x0310dd (0xd8f5 bytes) 000392 1781617400.521952 write 0x015d1e thru 0x016947 (0xc2a bytes) 000393 1781617400.522851 trunc from 0x0310de to 0x014f22 (0x1c1bb bytes) 000394 1781617400.531330 mapwrite 0x00128c thru 0x0058b9 (0x462e bytes) 000395 1781617400.565760 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x14f23): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14f23): Mirror 5: 0x6f578348 (stale) Mirror 7: 0xd6ba98d4 (stale) CRC-32 checksum value for chunk [0, 0x14f23): Mirror 1: 0xb8467caa Mirror 8: 0xb8467caa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14f23: skipped 000398 1781617400.915189 mapwrite 0x031b83 thru 0x03d23b (0xb6b9 bytes) 000399 1781617400.998846 MIRROR_RESYNC 4 mirrors 000404 1781617401.391871 write 0x01822f thru 0x026f69 (0xed3b bytes) 000407 1781617401.450377 MIRROR_RESYNC 4 mirrors 000409 1781617401.820101 mapwrite 0x03cb8c thru 0x03ffff (0x3474 bytes) 000413 1781617401.919287 MIRROR_RESYNC 4 mirrors 000416 1781617402.320627 mapwrite 0x010dab thru 0x011a65 (0xcbb bytes) 000417 1781617402.408449 trunc from 0x040000 to 0x013f78 (0x2c087 bytes) 000419 1781617402.420954 read 0x00c8a0 thru 0x013f78 (0x76d9 bytes) 000422 1781617402.422408 mapread 0x00f4f4 thru 0x0116e8 (0x21f5 bytes) 000425 1781617402.424590 trunc from 0x013f79 to 0x024646 (0x106ce bytes) 000428 1781617402.439259 read 0x00d0c7 thru 0x01c1dc (0xf116 bytes) 000429 1781617402.446644 trunc from 0x024647 to 0x02bcd0 (0x768a bytes) 000431 1781617402.455456 write 0x03908b thru 0x03ffff (0x6f75 bytes) 000432 1781617402.456597 read 0x02232d thru 0x022f11 (0xbe5 bytes) 000434 1781617402.457937 mapwrite 0x025d8e thru 0x03496b (0xebde bytes) 000437 1781617402.490827 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb9fe190 (stale) Mirror 7: 0xbb9fe190 (stale) Mirror 8: 0xbb9fe190 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xff049ea8 Mirror 9: 0xff049ea8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000440 1781617402.894462 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb9fe190 (stale) Mirror 7: 0xbb9fe190 (stale) Mirror 8: 0xbb9fe190 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xff049ea8 Mirror 9: 0xff049ea8 Mirror 10: 0xff049ea8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000442 1781617403.349254 mapread 0x0336df thru 0x03ffff (0xc921 bytes) 000443 1781617403.365178 mapwrite 0x01d148 thru 0x029f8c (0xce45 bytes) 000445 1781617403.447438 write 0x01818e thru 0x01cd9f (0x4c12 bytes) 000448 1781617403.449997 read 0x023874 thru 0x026fa6 (0x3733 bytes) 000450 1781617403.451284 mapread 0x030061 thru 0x03112e (0x10ce bytes) 000451 1781617403.457727 read 0x016507 thru 0x02514d (0xec47 bytes) 000452 1781617403.463650 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb9fe190 (stale) Mirror 7: 0xbb9fe190 (stale) Mirror 8: 0xbb9fe190 (stale) Mirror 9: 0xff049ea8 (stale) Mirror 10: 0xff049ea8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xccf575f Mirror 11: 0xccf575f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000454 1781617403.873950 mapwrite 0x00ac78 thru 0x01550c (0xa895 bytes) 000456 1781617403.957133 trunc from 0x040000 to 0x008e07 (0x371f8 bytes) 000457 1781617403.979718 write 0x01c557 thru 0x0257d5 (0x927f bytes) 000460 1781617403.983114 write 0x001ec0 thru 0x011c4f (0xfd90 bytes) 000462 1781617403.990988 mapread 0x0253f6 thru 0x0257d5 (0x3e0 bytes) 000465 1781617403.998375 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x257d6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x257d6): Mirror 5: 0x5cdd2158 (stale) Mirror 7: 0x5cdd2158 (stale) Mirror 8: 0x5cdd2158 (stale) Mirror 9: 0xfb65c057 (stale) Mirror 10: 0xfb65c057 (stale) Mirror 11: 0x6371c886 (stale) CRC-32 checksum value for chunk [0, 0x257d6): Mirror 1: 0xb78f9b41 Mirror 12: 0xb78f9b41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x257d6: skipped 000466 1781617404.431600 mapwrite 0x01b68d thru 0x0293ca (0xdd3e bytes) 000468 1781617404.499033 mapread 0x01d008 thru 0x0293ca (0xc3c3 bytes) 000470 1781617404.503011 MIRROR_RESYNC 8 mirrors 000471 1781617404.978657 trunc from 0x0293cb to 0x01dc1d (0xb7ad bytes) 000473 1781617405.021940 read 0x012504 thru 0x016f2d (0x4a2a bytes) 000475 1781617405.028639 read 0x0010e4 thru 0x007846 (0x6763 bytes) 000476 1781617405.033020 mapwrite 0x0147c8 thru 0x01c45b (0x7c94 bytes) 000477 1781617405.051878 read 0x01417b thru 0x019093 (0x4f19 bytes) 000478 1781617405.052741 mapwrite 0x00f51a thru 0x018280 (0x8d67 bytes) 000480 1781617405.073054 write 0x032c3a thru 0x034373 (0x173a bytes) 000482 1781617405.074763 mapwrite 0x00cc79 thru 0x016b83 (0x9f0b bytes) 000486 1781617405.100829 MIRROR_RESYNC 8 mirrors 000487 1781617405.586650 mapread 0x01a482 thru 0x01d707 (0x3286 bytes) 000489 1781617405.597288 read 0x02cd2d thru 0x0326fa (0x59ce bytes) 000490 1781617405.604454 MIRROR_EXTEND to 9 mirrors 000491 1781617405.901928 write 0x03c246 thru 0x03ffff (0x3dba bytes) 000492 1781617405.930580 write 0x0379ce thru 0x03dab2 (0x60e5 bytes) 000493 1781617405.936365 write 0x004449 thru 0x00fba0 (0xb758 bytes) 000494 1781617405.942853 write 0x01eb78 thru 0x02234f (0x37d8 bytes) 000496 1781617405.949018 write 0x0080d6 thru 0x01254b (0xa476 bytes) 000498 1781617405.953262 read 0x020401 thru 0x02c306 (0xbf06 bytes) 000499 1781617405.960076 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x233eb599 (stale) Mirror 7: 0x233eb599 (stale) Mirror 8: 0x233eb599 (stale) Mirror 9: 0x233eb599 (stale) Mirror 10: 0x233eb599 (stale) Mirror 11: 0x233eb599 (stale) Mirror 12: 0x233eb599 (stale) Mirror 13: 0x233eb599 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x924264b9 Mirror 14: 0x924264b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000501 1781617406.438515 MIRROR_RESYNC 10 mirrors 000503 1781617407.077986 mapwrite 0x03be5a thru 0x03ffff (0x41a6 bytes) 000506 1781617407.184965 mapwrite 0x018ed1 thru 0x023669 (0xa799 bytes) 000507 1781617407.219764 read 0x03428d thru 0x03b094 (0x6e08 bytes) 000510 1781617407.224897 read 0x038c57 thru 0x03ffff (0x73a9 bytes) 000511 1781617407.225862 mapwrite 0x02801a thru 0x02b4c3 (0x34aa bytes) 000512 1781617407.258973 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x924264b9 (stale) Mirror 8: 0x924264b9 (stale) Mirror 9: 0x924264b9 (stale) Mirror 10: 0x924264b9 (stale) Mirror 11: 0x924264b9 (stale) Mirror 12: 0x924264b9 (stale) Mirror 13: 0x924264b9 (stale) Mirror 14: 0x924264b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x924264b9 (stale) Mirror 8: 0x924264b9 (stale) Mirror 9: 0x924264b9 (stale) Mirror 10: 0x924264b9 (stale) Mirror 11: 0x924264b9 (stale) Mirror 12: 0x924264b9 (stale) Mirror 13: 0x924264b9 (stale) Mirror 14: 0x924264b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000516 1781617407.519988 read 0x0018fd thru 0x00b5e2 (0x9ce6 bytes) 000517 1781617407.528310 mapwrite 0x01c43b thru 0x02699f (0xa565 bytes) 000520 1781617407.578356 mapread 0x01be13 thru 0x027755 (0xb943 bytes) 000521 1781617407.584839 read 0x01ad2d thru 0x022ea2 (0x8176 bytes) 000523 1781617407.586263 mapread 0x01d5e9 thru 0x023646 (0x605e bytes) 000524 1781617407.587384 mapwrite 0x007b8f thru 0x009579 (0x19eb bytes) 000526 1781617407.608932 write 0x031464 thru 0x033cc0 (0x285d bytes) 000527 1781617407.616951 MIRROR_RESYNC 9 mirrors 000528 1781617408.265054 mapread 0x02533a thru 0x030713 (0xb3da bytes) 000529 1781617408.286913 trunc from 0x040000 to 0x002281 (0x3dd7e bytes) 000530 1781617408.356656 MIRROR_RESYNC 9 mirrors 000531 1781617409.114005 write 0x00444d thru 0x00b17b (0x6d2f bytes) 000534 1781617409.172647 read 0x001540 thru 0x00b17b (0x9c3c bytes) 000535 1781617409.179882 mapwrite 0x005433 thru 0x005733 (0x301 bytes) 000538 1781617409.224016 mapread 0x0069e9 thru 0x00b17b (0x4793 bytes) 000539 1781617409.225146 mapwrite 0x0290c9 thru 0x032482 (0x93ba bytes) 000543 1781617409.294051 mapread 0x00afdb thru 0x00db0c (0x2b32 bytes) 000544 1781617409.306931 read 0x01b892 thru 0x023918 (0x8087 bytes) 000545 1781617409.317429 read 0x0222c2 thru 0x02fc04 (0xd943 bytes) 000546 1781617409.318074 read 0x019920 thru 0x01f3b4 (0x5a95 bytes) 000548 1781617409.319142 mapwrite 0x007c03 thru 0x00fcb5 (0x80b3 bytes) 000551 1781617409.356635 MIRROR_RESYNC 9 mirrors 000552 1781617409.946416 MIRROR_RESYNC 9 mirrors 000553 1781617410.259981 mapwrite 0x02a491 thru 0x035528 (0xb098 bytes) 000556 1781617410.348992 trunc from 0x035529 to 0x01b194 (0x1a394 bytes) 000557 1781617410.357328 write 0x0206f9 thru 0x02fe6d (0xf775 bytes) 000558 1781617410.359085 read 0x01e37d thru 0x02ca1f (0xe6a3 bytes) 000559 1781617410.362095 trunc from 0x02fe6e to 0x0299f4 (0x6479 bytes) 000561 1781617410.381710 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x299f5): Mirror 5: 0xd14f6b57 (stale) Mirror 8: 0xd14f6b57 (stale) Mirror 9: 0xd14f6b57 (stale) Mirror 11: 0xd14f6b57 (stale) Mirror 12: 0xd14f6b57 (stale) Mirror 13: 0xd14f6b57 (stale) Mirror 14: 0xd14f6b57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x299f5): Mirror 5: 0xd14f6b57 (stale) Mirror 8: 0xd14f6b57 (stale) Mirror 9: 0xd14f6b57 (stale) Mirror 11: 0xd14f6b57 (stale) Mirror 12: 0xd14f6b57 (stale) Mirror 13: 0xd14f6b57 (stale) Mirror 14: 0xd14f6b57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x299f5: skipped 000562 1781617410.641717 trunc from 0x0299f5 to 0x0056ac (0x24348 bytes) 000566 1781617410.653164 mapread 0x00316e thru 0x0056ac (0x253f bytes) 000567 1781617410.660925 write 0x0051a6 thru 0x005acf (0x92a bytes) 000568 1781617410.661804 trunc from 0x005ad0 to 0x0211dc (0x1b70d bytes) 000570 1781617410.667751 trunc from 0x0211dd to 0x03c9ce (0x1b7f2 bytes) 000571 1781617410.672999 read 0x018671 thru 0x0239d9 (0xb369 bytes) 000574 1781617410.677497 trunc from 0x03c9cf to 0x015f2d (0x26aa1 bytes) 000575 1781617410.683563 read 0x0114e9 thru 0x013c72 (0x278a bytes) 000576 1781617410.688528 mapread 0x01286c thru 0x013505 (0xc9a bytes) 000578 1781617410.690010 mapread 0x00d287 thru 0x012c3c (0x59b6 bytes) 000580 1781617410.691784 write 0x00ce9a thru 0x00d193 (0x2fa bytes) 000581 1781617410.692687 write 0x0013e4 thru 0x005275 (0x3e92 bytes) 000590 1781617410.697307 write 0x02d7eb thru 0x02db4c (0x362 bytes) 000593 1781617410.699175 mapwrite 0x02732f thru 0x03142a (0xa0fc bytes) 000594 1781617410.739621 trunc from 0x03142b to 0x028ae4 (0x8946 bytes) 000597 1781617410.749055 read 0x01ca17 thru 0x024163 (0x774d bytes) 000598 1781617410.749843 mapwrite 0x01c12c thru 0x0235cb (0x74a0 bytes) 000599 1781617410.779588 mapread 0x024f25 thru 0x028ae4 (0x3bc0 bytes) 000600 1781617410.780526 mapread 0x003b6b thru 0x00d044 (0x94da bytes) 000601 1781617410.781620 mapread 0x00bb2d thru 0x01023a (0x470e bytes) 000602 1781617410.782592 mapread 0x001184 thru 0x00df1b (0xcd98 bytes) 000604 1781617410.784035 trunc from 0x028ae5 to 0x03c922 (0x13e3e bytes) 000606 1781617410.791989 write 0x034fef thru 0x03bb81 (0x6b93 bytes) 000609 1781617410.802565 trunc from 0x03c923 to 0x011cf7 (0x2ac2b bytes) 000610 1781617410.812347 mapwrite 0x00b489 thru 0x00d58a (0x2102 bytes) 000612 1781617410.834504 trunc from 0x011cf8 to 0x0352a8 (0x235b1 bytes) 000614 1781617410.844836 MIRROR_RESYNC 8 mirrors 000617 1781617411.382850 write 0x037753 thru 0x03d766 (0x6014 bytes) 000620 1781617411.414789 mapwrite 0x010a79 thru 0x014677 (0x3bff bytes) 000621 1781617411.472494 mapread 0x009ad6 thru 0x00b12c (0x1657 bytes) 000622 1781617411.477653 trunc from 0x03d767 to 0x0231ad (0x1a5b9 bytes) 000623 1781617411.486879 read 0x0157bc thru 0x01fc97 (0xa4dc bytes) 000624 1781617411.494618 mapwrite 0x0205c8 thru 0x02baf3 (0xb52c bytes) 000625 1781617411.530739 mapread 0x02a953 thru 0x02baf3 (0x11a1 bytes) 000626 1781617411.531664 write 0x013361 thru 0x0150fe (0x1d9e bytes) 000627 1781617411.535118 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2baf4): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2baf4): Mirror 5: 0x841bc1d8 (stale) Mirror 8: 0x841bc1d8 (stale) Mirror 9: 0x841bc1d8 (stale) Mirror 11: 0x841bc1d8 (stale) Mirror 12: 0x841bc1d8 (stale) Mirror 13: 0x841bc1d8 (stale) Mirror 14: 0x841bc1d8 (stale) CRC-32 checksum value for chunk [0, 0x2baf4): Mirror 1: 0x3f090ed3 Mirror 15: 0x3f090ed3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2baf4: skipped 000628 1781617412.001545 mapread 0x01a90b thru 0x01fd6e (0x5464 bytes) 000633 1781617412.022374 write 0x022a43 thru 0x0268d9 (0x3e97 bytes) 000638 1781617412.081609 trunc from 0x02baf4 to 0x00aec3 (0x20c30 bytes) 000639 1781617412.110159 write 0x039752 thru 0x03ffff (0x68ae bytes) 000641 1781617412.112037 write 0x01ac0e thru 0x021af9 (0x6eec bytes) 000643 1781617412.119283 write 0x0128d5 thru 0x014a21 (0x214d bytes) 000645 1781617412.128044 mapread 0x0190d4 thru 0x01ebab (0x5ad8 bytes) truncating to largest ever: 0x3f66b 000646 1781617412.137024 trunc from 0x040000 to 0x03f66a (0x995 bytes) 000648 1781617412.148079 mapwrite 0x0025be thru 0x009849 (0x728c bytes) 000650 1781617412.182604 mapwrite 0x03f044 thru 0x03faad (0xa6a bytes) 000651 1781617412.224592 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3faae): Mirror 5: 0x7c991353 (stale) Mirror 8: 0x7c991353 (stale) Mirror 9: 0x7c991353 (stale) Mirror 11: 0x7c991353 (stale) Mirror 13: 0x7c991353 (stale) Mirror 14: 0x7c991353 (stale) Mirror 15: 0x83193369 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3faae): Mirror 5: 0x7c991353 (stale) Mirror 8: 0x7c991353 (stale) Mirror 9: 0x7c991353 (stale) Mirror 11: 0x7c991353 (stale) Mirror 13: 0x7c991353 (stale) Mirror 14: 0x7c991353 (stale) Mirror 15: 0x83193369 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3faae: skipped 000652 1781617412.511893 mapread 0x02f5c3 thru 0x03e4b7 (0xeef5 bytes) 000655 1781617412.529670 read 0x01229f thru 0x021a88 (0xf7ea bytes) 000660 1781617412.536507 mapwrite 0x03abc9 thru 0x03ffff (0x5437 bytes) 000662 1781617412.572016 read 0x030143 thru 0x031c98 (0x1b56 bytes) 000665 1781617412.573972 mapwrite 0x027246 thru 0x0278f8 (0x6b3 bytes) 000666 1781617412.604760 mapread 0x01777e thru 0x022084 (0xa907 bytes) 000667 1781617412.609761 MIRROR_RESYNC 8 mirrors 000669 1781617413.209876 read 0x01758d thru 0x025595 (0xe009 bytes) 000670 1781617413.218373 mapwrite 0x00ffb9 thru 0x01bfeb (0xc033 bytes) 000671 1781617413.325988 read 0x00233a thru 0x00dc87 (0xb94e bytes) 000672 1781617413.333412 mapread 0x0285a6 thru 0x0371cb (0xec26 bytes) 000673 1781617413.334743 read 0x03cf61 thru 0x03ffff (0x309f bytes) 000674 1781617413.335341 trunc from 0x040000 to 0x024385 (0x1bc7a bytes) 000678 1781617413.353923 trunc from 0x024386 to 0x0231f6 (0x118f bytes) 000679 1781617413.364570 read 0x005722 thru 0x0137df (0xe0be bytes) 000680 1781617413.365387 trunc from 0x0231f7 to 0x03a0c6 (0x16ed0 bytes) 000683 1781617413.378778 write 0x010676 thru 0x013c62 (0x35ed bytes) 000684 1781617413.379874 write 0x0334f6 thru 0x034940 (0x144b bytes) 000686 1781617413.386085 trunc from 0x03a0c7 to 0x02da7d (0xc649 bytes) 000687 1781617413.399162 mapwrite 0x018c51 thru 0x02624c (0xd5fc bytes) 000689 1781617413.442306 read 0x003860 thru 0x011542 (0xdce3 bytes) 000691 1781617413.443526 write 0x00f0af thru 0x00f416 (0x368 bytes) 000692 1781617413.444812 mapread 0x026a21 thru 0x02b347 (0x4927 bytes) 000693 1781617413.446082 write 0x01a60d thru 0x01f683 (0x5077 bytes) 000694 1781617413.447161 mapwrite 0x03b331 thru 0x03ffff (0x4ccf bytes) 000695 1781617413.495442 read 0x011f4f thru 0x01acd2 (0x8d84 bytes) 000696 1781617413.496190 mapwrite 0x0342da thru 0x036880 (0x25a7 bytes) 000698 1781617413.521893 read 0x002e73 thru 0x0035c2 (0x750 bytes) 000699 1781617413.522594 mapread 0x0055d9 thru 0x006941 (0x1369 bytes) 000700 1781617413.523487 write 0x0179df thru 0x023def (0xc411 bytes) 000703 1781617413.529745 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5ab3af32 (stale) Mirror 8: 0x5ab3af32 (stale) Mirror 11: 0x5ab3af32 (stale) Mirror 13: 0x5ab3af32 (stale) Mirror 14: 0x5ab3af32 (stale) Mirror 15: 0x5ab3af32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5ab3af32 (stale) Mirror 8: 0x5ab3af32 (stale) Mirror 11: 0x5ab3af32 (stale) Mirror 13: 0x5ab3af32 (stale) Mirror 14: 0x5ab3af32 (stale) Mirror 15: 0x5ab3af32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000704 1781617413.816461 trunc from 0x040000 to 0x009360 (0x36c9f bytes) 000705 1781617413.835869 read 0x007fa5 thru 0x009360 (0x13bc bytes) 000706 1781617413.840688 mapwrite 0x0125c2 thru 0x01c662 (0xa0a1 bytes) 000707 1781617413.892712 read 0x007845 thru 0x0105e5 (0x8da1 bytes) 000709 1781617413.898730 read 0x0096cb thru 0x00ece0 (0x5616 bytes) 000711 1781617413.905467 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1c663): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c663): Mirror 5: 0x24d379b6 (stale) Mirror 8: 0x24d379b6 (stale) Mirror 11: 0x24d379b6 (stale) Mirror 13: 0x24d379b6 (stale) Mirror 14: 0x24d379b6 (stale) Mirror 15: 0x24d379b6 (stale) CRC-32 checksum value for chunk [0, 0x1c663): Mirror 1: 0x2db81af3 Mirror 16: 0x2db81af3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c663: skipped 000712 1781617414.533061 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1c663): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c663): Mirror 5: 0x24d379b6 (stale) Mirror 8: 0x24d379b6 (stale) Mirror 11: 0x24d379b6 (stale) Mirror 13: 0x24d379b6 (stale) Mirror 14: 0x24d379b6 (stale) Mirror 15: 0x24d379b6 (stale) CRC-32 checksum value for chunk [0, 0x1c663): Mirror 1: 0x2db81af3 Mirror 16: 0x2db81af3 Mirror 17: 0x2db81af3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c663: skipped 000713 1781617415.064118 write 0x011475 thru 0x0188be (0x744a bytes) 000717 1781617415.135143 mapread 0x015b82 thru 0x01985b (0x3cda bytes) 000718 1781617415.138761 read 0x00b8ab thru 0x0121de (0x6934 bytes) 000719 1781617415.144177 read 0x009f0f thru 0x00e332 (0x4424 bytes) 000720 1781617415.150869 read 0x008aee thru 0x009f12 (0x1425 bytes) 000721 1781617415.151755 read 0x001ef7 thru 0x00eb24 (0xcc2e bytes) 000722 1781617415.156360 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1c663): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c663): Mirror 5: 0x24d379b6 (stale) Mirror 8: 0x24d379b6 (stale) Mirror 11: 0x24d379b6 (stale) Mirror 13: 0x24d379b6 (stale) Mirror 14: 0x24d379b6 (stale) Mirror 15: 0x24d379b6 (stale) Mirror 16: 0x2db81af3 (stale) Mirror 17: 0x2db81af3 (stale) CRC-32 checksum value for chunk [0, 0x1c663): Mirror 1: 0xbb1b3d75 Mirror 18: 0xbb1b3d75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c663: skipped 000723 1781617415.640429 write 0x01d87a thru 0x0208fd (0x3084 bytes) 000724 1781617415.684011 trunc from 0x0208fe to 0x037ae4 (0x171e7 bytes) 000725 1781617415.716999 write 0x001f02 thru 0x0109de (0xeadd bytes) 000726 1781617415.733908 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ae5): Mirror 5: 0xb2ee7843 (stale) Mirror 11: 0xb2ee7843 (stale) Mirror 13: 0xb2ee7843 (stale) Mirror 14: 0xb2ee7843 (stale) Mirror 15: 0xb2ee7843 (stale) Mirror 16: 0x73b01e89 (stale) Mirror 17: 0x73b01e89 (stale) Mirror 18: 0xb000e6fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ae5): Mirror 5: 0xb2ee7843 (stale) Mirror 11: 0xb2ee7843 (stale) Mirror 13: 0xb2ee7843 (stale) Mirror 14: 0xb2ee7843 (stale) Mirror 15: 0xb2ee7843 (stale) Mirror 16: 0x73b01e89 (stale) Mirror 17: 0x73b01e89 (stale) Mirror 18: 0xb000e6fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ae5: skipped 000727 1781617416.096406 write 0x0271c5 thru 0x027c95 (0xad1 bytes) 000729 1781617416.102383 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ae5): Mirror 5: 0xb2ee7843 (stale) Mirror 13: 0xb2ee7843 (stale) Mirror 14: 0xb2ee7843 (stale) Mirror 15: 0xb2ee7843 (stale) Mirror 16: 0x73b01e89 (stale) Mirror 17: 0x73b01e89 (stale) Mirror 18: 0xb000e6fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ae5): Mirror 5: 0xb2ee7843 (stale) Mirror 13: 0xb2ee7843 (stale) Mirror 14: 0xb2ee7843 (stale) Mirror 15: 0xb2ee7843 (stale) Mirror 16: 0x73b01e89 (stale) Mirror 17: 0x73b01e89 (stale) Mirror 18: 0xb000e6fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ae5: skipped 000730 1781617416.378119 read 0x000a84 thru 0x001be7 (0x1164 bytes) 000731 1781617416.381761 trunc from 0x037ae5 to 0x0076ba (0x3042a bytes) 000734 1781617416.400744 mapwrite 0x00913c thru 0x017dfb (0xecc0 bytes) 000735 1781617416.459307 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17dfc): Mirror 5: 0x584aadd7 (stale) Mirror 14: 0x584aadd7 (stale) Mirror 15: 0x584aadd7 (stale) Mirror 16: 0x92ffc465 (stale) Mirror 17: 0x92ffc465 (stale) Mirror 18: 0xf629755c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17dfc): Mirror 5: 0x584aadd7 (stale) Mirror 14: 0x584aadd7 (stale) Mirror 15: 0x584aadd7 (stale) Mirror 16: 0x92ffc465 (stale) Mirror 17: 0x92ffc465 (stale) Mirror 18: 0xf629755c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17dfc: skipped 000737 1781617416.647396 mapread 0x00cbce thru 0x017dfb (0xb22e bytes) 000739 1781617416.660776 write 0x0201ff thru 0x021574 (0x1376 bytes) 000741 1781617416.662759 mapread 0x0102f7 thru 0x01e6ea (0xe3f4 bytes) 000742 1781617416.671273 MIRROR_RESYNC 7 mirrors 000743 1781617417.262618 read 0x001525 thru 0x003687 (0x2163 bytes) 000744 1781617417.268944 MIRROR_EXTEND to 8 mirrors 000745 1781617417.628687 read 0x010927 thru 0x016628 (0x5d02 bytes) 000746 1781617417.636510 trunc from 0x021575 to 0x03a3cf (0x18e5b bytes) 000747 1781617417.677248 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3d0): Mirror 5: 0x538419a (stale) Mirror 14: 0x538419a (stale) Mirror 15: 0x538419a (stale) Mirror 16: 0x538419a (stale) Mirror 18: 0x538419a (stale) Mirror 19: 0x538419a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3d0): Mirror 5: 0x538419a (stale) Mirror 14: 0x538419a (stale) Mirror 15: 0x538419a (stale) Mirror 16: 0x538419a (stale) Mirror 18: 0x538419a (stale) Mirror 19: 0x538419a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3d0: skipped 000748 1781617417.876766 trunc from 0x03a3d0 to 0x038221 (0x21ae bytes) 000752 1781617417.885407 trunc from 0x038222 to 0x037a3b (0x7e6 bytes) 000753 1781617417.900913 write 0x016c65 thru 0x017f7d (0x1319 bytes) 000756 1781617417.916652 MIRROR_RESYNC 7 mirrors 000758 1781617418.468368 MIRROR_RESYNC 7 mirrors 000759 1781617418.747814 mapwrite 0x02c73a thru 0x02ca73 (0x33a bytes) 000761 1781617418.837015 mapwrite 0x0132d5 thru 0x013e48 (0xb74 bytes) 000762 1781617418.883614 read 0x02e858 thru 0x03533a (0x6ae3 bytes) 000763 1781617418.890562 mapread 0x00b4a9 thru 0x0182dc (0xce34 bytes) 000765 1781617418.900907 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37a3c): Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a3c): Mirror 5: 0xea183fdc (stale) Mirror 14: 0xea183fdc (stale) Mirror 15: 0xea183fdc (stale) Mirror 16: 0xea183fdc (stale) Mirror 18: 0xea183fdc (stale) Mirror 19: 0xea183fdc (stale) CRC-32 checksum value for chunk [0, 0x37a3c): Mirror 1: 0x2b1b0b11 Mirror 20: 0x2b1b0b11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a3c: skipped 000766 1781617419.371323 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a3c): Mirror 5: 0xea183fdc (stale) Mirror 14: 0xea183fdc (stale) Mirror 16: 0xea183fdc (stale) Mirror 18: 0xea183fdc (stale) Mirror 19: 0xea183fdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a3c): Mirror 5: 0xea183fdc (stale) Mirror 14: 0xea183fdc (stale) Mirror 16: 0xea183fdc (stale) Mirror 18: 0xea183fdc (stale) Mirror 19: 0xea183fdc (stale) CRC-32 checksum value for chunk [0, 0x37a3c): Mirror 1: 0x2b1b0b11 Mirror 20: 0x2b1b0b11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a3c: skipped 000767 1781617419.720538 write 0x03d1a8 thru 0x03ffff (0x2e58 bytes) 000768 1781617419.751439 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc1aee0f9 (stale) Mirror 16: 0xc1aee0f9 (stale) Mirror 18: 0xc1aee0f9 (stale) Mirror 19: 0xc1aee0f9 (stale) Mirror 20: 0x45beca70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc1aee0f9 (stale) Mirror 16: 0xc1aee0f9 (stale) Mirror 18: 0xc1aee0f9 (stale) Mirror 19: 0xc1aee0f9 (stale) Mirror 20: 0x45beca70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000769 1781617419.958417 trunc from 0x040000 to 0x0373a7 (0x8c58 bytes) 000770 1781617419.965582 read 0x032a7c thru 0x034afb (0x2080 bytes) 000771 1781617419.968580 trunc from 0x0373a8 to 0x000668 (0x36d3f bytes) 000774 1781617419.975885 read 0x00018b thru 0x000668 (0x4de bytes) 000775 1781617419.979697 mapread 0x000082 thru 0x000668 (0x5e7 bytes) 000777 1781617419.980944 trunc from 0x000669 to 0x00a9e2 (0xa37a bytes) 000778 1781617419.997954 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xa9e3): Mirror 5: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa9e3): Mirror 5: 0xbb5a80ca (stale) Mirror 16: 0xbb5a80ca (stale) Mirror 18: 0xbb5a80ca (stale) Mirror 19: 0xbb5a80ca (stale) Mirror 20: 0xbb5a80ca (stale) CRC-32 checksum value for chunk [0, 0xa9e3): Mirror 1: 0x9f0f0e7a Mirror 21: 0x9f0f0e7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa9e3: skipped 000779 1781617420.382364 trunc from 0x00a9e3 to 0x008997 (0x204b bytes) 000780 1781617420.413015 trunc from 0x008998 to 0x00a01f (0x1688 bytes) 000781 1781617420.421742 mapwrite 0x01d56f thru 0x0262e5 (0x8d77 bytes) 000782 1781617420.472988 mapwrite 0x01864c thru 0x01a65c (0x2011 bytes) 000785 1781617420.491975 write 0x033a56 thru 0x033d81 (0x32c bytes) 000789 1781617420.494900 mapread 0x00155f thru 0x0034b9 (0x1f5b bytes) 000793 1781617420.500735 mapread 0x0263de thru 0x030b69 (0xa78c bytes) 000794 1781617420.507594 read 0x00883d thru 0x00b230 (0x29f4 bytes) 000795 1781617420.508232 write 0x0346fb thru 0x03f243 (0xab49 bytes) 000796 1781617420.509544 write 0x000e8e thru 0x0055ef (0x4762 bytes) 000797 1781617420.515155 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f244): Mirror 5: 0xca510400 (stale) Mirror 16: 0xca510400 (stale) Mirror 18: 0xca510400 (stale) Mirror 19: 0xca510400 (stale) Mirror 20: 0xcf4502b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f244): Mirror 5: 0xca510400 (stale) Mirror 16: 0xca510400 (stale) Mirror 18: 0xca510400 (stale) Mirror 19: 0xca510400 (stale) Mirror 20: 0xcf4502b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f244: skipped 000798 1781617420.717050 write 0x0178ac thru 0x01dc43 (0x6398 bytes) 000801 1781617420.724649 mapwrite 0x03ff6c thru 0x03ffff (0x94 bytes) 000805 1781617420.762179 read 0x02ccef thru 0x03707e (0xa390 bytes) 000806 1781617420.771350 mapread 0x03730e thru 0x03ae0e (0x3b01 bytes) 000809 1781617420.781281 read 0x0186ed thru 0x01f2b9 (0x6bcd bytes) 000810 1781617420.787026 write 0x02d22c thru 0x0331ef (0x5fc4 bytes) 000813 1781617420.789090 trunc from 0x040000 to 0x012fa8 (0x2d057 bytes) 000817 1781617420.798690 read 0x00e1fc thru 0x012fa8 (0x4dad bytes) 000819 1781617420.800104 read 0x00cb3f thru 0x00f202 (0x26c4 bytes) 000820 1781617420.803409 MIRROR_RESYNC 6 mirrors 000822 1781617421.280580 write 0x026824 thru 0x028c99 (0x2476 bytes) 000823 1781617421.315109 read 0x023f8b thru 0x02811f (0x4195 bytes) 000825 1781617421.321605 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c9a): Mirror 5: 0x4b932bbf (stale) Mirror 16: 0x4b932bbf (stale) Mirror 18: 0x4b932bbf (stale) Mirror 20: 0x4b932bbf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c9a): Mirror 5: 0x4b932bbf (stale) Mirror 16: 0x4b932bbf (stale) Mirror 18: 0x4b932bbf (stale) Mirror 20: 0x4b932bbf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c9a: skipped 000826 1781617421.517371 write 0x0335dd thru 0x03e1bb (0xabdf bytes) 000827 1781617421.518999 mapwrite 0x0202d2 thru 0x02c042 (0xbd71 bytes) 000830 1781617421.555066 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1bc): Mirror 5: 0xc0bc466f (stale) Mirror 16: 0xc0bc466f (stale) Mirror 18: 0xc0bc466f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1bc): Mirror 5: 0xc0bc466f (stale) Mirror 16: 0xc0bc466f (stale) Mirror 18: 0xc0bc466f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1bc: skipped 000831 1781617421.733984 write 0x00ade0 thru 0x00fd10 (0x4f31 bytes) 000832 1781617421.739311 mapread 0x02b415 thru 0x02f477 (0x4063 bytes) 000834 1781617421.749341 mapread 0x02c4e2 thru 0x034c17 (0x8736 bytes) 000835 1781617421.750519 trunc from 0x03e1bc to 0x01a852 (0x23969 bytes) 000838 1781617421.759013 mapwrite 0x037c74 thru 0x03ffff (0x838c bytes) 000839 1781617421.796107 mapwrite 0x02ef0f thru 0x032f90 (0x4082 bytes) 000840 1781617421.823424 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf0f598bd (stale) Mirror 16: 0xf0f598bd (stale) Mirror 18: 0xf0f598bd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x629b4e22 Mirror 19: 0x629b4e22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000841 1781617422.258088 mapwrite 0x03c79b thru 0x03ffff (0x3865 bytes) 000843 1781617422.369698 mapwrite 0x01eb0c thru 0x027b63 (0x9058 bytes) 000845 1781617422.417955 write 0x018c95 thru 0x026cbe (0xe02a bytes) 000847 1781617422.424714 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf0f598bd (stale) Mirror 16: 0xf0f598bd (stale) Mirror 18: 0xf0f598bd (stale) Mirror 19: 0x629b4e22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66f39c0b Mirror 20: 0x66f39c0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000848 1781617422.834898 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf0f598bd (stale) Mirror 16: 0xf0f598bd (stale) Mirror 18: 0xf0f598bd (stale) Mirror 19: 0x629b4e22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66f39c0b Mirror 20: 0x66f39c0b Mirror 21: 0x66f39c0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000850 1781617423.218146 trunc from 0x040000 to 0x00d72f (0x328d0 bytes) 000853 1781617423.271353 read 0x00c4d6 thru 0x00d72f (0x125a bytes) 000854 1781617423.277904 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd730): Mirror 16: 0xfb03ea97 (stale) Mirror 18: 0xfb03ea97 (stale) Mirror 19: 0xf5c13110 (stale) Mirror 20: 0xf5c13110 (stale) Mirror 21: 0xf5c13110 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd730): Mirror 16: 0xfb03ea97 (stale) Mirror 18: 0xfb03ea97 (stale) Mirror 19: 0xf5c13110 (stale) Mirror 20: 0xf5c13110 (stale) Mirror 21: 0xf5c13110 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd730: skipped 000855 1781617423.459986 trunc from 0x00d730 to 0x01bd87 (0xe658 bytes) 000857 1781617423.468708 write 0x026e03 thru 0x034759 (0xd957 bytes) 000860 1781617423.471669 trunc from 0x03475a to 0x0331df (0x157a bytes) 000861 1781617423.479497 mapwrite 0x024b86 thru 0x026dc0 (0x223b bytes) 000866 1781617423.506100 mapwrite 0x01bc0b thru 0x0264a4 (0xa89a bytes) 000867 1781617423.536900 read 0x00e98d thru 0x014c69 (0x62dd bytes) 000869 1781617423.541745 write 0x034956 thru 0x03f6b1 (0xad5c bytes) 000872 1781617423.544146 mapwrite 0x0210f3 thru 0x02a2de (0x91ec bytes) 000874 1781617423.569988 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f6b2): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6b2): Mirror 16: 0x2f29dd97 (stale) Mirror 18: 0x2f29dd97 (stale) Mirror 19: 0x51c77df (stale) Mirror 20: 0xda885b85 (stale) Mirror 21: 0xda885b85 (stale) CRC-32 checksum value for chunk [0, 0x3f6b2): Mirror 1: 0xac5531b1 Mirror 22: 0xac5531b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6b2: skipped 000875 1781617424.013633 mapread 0x03476c thru 0x039487 (0x4d1c bytes) 000876 1781617424.033063 write 0x00f7c2 thru 0x015719 (0x5f58 bytes) 000878 1781617424.094870 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f6b2): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6b2): Mirror 16: 0x2f29dd97 (stale) Mirror 18: 0x2f29dd97 (stale) Mirror 19: 0x51c77df (stale) Mirror 20: 0xda885b85 (stale) Mirror 21: 0xda885b85 (stale) Mirror 22: 0xac5531b1 (stale) CRC-32 checksum value for chunk [0, 0x3f6b2): Mirror 1: 0x8176362f Mirror 23: 0x8176362f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6b2: skipped 000879 1781617424.664549 mapread 0x03f3ff thru 0x03f6b1 (0x2b3 bytes) 000881 1781617424.687161 mapwrite 0x0316a9 thru 0x034f8e (0x38e6 bytes) 000882 1781617424.778265 mapread 0x008ed4 thru 0x00bc2e (0x2d5b bytes) 000883 1781617424.784176 mapwrite 0x03cbcb thru 0x03cf51 (0x387 bytes) 000887 1781617424.808375 write 0x0094e1 thru 0x012e2e (0x994e bytes) 000888 1781617424.813296 trunc from 0x03f6b2 to 0x019af5 (0x25bbc bytes) 000890 1781617424.822678 read 0x008c6b thru 0x018556 (0xf8ec bytes) 000892 1781617424.823860 read 0x012fe3 thru 0x013c3f (0xc5d bytes) 000893 1781617424.826288 MIRROR_RESYNC 8 mirrors 000895 1781617425.539630 mapread 0x01221a thru 0x016ed7 (0x4cbe bytes) 000896 1781617425.554974 write 0x03d469 thru 0x03f201 (0x1d99 bytes) 000897 1781617425.589863 trunc from 0x03f202 to 0x00034c (0x3eeb5 bytes) 000898 1781617425.609292 MIRROR_RESYNC 8 mirrors 000899 1781617426.131865 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) 000900 1781617426.308473 MIRROR_RESYNC 7 mirrors 000901 1781617426.466642 trunc from 0x00034d to 0x009972 (0x9626 bytes) 000904 1781617426.493760 write 0x00a339 thru 0x00fb27 (0x57ef bytes) 000908 1781617426.496289 read 0x006640 thru 0x00e68e (0x804f bytes) 000910 1781617426.500542 read 0x0000c3 thru 0x007870 (0x77ae bytes) 000911 1781617426.504560 mapread 0x00048e thru 0x00511f (0x4c92 bytes) 000912 1781617426.507969 MIRROR_RESYNC 7 mirrors 000914 1781617426.872713 trunc from 0x00fb28 to 0x004e1b (0xad0c bytes) 000916 1781617426.914073 write 0x015cee thru 0x01e4ba (0x87cd bytes) 000918 1781617426.915967 read 0x010c8a thru 0x01509a (0x4411 bytes) 000919 1781617426.920172 mapwrite 0x03b1c5 thru 0x03ffff (0x4e3b bytes) 000920 1781617426.961858 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xccc4dd5d (stale) Mirror 19: 0xccc4dd5d (stale) Mirror 20: 0xccc4dd5d (stale) Mirror 21: 0xccc4dd5d (stale) Mirror 23: 0xccc4dd5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xccc4dd5d (stale) Mirror 19: 0xccc4dd5d (stale) Mirror 20: 0xccc4dd5d (stale) Mirror 21: 0xccc4dd5d (stale) Mirror 23: 0xccc4dd5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000922 1781617427.132346 trunc from 0x040000 to 0x013833 (0x2c7cc bytes) 000924 1781617427.138264 trunc from 0x013834 to 0x0101c4 (0x366f bytes) 000925 1781617427.144192 read 0x002a99 thru 0x0079a3 (0x4f0b bytes) 000926 1781617427.147415 mapwrite 0x026f7e thru 0x03673c (0xf7bf bytes) 000927 1781617427.188792 mapwrite 0x03d40f thru 0x03ffff (0x2bf1 bytes) 000928 1781617427.221514 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xccc4dd5d (stale) Mirror 19: 0xccc4dd5d (stale) Mirror 20: 0xccc4dd5d (stale) Mirror 21: 0xccc4dd5d (stale) Mirror 23: 0xccc4dd5d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf424b7bd Mirror 24: 0xf424b7bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000929 1781617427.531626 write 0x0193f5 thru 0x022e4f (0x9a5b bytes) 000930 1781617427.561796 read 0x0009a0 thru 0x00c85f (0xbec0 bytes) 000938 1781617427.569847 write 0x026fb6 thru 0x035b70 (0xebbb bytes) 000939 1781617427.575870 mapwrite 0x03f6ac thru 0x03ffff (0x954 bytes) 000940 1781617427.599470 trunc from 0x040000 to 0x01c02a (0x23fd5 bytes) 000942 1781617427.621755 read 0x01ae91 thru 0x01c02a (0x119a bytes) 000945 1781617427.623510 write 0x031b47 thru 0x03d786 (0xbc40 bytes) 000948 1781617427.626471 mapwrite 0x018c4e thru 0x01b396 (0x2749 bytes) 000950 1781617427.656928 trunc from 0x03d787 to 0x01743f (0x26347 bytes) 000951 1781617427.665306 mapwrite 0x01e727 thru 0x0275f3 (0x8ecd bytes) 000952 1781617427.714920 read 0x00b55a thru 0x010df4 (0x589b bytes) 000953 1781617427.717772 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x275f4): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x275f4): Mirror 18: 0x83ec5982 (stale) Mirror 19: 0x83ec5982 (stale) Mirror 20: 0x83ec5982 (stale) Mirror 21: 0x83ec5982 (stale) Mirror 23: 0x83ec5982 (stale) Mirror 24: 0xb7778649 (stale) CRC-32 checksum value for chunk [0, 0x275f4): Mirror 1: 0x5166d099 Mirror 25: 0x5166d099 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x275f4: skipped 000954 1781617428.110500 mapwrite 0x036296 thru 0x03eb1a (0x8885 bytes) 000955 1781617428.205323 trunc from 0x03eb1b to 0x00d68a (0x31490 bytes) 000956 1781617428.220677 read 0x001a6a thru 0x0091c7 (0x775e bytes) 000957 1781617428.226242 write 0x00d055 thru 0x0118ec (0x4898 bytes) 000958 1781617428.230693 read 0x0038a5 thru 0x00fc6d (0xc3c9 bytes) 000961 1781617428.243308 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x118ed): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x118ed): Mirror 18: 0x74c5ec7a (stale) Mirror 19: 0x74c5ec7a (stale) Mirror 20: 0x74c5ec7a (stale) Mirror 21: 0x74c5ec7a (stale) Mirror 23: 0x74c5ec7a (stale) Mirror 24: 0xc517f8e2 (stale) Mirror 25: 0xc517f8e2 (stale) CRC-32 checksum value for chunk [0, 0x118ed): Mirror 1: 0x7383b90b Mirror 26: 0x7383b90b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x118ed: skipped 000962 1781617428.646272 mapwrite 0x03dbe9 thru 0x03ffff (0x2417 bytes) 000963 1781617428.725554 write 0x00142e thru 0x00410d (0x2ce0 bytes) 000964 1781617428.731732 trunc from 0x040000 to 0x01ca40 (0x235bf bytes) 000966 1781617428.742300 mapread 0x01b673 thru 0x01ca40 (0x13ce bytes) 000969 1781617428.754812 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ca41): Mirror 19: 0xd21d5f6a (stale) Mirror 20: 0xd21d5f6a (stale) Mirror 21: 0xd21d5f6a (stale) Mirror 23: 0xd21d5f6a (stale) Mirror 24: 0xbf890223 (stale) Mirror 25: 0xbf890223 (stale) Mirror 26: 0x45c4a162 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca41): Mirror 19: 0xd21d5f6a (stale) Mirror 20: 0xd21d5f6a (stale) Mirror 21: 0xd21d5f6a (stale) Mirror 23: 0xd21d5f6a (stale) Mirror 24: 0xbf890223 (stale) Mirror 25: 0xbf890223 (stale) Mirror 26: 0x45c4a162 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca41: skipped 000970 1781617428.970073 mapwrite 0x02f831 thru 0x03b547 (0xbd17 bytes) 000972 1781617429.008763 trunc from 0x03b548 to 0x02457c (0x16fcb bytes) 000973 1781617429.018950 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2457d): Mirror 19: 0xe7abf8d1 (stale) Mirror 20: 0xe7abf8d1 (stale) Mirror 21: 0xe7abf8d1 (stale) Mirror 23: 0xe7abf8d1 (stale) Mirror 24: 0x64a1d5d3 (stale) Mirror 26: 0x69d59d90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2457d): Mirror 19: 0xe7abf8d1 (stale) Mirror 20: 0xe7abf8d1 (stale) Mirror 21: 0xe7abf8d1 (stale) Mirror 23: 0xe7abf8d1 (stale) Mirror 24: 0x64a1d5d3 (stale) Mirror 26: 0x69d59d90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2457d: skipped 000974 1781617429.218501 mapwrite 0x02442d thru 0x0283df (0x3fb3 bytes) 000975 1781617429.266762 read 0x0194f7 thru 0x027c39 (0xe743 bytes) 000976 1781617429.273957 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x424:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x283e0): Mirror 19: 0xf582439d (stale) Mirror 20: 0xf582439d (stale) Mirror 23: 0xf582439d (stale) Mirror 24: 0x8e5cd7bb (stale) Mirror 26: 0x3655ff9b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x283e0): Mirror 19: 0xf582439d (stale) Mirror 20: 0xf582439d (stale) Mirror 23: 0xf582439d (stale) Mirror 24: 0x8e5cd7bb (stale) Mirror 26: 0x3655ff9b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x283e0: skipped 000977 1781617429.448162 mapread 0x017bfd thru 0x025799 (0xdb9d bytes) 000979 1781617429.471652 trunc from 0x0283e0 to 0x0097e7 (0x1ebf8 bytes) 000980 1781617429.480135 write 0x014a27 thru 0x0165c9 (0x1ba3 bytes) 000982 1781617429.481672 mapread 0x00f312 thru 0x0165c9 (0x72b8 bytes) 000984 1781617429.488267 mapread 0x00c7f9 thru 0x014c27 (0x842f bytes) 000988 1781617429.492740 trunc from 0x0165ca to 0x01476d (0x1e5c bytes) 000989 1781617429.501778 mapwrite 0x033c38 thru 0x03647e (0x2847 bytes) 000990 1781617429.563245 read 0x032f5a thru 0x03647e (0x3525 bytes) 000991 1781617429.563879 write 0x0329a7 thru 0x036cca (0x4324 bytes) 000992 1781617429.564854 mapread 0x015759 thru 0x017516 (0x1dbe bytes) 000993 1781617429.573549 write 0x01de39 thru 0x01e733 (0x8fb bytes) 000994 1781617429.577725 mapwrite 0x0229c7 thru 0x0316ff (0xed39 bytes) 000995 1781617429.630764 read 0x022fa0 thru 0x030d81 (0xdde2 bytes) 000998 1781617429.632589 write 0x018fc8 thru 0x023914 (0xa94d bytes) All operations completed A-OK! PASS 70a (45s) == sanity-flr test 71: check mirror extend parent fid ==== 09:43:51 (1781617431) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 09:43:53 (1781617433) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x67f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x663:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11952 s, 17.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x67f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x680:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x663:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853585 s, 12.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x67f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x680:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x664:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x663:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x67f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x680:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x664:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x663:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x681:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 09:43:56 (1781617436) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7050016 .. Extending file size to 4690016 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7688864 .. Extending file size to 6667648 .. Extending file size to 8974048 .. Extending file size to 7812064 .. Extending file size to 9393856 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9030816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9037920 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58560 58561 58563 58564 58566 done 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) 192.168.201.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:45:04 (1781617504) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7619712 .. Extending file size to 5862464 .. done Extending file size to 8012352 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7728608 .. Extending file size to 8756160 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8525728 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9179936 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8899616 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9299872 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61178 61179 61181 61184 61185 done 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) 192.168.201.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (70s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:46:14 (1781617574) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3328 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3328 2204928 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130835 s, 16.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280025 s, 7.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x612bf935 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:46:33 (1781617593) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173946 s, 12.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 09:46:40 (1781617600) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x47c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67b:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x696:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:46:48 (1781617608) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171403 s, 12.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x697:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x698:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:46:55 (1781617615) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211255 s, 9.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x699:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x699:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69a:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:47:02 (1781617622) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.260777 s, 8.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69c:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:47:31 (1781617651) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954978 s, 11.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x680:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x680:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x681:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:48:25 (1781617705) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148537 s, 14.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x682:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x69f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x682:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:49:21 (1781617761) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. Creating new pool test_204f oleg118-server: Pool lustre.test_204f created Adding targets to pool oleg118-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg118-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147811 s, 14.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x683:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x6a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x683:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg118-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg118-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg118-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:50:32 (1781617832) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:50:36 (1781617836) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:50:40 (1781617840) oleg118-server: Pool lustre.test_206 created Adding targets to pool oleg118-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg118-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg118-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg118-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg118-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:50:58 (1781617858) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451672 s, 23.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252273 s, 41.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe6787296 Mirror 3: 0xe6787296 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:51:03 (1781617863) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:51:05 (1781617865) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:51:06 (1781617866) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0635398 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2240000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2240000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2240000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc50000 Read this (4 bytes): BBBB 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) PASS 209a (20s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:51:26 (1781617886) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00529219 s, 1.5 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:51:29 (1781617889) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225324 s, 46.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x4a1 lmm_fid: [0x200000404:0x4a1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x4a1 lmm_fid: [0x200000404:0x4a1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835105 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 09:51:30.000000000 -0400 Modify: 2026-06-16 09:51:30.000000000 -0400 Change: 2026-06-16 09:51:30.000000000 -0400 Birth: 2026-06-16 09:51:29.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:51:33 (1781617893) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:51:34 (1781617894) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.311712 s, 131 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0349831 s, 117 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 16 09:51 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4a3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 16 09:51 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:51:38 (1781617898) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:51:41 (1781617901) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1184 sec =========== 09:51:45 (1781617905) === sanity-flr: start cleanup 09:51:46 (1781617906) === === sanity-flr: finish cleanup 09:51:49 (1781617909) ===