-----============= acceptance-small: sanity-flr ============----- Wed May 27 04:17:28 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg257-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:17:44 (1779869864) === oleg257-client.virtnet: executing check_config_client /mnt/lustre oleg257-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg257-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0e2896a7000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0e2896a7000.idle_timeout=debug disable quota as required oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:18:10 (1779869890) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7578] == sanity-flr test 0a: lfs mirror create with -N option == 04:18:12 (1779869892) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:18:25 (1779869905) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:18:27 (1779869907) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:18:30 (1779869910) PASS 0d (17s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:18:47 (1779869927) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:18:49 (1779869929) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:18:52 (1779869932) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa0e2896a7000.stats=clear osc.lustre-OST0001-osc-ffffa0e2896a7000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:18:59 (1779869939) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:19:07 (1779869947) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:19:14 (1779869954) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:19:17 (1779869957) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:19:22 (1779869962) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 04:19:28 (1779869968) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:19:34 (1779869974) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:19:36 (1779869976) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:19:42 (1779869982) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:19:50 (1779869990) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:19:55 (1779869995) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.831057 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894994 s, 11.7 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:20:03 (1779870003) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.130969 s, 64.1 MB/s PASS 22 (5s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:20:08 (1779870008) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:20:16 (1779870016) Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg257-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (53s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:21:09 (1779870069) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.317043 s, 29.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (28s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:21:37 (1779870097) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg257-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (54s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:22:32 (1779870152) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.05486 s, 17.2 MB/s -rw-r--r-- 1 root root 52428800 May 27 04:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.827s user 0m0.001s sys 0m0.177s Stopping /mnt/lustre-ost1 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0e2896a7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.898s user 0m0.001s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg257-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg257-server oleg257-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.826s user 0m0.000s sys 0m0.157s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg257-server: oleg257-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg257-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg257-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0e2896a7000.ost_server_uuid in FULL state after 0 sec PASS 33b (65s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:23:37 (1779870217) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:23:39 (1779870219) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:23:42 (1779870222) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:23:45 (1779870225) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 04:23:50 (1779870230) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.466293 s, 27.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:23:59 (1779870239) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191817 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274106 s, 38.3 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:24:05 (1779870245) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:24:11 (1779870251) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000774562 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000932521 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000967108 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00083249 s, 1.2 MB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 04:24:19 (1779870259) create test files with size 15752776 .. f6750b64bb7ac9634e82babdaaff3a6d - 34fef672ccfaa247609b22e6a8fe8b8b - 70a69008c6ab6102f7f2a74b55b3a239 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg257-client.virtnet /mnt/lustre (opts:) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 04:24:38 (1779870278) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (9s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:24:48 (1779870288) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:24:53 (1779870293) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145684 s, 7.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422573 s, 24.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 04:25:03 (1779870303) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181028 s, 11.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.362771 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203252 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 04:25:16 (1779870316) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 04:25:19 (1779870319) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:25:21 (1779870321) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199561 s, 52.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 04:25:27 (1779870327) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:25:30 (1779870330) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:25:38 (1779870338) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.214464 s, 48.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg257-client.virtnet /mnt/lustre (opts:) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:25:47 (1779870347) llite.lustre-ffffa0e290f57800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.25732 s, 40.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg257-client.virtnet /mnt/lustre (opts:) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffffa0e2985c6800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:25:59 (1779870359) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (26s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:26:25 (1779870385) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (25s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:26:50 (1779870410) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:26:53 (1779870413) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.220472 s, 14.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.143888 s, 21.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x68cea9f7 (nosync,stale) Mirror 2: 0x68cea9f7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f09f96d5a76f6f68d82c398a987730f0 - mirror 1 checksum: 22035360773ecce00c90c1f64baf6507 - mirror 2 checksum: f09f96d5a76f6f68d82c398a987730f0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 27 04:26:56 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed May 27 04:26:56 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f09f96d5a76f6f68d82c398a987730f0 - mirror 1 checksum: 22035360773ecce00c90c1f64baf6507 - mirror 2 checksum: 22035360773ecce00c90c1f64baf6507 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 27 04:26:56 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed May 27 04:26:56 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } PASS 48 (8s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:27:02 (1779870422) Failing mds1 on oleg257-server Stopping /mnt/lustre-mds1 (opts:) on oleg257-server 04:27:06 (1779870426) shut down facet: mds1 facet_host: oleg257-server facet_failover_host: oleg257-server Failover mds1 to oleg257-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 04:27:25 (1779870445) targets are mounted 04:27:25 (1779870445) facet_failover done oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217008 s, 48.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } PASS 50A (35s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:27:37 (1779870457) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:27:39 (1779870459) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:27:42 (1779870462) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00682939 s, 2.4 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:27:48 (1779870468) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:27:50 (1779870470) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:27:53 (1779870473) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:28:10 (1779870490) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (25s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:28:36 (1779870516) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:28:58 (1779870538) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:29:20 (1779870560) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 04:29:27 (1779870567) mirror create pid 53485 mirror split pid 53486 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53485 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53486 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 04:30:32 (1779870632) Seed set to 633 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1779870634.217412 write 0x037329 thru 0x03c031 (0x4d09 bytes) 000002 1779870634.228951 mapwrite 0x024f11 thru 0x033eb7 (0xefa7 bytes) truncating to largest ever: 0x18708 000004 1779870634.309034 trunc from 0x03c032 to 0x018707 (0x2392a bytes) truncating to largest ever: 0x34d43 000006 1779870634.329948 trunc from 0x018708 to 0x034d42 (0x1c63b bytes) 000007 1779870634.347330 mapwrite 0x00b51a thru 0x00dd71 (0x2858 bytes) 000010 1779870634.402267 mapwrite 0x00cd08 thru 0x01664b (0x9944 bytes) 000011 1779870634.444439 read 0x022d8a thru 0x02d58e (0xa805 bytes) 000013 1779870634.451207 read 0x017e9f thru 0x026002 (0xe164 bytes) 000015 1779870634.461969 read 0x024534 thru 0x0344ea (0xffb7 bytes) 000016 1779870634.462647 read 0x00c694 thru 0x010abb (0x4428 bytes) 000017 1779870634.465354 trunc from 0x034d43 to 0x017fb9 (0x1cd89 bytes) 000018 1779870634.479913 write 0x008e7d thru 0x0105c5 (0x7749 bytes) 000019 1779870634.481158 mapread 0x00f77e thru 0x017fb9 (0x883c bytes) 000024 1779870634.486659 mapread 0x012806 thru 0x017fb9 (0x57b4 bytes) 000026 1779870634.489806 write 0x032390 thru 0x032ef0 (0xb61 bytes) 000028 1779870634.492782 trunc from 0x032ef1 to 0x01b9e6 (0x1750a bytes) 000029 1779870634.508447 mapwrite 0x0297e9 thru 0x0340af (0xa8c7 bytes) 000030 1779870634.574628 mapwrite 0x017cbf thru 0x024edc (0xd21e bytes) 000031 1779870634.622175 mapwrite 0x00dca8 thru 0x019ef3 (0xc24c bytes) 000032 1779870634.652927 MIRROR_EXTEND to 2 mirrors 000033 1779870635.038456 trunc from 0x0340b0 to 0x01bf3b (0x18174 bytes) 000034 1779870635.124110 trunc from 0x01bf3c to 0x028bbe (0xcc83 bytes) 000036 1779870635.159936 MIRROR_RESYNC 2 mirrors 000039 1779870635.769200 trunc from 0x028bbf to 0x030594 (0x79d6 bytes) truncating to largest ever: 0x3d38c 000043 1779870635.847059 trunc from 0x030595 to 0x03d38b (0xcdf7 bytes) 000044 1779870635.869775 MIRROR_RESYNC 2 mirrors 000045 1779870636.456489 mapwrite 0x00e95b thru 0x0178c6 (0x8f6c bytes) 000047 1779870636.618013 mapread 0x027265 thru 0x02fb7b (0x8917 bytes) 000048 1779870636.637983 read 0x00a7d2 thru 0x017a83 (0xd2b2 bytes) 000054 1779870636.647253 trunc from 0x03d38c to 0x03678f (0x6bfc bytes) 000055 1779870636.669616 write 0x03d10c thru 0x03ffff (0x2ef4 bytes) 000057 1779870636.676636 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000060 1779870636.832663 write 0x03ab2f thru 0x03ffff (0x54d1 bytes) 000062 1779870636.837244 mapwrite 0x022944 thru 0x031d55 (0xf412 bytes) 000063 1779870636.897277 mapread 0x00e03a thru 0x01768d (0x9654 bytes) 000064 1779870636.915430 read 0x026a72 thru 0x02ff7f (0x950e bytes) 000067 1779870636.918207 mapread 0x0227fd thru 0x02b3e6 (0x8bea bytes) 000068 1779870636.919433 mapwrite 0x01aff7 thru 0x01e332 (0x333c bytes) 000070 1779870636.956807 trunc from 0x040000 to 0x03259d (0xda62 bytes) 000073 1779870636.982711 write 0x03756c thru 0x03ffff (0x8a94 bytes) 000074 1779870636.985793 trunc from 0x040000 to 0x027286 (0x18d79 bytes) 000077 1779870637.002452 read 0x000400 thru 0x00f37a (0xef7b bytes) 000079 1779870637.014610 write 0x023e1a thru 0x0316c3 (0xd8aa bytes) 000082 1779870637.020202 write 0x0011b7 thru 0x003293 (0x20dd bytes) 000083 1779870637.022616 mapread 0x030c9c thru 0x0316c3 (0xa28 bytes) 000084 1779870637.023682 mapwrite 0x038e05 thru 0x03a550 (0x174c bytes) 000086 1779870637.091651 mapread 0x002f71 thru 0x00cec9 (0x9f59 bytes) 000093 1779870637.099222 trunc from 0x03a551 to 0x027f7b (0x125d5 bytes) 000094 1779870637.111991 write 0x001fcd thru 0x005056 (0x308a bytes) 000095 1779870637.114170 mapread 0x00a942 thru 0x00b7bf (0xe7e bytes) 000096 1779870637.115654 read 0x000b4b thru 0x001fd4 (0x148a bytes) 000097 1779870637.116460 mapread 0x00fe52 thru 0x013820 (0x39cf bytes) 000101 1779870637.122002 write 0x039154 thru 0x03ffff (0x6eac bytes) 000103 1779870637.124612 mapwrite 0x032692 thru 0x039d1e (0x768d bytes) 000104 1779870637.174276 mapwrite 0x00a8bd thru 0x01077b (0x5ebf bytes) truncating to largest ever: 0x3e6fe 000105 1779870637.215437 trunc from 0x040000 to 0x03e6fd (0x1902 bytes) 000107 1779870637.232600 write 0x026eea thru 0x034f0f (0xe026 bytes) 000110 1779870637.236441 mapread 0x01b755 thru 0x0233f7 (0x7ca3 bytes) 000112 1779870637.238916 mapwrite 0x00dd8d thru 0x010c12 (0x2e86 bytes) 000113 1779870637.283277 write 0x009763 thru 0x016c43 (0xd4e1 bytes) 000115 1779870637.285736 mapread 0x0035dd thru 0x0101d1 (0xcbf5 bytes) 000117 1779870637.287693 mapread 0x02b4f6 thru 0x034258 (0x8d63 bytes) 000118 1779870637.288812 write 0x013dc4 thru 0x01848b (0x46c8 bytes) 000119 1779870637.290164 trunc from 0x03e6fe to 0x03486a (0x9e93 bytes) 000120 1779870637.301754 read 0x008f1b thru 0x012b44 (0x9c2a bytes) 000121 1779870637.302387 write 0x030e58 thru 0x03991d (0x8ac6 bytes) 000122 1779870637.304558 mapread 0x0007d8 thru 0x001325 (0xb4e bytes) 000124 1779870637.309525 MIRROR_EXTEND to 2 mirrors 000125 1779870637.754657 mapread 0x004bd0 thru 0x00d239 (0x866a bytes) 000126 1779870637.779729 trunc from 0x03991e to 0x02b20c (0xe711 bytes) 000127 1779870637.855941 mapwrite 0x0225a5 thru 0x029055 (0x6ab1 bytes) 000129 1779870637.905067 MIRROR_RESYNC 2 mirrors 000130 1779870638.541010 mapread 0x00cd2b thru 0x01a5f4 (0xd8ca bytes) 000133 1779870638.569873 read 0x027372 thru 0x0295cf (0x225e bytes) 000134 1779870638.575416 mapread 0x01b449 thru 0x01f5f7 (0x41af bytes) 000135 1779870638.576455 read 0x023d96 thru 0x02a9e7 (0x6c52 bytes) 000136 1779870638.585266 write 0x037b71 thru 0x038eb1 (0x1341 bytes) 000137 1779870638.650662 mapwrite 0x009f12 thru 0x017612 (0xd701 bytes) 000138 1779870638.758660 trunc from 0x038eb2 to 0x03b573 (0x26c2 bytes) 000140 1779870638.775097 mapwrite 0x023b3c thru 0x025712 (0x1bd7 bytes) 000143 1779870638.818729 MIRROR_RESYNC 2 mirrors 000144 1779870639.389860 trunc from 0x03b574 to 0x03756a (0x4009 bytes) 000145 1779870639.461433 read 0x0135dd thru 0x01532a (0x1d4e bytes) 000146 1779870639.467152 write 0x0150db thru 0x01ae8a (0x5db0 bytes) 000147 1779870639.471722 mapread 0x029147 thru 0x02db69 (0x4a23 bytes) 000148 1779870639.485680 mapread 0x026688 thru 0x031712 (0xb08b bytes) 000150 1779870639.487298 read 0x00a66a thru 0x015a21 (0xb3b8 bytes) 000151 1779870639.503276 MIRROR_RESYNC 2 mirrors 000152 1779870639.964343 trunc from 0x03756b to 0x0158e0 (0x21c8a bytes) 000153 1779870640.036255 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000156 1779870640.136018 trunc from 0x0158e1 to 0x008bfb (0xcce5 bytes) 000157 1779870640.149111 mapread 0x0000f8 thru 0x008bfb (0x8b04 bytes) 000158 1779870640.166131 mapread 0x0038e1 thru 0x008bfb (0x531b bytes) 000160 1779870640.167360 trunc from 0x008bfc to 0x00642d (0x27ce bytes) 000162 1779870640.182592 MIRROR_EXTEND to 2 mirrors 000163 1779870640.736948 MIRROR_RESYNC 2 mirrors 000165 1779870641.065759 write 0x013406 thru 0x01d575 (0xa170 bytes) 000166 1779870641.128690 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d576): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d576): Mirror 2: 0x1ae31efe (stale) CRC-32 checksum value for chunk [0, 0x1d576): Mirror 1: 0xf059dd25 Mirror 3: 0xf059dd25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d576: skipped 000172 1779870641.855583 write 0x0121b8 thru 0x01c8d6 (0xa71f bytes) 000174 1779870642.001132 MIRROR_RESYNC 3 mirrors 000175 1779870642.592105 mapread 0x0071e9 thru 0x007c42 (0xa5a bytes) 000176 1779870642.598757 mapread 0x009a37 thru 0x00b8d9 (0x1ea3 bytes) 000177 1779870642.617630 read 0x00deab thru 0x015c3a (0x7d90 bytes) 000178 1779870642.625339 mapread 0x012d3f thru 0x01d575 (0xa837 bytes) 000183 1779870642.639171 mapwrite 0x02dd5e thru 0x02efb4 (0x1257 bytes) 000184 1779870642.779157 write 0x0235d3 thru 0x02a505 (0x6f33 bytes) 000185 1779870642.788230 mapread 0x016ddb thru 0x0204bd (0x96e3 bytes) 000186 1779870642.799286 trunc from 0x02efb5 to 0x00fb97 (0x1f41d bytes) 000187 1779870642.823360 trunc from 0x00fb98 to 0x026e4f (0x172b8 bytes) 000188 1779870642.842194 mapread 0x00edca thru 0x01681e (0x7a55 bytes) 000189 1779870642.853052 mapread 0x0154e4 thru 0x0158b4 (0x3d1 bytes) 000190 1779870642.854404 mapwrite 0x008c0d thru 0x011217 (0x860b bytes) 000191 1779870642.915649 read 0x003ccb thru 0x00a141 (0x6477 bytes) 000194 1779870642.925266 trunc from 0x026e50 to 0x01f1ee (0x7c61 bytes) 000195 1779870642.946152 mapwrite 0x018b3a thru 0x022f2d (0xa3f4 bytes) 000197 1779870643.034308 read 0x01b159 thru 0x022f2d (0x7dd5 bytes) 000199 1779870643.036395 write 0x038bbc thru 0x03a4d8 (0x191d bytes) 000200 1779870643.037690 trunc from 0x03a4d9 to 0x00512f (0x353a9 bytes) 000201 1779870643.062211 mapread 0x0042b8 thru 0x00512f (0xe78 bytes) 000202 1779870643.063389 write 0x025848 thru 0x027aea (0x22a3 bytes) 000204 1779870643.066542 read 0x00ebd3 thru 0x010e11 (0x223f bytes) 000205 1779870643.075163 mapwrite 0x000733 thru 0x00b83c (0xb10a bytes) 000206 1779870643.118113 read 0x0066d7 thru 0x00e21d (0x7b47 bytes) 000207 1779870643.118914 mapread 0x015a12 thru 0x01a702 (0x4cf1 bytes) 000212 1779870643.123689 mapread 0x016658 thru 0x01a82b (0x41d4 bytes) 000213 1779870643.124919 trunc from 0x027aeb to 0x00a88d (0x1d25d bytes) 000216 1779870643.139516 mapread 0x00740f thru 0x00a88d (0x347f bytes) 000217 1779870643.140585 mapwrite 0x016c89 thru 0x018505 (0x187d bytes) 000218 1779870643.201499 read 0x015b8f thru 0x018505 (0x2977 bytes) 000220 1779870643.203936 mapwrite 0x002602 thru 0x010086 (0xda85 bytes) 000222 1779870643.250528 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x18506): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18506): Mirror 2: 0xdd176e39 (stale) Mirror 3: 0xdd176e39 (stale) CRC-32 checksum value for chunk [0, 0x18506): Mirror 1: 0x156bb68c Mirror 4: 0x156bb68c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18506: skipped 000224 1779870643.849609 mapread 0x016c76 thru 0x018505 (0x1890 bytes) 000226 1779870643.898703 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18506): Mirror 2: 0xdd176e39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18506): Mirror 2: 0xdd176e39 (stale) CRC-32 checksum value for chunk [0, 0x18506): Mirror 1: 0x156bb68c Mirror 4: 0x156bb68c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18506: skipped 000228 1779870644.485113 write 0x01e0b4 thru 0x02d10d (0xf05a bytes) 000235 1779870644.559393 mapread 0x005517 thru 0x0090ec (0x3bd6 bytes) 000236 1779870644.572891 trunc from 0x02d10e to 0x018278 (0x14e95 bytes) 000237 1779870644.601068 trunc from 0x018279 to 0x035fbf (0x1dd47 bytes) 000238 1779870644.616499 write 0x016a34 thru 0x017e12 (0x13df bytes) 000241 1779870644.627261 write 0x010cb6 thru 0x01e2a4 (0xd5ef bytes) 000242 1779870644.628653 mapwrite 0x009685 thru 0x013ddc (0xa758 bytes) 000244 1779870644.671231 write 0x029216 thru 0x0355f8 (0xc3e3 bytes) 000245 1779870644.687735 MIRROR_RESYNC 3 mirrors 000247 1779870645.379488 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000249 1779870645.784754 MIRROR_EXTEND to 3 mirrors 000250 1779870646.359035 write 0x010f40 thru 0x0199c8 (0x8a89 bytes) 000255 1779870646.438148 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35fc0): Mirror 4: 0xbbaaa072 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fc0): Mirror 4: 0xbbaaa072 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fc0: skipped 000257 1779870646.685607 MIRROR_RESYNC 2 mirrors 000260 1779870647.190684 write 0x00d1d6 thru 0x00e407 (0x1232 bytes) 000262 1779870647.254510 trunc from 0x035fc0 to 0x01dd9f (0x18220 bytes) 000268 1779870647.284302 mapwrite 0x03b250 thru 0x03ffff (0x4db0 bytes) 000270 1779870647.349472 read 0x033d18 thru 0x03e67b (0xa964 bytes) 000271 1779870647.355515 trunc from 0x040000 to 0x018213 (0x27dec bytes) 000272 1779870647.370491 read 0x014f0b thru 0x018213 (0x3309 bytes) 000273 1779870647.380033 write 0x013e22 thru 0x015368 (0x1547 bytes) 000274 1779870647.381065 write 0x0059ce thru 0x00eae8 (0x911b bytes) 000275 1779870647.382212 mapread 0x00532d thru 0x00e152 (0x8e26 bytes) 000277 1779870647.384653 trunc from 0x018214 to 0x0132a6 (0x4f6d bytes) 000280 1779870647.398295 write 0x014cd6 thru 0x01aa78 (0x5da3 bytes) 000281 1779870647.400509 trunc from 0x01aa79 to 0x0276d3 (0xcc5b bytes) 000282 1779870647.414769 mapwrite 0x00693b thru 0x0148d8 (0xdf9e bytes) 000285 1779870647.457597 mapread 0x0057cd thru 0x00ecbc (0x94f0 bytes) 000286 1779870647.458973 trunc from 0x0276d4 to 0x00014c (0x27587 bytes) 000289 1779870647.479492 write 0x0235b5 thru 0x027320 (0x3d6c bytes) 000290 1779870647.480741 mapwrite 0x01adc5 thru 0x022505 (0x7741 bytes) 000291 1779870647.550460 mapread 0x01345d thru 0x017c4a (0x47ee bytes) 000292 1779870647.599977 MIRROR_RESYNC 2 mirrors 000295 1779870648.045167 write 0x0019df thru 0x00cb8f (0xb1b1 bytes) 000296 1779870648.078568 mapread 0x0099b3 thru 0x0160b9 (0xc707 bytes) 000299 1779870648.089504 trunc from 0x027321 to 0x03367f (0xc35f bytes) 000301 1779870648.108391 mapwrite 0x000dbf thru 0x005e1b (0x505d bytes) 000303 1779870648.138033 read 0x027428 thru 0x02af87 (0x3b60 bytes) 000304 1779870648.142627 read 0x026fac thru 0x02bf65 (0x4fba bytes) 000305 1779870648.149401 write 0x02b989 thru 0x02d1e2 (0x185a bytes) 000306 1779870648.150204 mapwrite 0x0161e3 thru 0x0214c8 (0xb2e6 bytes) 000309 1779870648.194859 trunc from 0x033680 to 0x0227d3 (0x10eac bytes) 000310 1779870648.217364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x227d4): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x227d4): Mirror 4: 0x9b7ae3e (stale) CRC-32 checksum value for chunk [0, 0x227d4): Mirror 1: 0xb73afffa Mirror 5: 0xb73afffa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x227d4: skipped 000313 1779870648.705054 mapread 0x014711 thru 0x01f1a5 (0xaa95 bytes) 000317 1779870648.720934 mapread 0x0137e9 thru 0x01e8f1 (0xb109 bytes) 000318 1779870648.723953 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x227d4): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x227d4): Mirror 4: 0x9b7ae3e (stale) CRC-32 checksum value for chunk [0, 0x227d4): Mirror 1: 0xb73afffa Mirror 5: 0xb73afffa Mirror 6: 0xb73afffa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x227d4: skipped 000319 1779870649.414409 write 0x029068 thru 0x02f0c7 (0x6060 bytes) truncating to largest ever: 0x3e928 000321 1779870649.479566 trunc from 0x02f0c8 to 0x03e927 (0xf860 bytes) 000323 1779870649.497340 mapread 0x01b96e thru 0x0252a9 (0x993c bytes) 000325 1779870649.514049 MIRROR_RESYNC 4 mirrors 000326 1779870650.269183 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000328 1779870650.585536 mapread 0x00c7bd thru 0x0153dd (0x8c21 bytes) 000331 1779870650.602798 read 0x02e7e3 thru 0x035a6c (0x728a bytes) 000332 1779870650.612239 read 0x03a4c2 thru 0x03e927 (0x4466 bytes) 000333 1779870650.635743 write 0x004217 thru 0x008197 (0x3f81 bytes) 000335 1779870650.721920 write 0x021345 thru 0x02ebd5 (0xd891 bytes) 000337 1779870650.734019 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e928): Mirror 5: 0x5eeb2b8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e928): Mirror 5: 0x5eeb2b8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e928: skipped 000339 1779870651.022180 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e928): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e928): Mirror 5: 0x5eeb2b8e (stale) CRC-32 checksum value for chunk [0, 0x3e928): Mirror 1: 0x636fd789 Mirror 6: 0x636fd789 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e928: skipped 000341 1779870651.755520 trunc from 0x03e928 to 0x00e052 (0x308d5 bytes) 000344 1779870651.887810 read 0x00c2de thru 0x00e052 (0x1d75 bytes) 000345 1779870651.893387 read 0x004327 thru 0x00e052 (0x9d2c bytes) 000346 1779870651.905070 mapwrite 0x0165a7 thru 0x024de4 (0xe83e bytes) 000347 1779870651.985707 MIRROR_RESYNC 3 mirrors 000348 1779870652.515762 mapwrite 0x0145b8 thru 0x01b5eb (0x7034 bytes) 000350 1779870652.639595 write 0x003859 thru 0x00c192 (0x893a bytes) 000351 1779870652.653974 trunc from 0x024de5 to 0x0331c7 (0xe3e3 bytes) 000352 1779870652.673071 trunc from 0x0331c8 to 0x03a79b (0x75d4 bytes) 000353 1779870652.688628 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a79c): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a79c): Mirror 5: 0x94513898 (stale) Mirror 6: 0x94513898 (stale) CRC-32 checksum value for chunk [0, 0x3a79c): Mirror 1: 0x9653e4b0 Mirror 7: 0x9653e4b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a79c: skipped 000354 1779870653.437646 trunc from 0x03a79c to 0x035d51 (0x4a4a bytes) 000355 1779870653.485381 write 0x00dfe1 thru 0x00e9df (0x9ff bytes) 000356 1779870653.491512 mapwrite 0x02828b thru 0x028fe4 (0xd5a bytes) 000357 1779870653.530685 mapread 0x016591 thru 0x016593 (0x3 bytes) 000363 1779870653.543162 write 0x037229 thru 0x03939d (0x2175 bytes) 000365 1779870653.545279 write 0x037bdf thru 0x03ffff (0x8421 bytes) 000367 1779870653.547645 mapwrite 0x0025db thru 0x00517e (0x2ba4 bytes) 000368 1779870653.584531 mapwrite 0x0287b0 thru 0x035b20 (0xd371 bytes) 000369 1779870653.624013 mapwrite 0x004517 thru 0x01428a (0xfd74 bytes) 000371 1779870653.665346 write 0x029d90 thru 0x02fb8a (0x5dfb bytes) 000372 1779870653.666345 mapread 0x00a758 thru 0x00fa06 (0x52af bytes) 000373 1779870653.667358 mapwrite 0x03281f thru 0x03ffff (0xd7e1 bytes) 000375 1779870653.699032 read 0x02d9cf thru 0x03c9ea (0xf01c bytes) 000381 1779870653.703739 write 0x012223 thru 0x01f4b7 (0xd295 bytes) 000384 1779870653.711190 mapread 0x012143 thru 0x018d7a (0x6c38 bytes) 000385 1779870653.712948 write 0x0190d2 thru 0x022baf (0x9ade bytes) 000386 1779870653.715575 read 0x0074a7 thru 0x00852e (0x1088 bytes) 000387 1779870653.716783 mapread 0x03743b thru 0x03d3bd (0x5f83 bytes) 000390 1779870653.720137 read 0x015a10 thru 0x0185f5 (0x2be6 bytes) 000391 1779870653.721042 mapwrite 0x019299 thru 0x023900 (0xa668 bytes) 000392 1779870653.757382 mapread 0x00038e thru 0x00a3a2 (0xa015 bytes) 000393 1779870653.767684 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x596a8ec1 (stale) Mirror 7: 0xd2c5cfa6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x596a8ec1 (stale) Mirror 7: 0xd2c5cfa6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000394 1779870654.013471 mapread 0x02e133 thru 0x035a55 (0x7923 bytes) 000395 1779870654.031282 read 0x033309 thru 0x03ffff (0xccf7 bytes) 000396 1779870654.038252 write 0x00d2f8 thru 0x01086a (0x3573 bytes) 000399 1779870654.053674 mapread 0x008a3d thru 0x00d8c7 (0x4e8b bytes) 000400 1779870654.059191 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd2c5cfa6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd2c5cfa6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000401 1779870654.245218 mapread 0x01993c thru 0x026907 (0xcfcc bytes) 000402 1779870654.271005 mapwrite 0x037dbe thru 0x03c55c (0x479f bytes) 000403 1779870654.330885 trunc from 0x040000 to 0x0115a5 (0x2ea5a bytes) 000404 1779870654.352015 read 0x0023ad thru 0x007bfb (0x584f bytes) 000405 1779870654.361636 write 0x001633 thru 0x007b24 (0x64f2 bytes) 000409 1779870654.373385 trunc from 0x0115a6 to 0x026783 (0x151de bytes) 000410 1779870654.392536 write 0x01a1fc thru 0x020622 (0x6427 bytes) 000413 1779870654.401766 mapwrite 0x007a73 thru 0x012fe6 (0xb574 bytes) 000415 1779870654.441164 trunc from 0x026784 to 0x02d579 (0x6df6 bytes) 000419 1779870654.462908 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d57a): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d57a): Mirror 7: 0x1b610290 (stale) CRC-32 checksum value for chunk [0, 0x2d57a): Mirror 1: 0xdd89ed2e Mirror 8: 0xdd89ed2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d57a: skipped 000420 1779870655.121196 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d57a): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d57a): Mirror 7: 0x1b610290 (stale) CRC-32 checksum value for chunk [0, 0x2d57a): Mirror 1: 0xdd89ed2e Mirror 8: 0xdd89ed2e Mirror 9: 0xdd89ed2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d57a: skipped 000423 1779870655.821517 write 0x02dea7 thru 0x0332ab (0x5405 bytes) 000424 1779870655.902119 mapread 0x02e5f9 thru 0x0332ab (0x4cb3 bytes) 000425 1779870655.903261 read 0x026e83 thru 0x02e023 (0x71a1 bytes) 000426 1779870655.911266 read 0x023806 thru 0x026fd0 (0x37cb bytes) 000427 1779870655.920514 read 0x0000ed thru 0x007017 (0x6f2b bytes) 000430 1779870655.922957 read 0x01ece8 thru 0x027004 (0x831d bytes) 000432 1779870655.924307 write 0x011e85 thru 0x0169e1 (0x4b5d bytes) 000434 1779870655.926788 mapread 0x032f40 thru 0x0332ab (0x36c bytes) 000435 1779870655.928034 mapwrite 0x025bf3 thru 0x0283b7 (0x27c5 bytes) 000437 1779870655.988254 mapread 0x02d1ab thru 0x02f0af (0x1f05 bytes) 000439 1779870655.990499 mapread 0x018a9d thru 0x025a33 (0xcf97 bytes) 000440 1779870655.992469 mapread 0x023e33 thru 0x02d8ca (0x9a98 bytes) 000441 1779870655.998548 trunc from 0x0332ac to 0x037a26 (0x477b bytes) 000445 1779870656.023074 read 0x0306fd thru 0x037a26 (0x732a bytes) 000446 1779870656.028516 trunc from 0x037a27 to 0x0261d2 (0x11854 bytes) 000447 1779870656.048033 mapwrite 0x004eaf thru 0x00903c (0x418e bytes) 000449 1779870656.088001 read 0x0088ae thru 0x0114a6 (0x8bf9 bytes) 000450 1779870656.088713 trunc from 0x0261d3 to 0x035485 (0xf2b3 bytes) 000451 1779870656.103216 mapwrite 0x02430a thru 0x0336b9 (0xf3b0 bytes) 000452 1779870656.147171 mapwrite 0x006c66 thru 0x0164b4 (0xf84f bytes) 000454 1779870656.204301 trunc from 0x035486 to 0x0313e8 (0x409d bytes) 000455 1779870656.242091 MIRROR_RESYNC 4 mirrors 000456 1779870657.026064 read 0x021ddf thru 0x02f7a6 (0xd9c8 bytes) 000458 1779870657.036745 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000460 1779870657.335038 MIRROR_RESYNC 3 mirrors 000461 1779870657.576418 trunc from 0x0313e9 to 0x00a6e9 (0x26cff bytes) 000462 1779870657.721447 mapwrite 0x01654f thru 0x01aba8 (0x465a bytes) 000464 1779870657.796331 read 0x00e17c thru 0x0138a3 (0x5728 bytes) 000466 1779870657.807591 read 0x018261 thru 0x01aba8 (0x2948 bytes) 000467 1779870657.808180 write 0x03ffee thru 0x03ffff (0x12 bytes) 000468 1779870657.809133 read 0x00a5a0 thru 0x01659b (0xbffc bytes) 000471 1779870657.834746 read 0x03e35e thru 0x03ffff (0x1ca2 bytes) 000472 1779870657.836229 write 0x03a568 thru 0x03ffff (0x5a98 bytes) 000473 1779870657.839625 trunc from 0x040000 to 0x0320e5 (0xdf1a bytes) 000474 1779870657.867818 write 0x025f15 thru 0x02700f (0x10fb bytes) 000475 1779870657.870938 mapwrite 0x035ccd thru 0x03ffff (0xa333 bytes) 000476 1779870657.971193 trunc from 0x040000 to 0x01ce25 (0x231da bytes) 000478 1779870657.995481 mapread 0x01464e thru 0x018590 (0x3f43 bytes) 000479 1779870657.999308 mapread 0x00fd6a thru 0x01b3b6 (0xb64d bytes) 000480 1779870658.002302 write 0x0180ab thru 0x020258 (0x81ae bytes) 000482 1779870658.008417 mapread 0x00078c thru 0x0054b4 (0x4d29 bytes) 000484 1779870658.013957 mapwrite 0x03dca7 thru 0x03ffff (0x2359 bytes) 000485 1779870658.122290 read 0x01f5f8 thru 0x02765b (0x8064 bytes) 000488 1779870658.136916 trunc from 0x040000 to 0x032c1d (0xd3e2 bytes) 000489 1779870658.159621 mapwrite 0x0303df thru 0x034754 (0x4376 bytes) 000492 1779870658.268200 read 0x0290da thru 0x034754 (0xb67b bytes) 000493 1779870658.268900 trunc from 0x034755 to 0x02e21c (0x6538 bytes) 000495 1779870658.291487 read 0x02e15d thru 0x02e21c (0xc0 bytes) 000496 1779870658.316236 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e21d): Mirror 9: 0x3d815ed5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e21d): Mirror 9: 0x3d815ed5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e21d: skipped 000497 1779870658.576449 trunc from 0x02e21d to 0x039225 (0xb009 bytes) 000499 1779870658.596259 read 0x0169a9 thru 0x01c3e4 (0x5a3c bytes) 000501 1779870658.605947 MIRROR_RESYNC 2 mirrors 000502 1779870659.062052 trunc from 0x039226 to 0x017566 (0x21cbf bytes) 000503 1779870659.148587 mapwrite 0x014c79 thru 0x0235c7 (0xe94f bytes) 000505 1779870659.248046 mapread 0x0090b6 thru 0x014b1d (0xba68 bytes) 000506 1779870659.261394 write 0x02f023 thru 0x034c40 (0x5c1e bytes) 000507 1779870659.262662 mapwrite 0x03baf1 thru 0x03ffff (0x450f bytes) 000511 1779870659.366600 mapwrite 0x01bd21 thru 0x02895a (0xcc3a bytes) 000513 1779870659.415664 write 0x03b74b thru 0x03ffff (0x48b5 bytes) 000514 1779870659.421138 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x5326f80f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x61b7ed92 Mirror 10: 0x61b7ed92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000516 1779870660.180644 read 0x03923d thru 0x03d9e4 (0x47a8 bytes) 000517 1779870660.189671 mapwrite 0x03e5af thru 0x03ffff (0x1a51 bytes) 000519 1779870660.335462 mapread 0x011bd1 thru 0x01a908 (0x8d38 bytes) 000520 1779870660.352750 mapread 0x03a71e thru 0x03e551 (0x3e34 bytes) 000521 1779870660.353963 read 0x010447 thru 0x01dc53 (0xd80d bytes) 000522 1779870660.356180 write 0x012416 thru 0x021f0c (0xfaf7 bytes) 000524 1779870660.359123 mapwrite 0x02eadd thru 0x0374c3 (0x89e7 bytes) 000526 1779870660.413815 write 0x011f19 thru 0x01e57e (0xc666 bytes) 000527 1779870660.415227 mapread 0x00d9b3 thru 0x00e65a (0xca8 bytes) 000528 1779870660.418014 write 0x02eb90 thru 0x034308 (0x5779 bytes) 000529 1779870660.419611 trunc from 0x040000 to 0x0103dd (0x2fc22 bytes) 000531 1779870660.461606 write 0x025cda thru 0x032961 (0xcc88 bytes) 000532 1779870660.464391 mapread 0x02debf thru 0x0306d0 (0x2812 bytes) 000534 1779870660.467462 mapwrite 0x011ae2 thru 0x01cd7a (0xb299 bytes) 000537 1779870660.586655 read 0x023f97 thru 0x02aef9 (0x6f63 bytes) 000538 1779870660.593976 mapwrite 0x02e8bb thru 0x030ca6 (0x23ec bytes) 000539 1779870660.627947 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x32962): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32962): Mirror 9: 0x25da16a9 (stale) Mirror 10: 0xda31ad71 (stale) CRC-32 checksum value for chunk [0, 0x32962): Mirror 1: 0xf41d41fb Mirror 11: 0xf41d41fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32962: skipped 000542 1779870661.308975 trunc from 0x032962 to 0x011c47 (0x20d1a bytes) 000543 1779870661.409560 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x11c48): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c48): Mirror 9: 0xc4b91ef9 (stale) Mirror 10: 0xc4b91ef9 (stale) Mirror 11: 0x2bf48a4e (stale) CRC-32 checksum value for chunk [0, 0x11c48): Mirror 1: 0x2bf48a4e Mirror 12: 0x2bf48a4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c48: skipped 000544 1779870662.130374 write 0x0294a0 thru 0x03292f (0x9490 bytes) 000545 1779870662.192173 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32930): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32930): Mirror 9: 0x2dd9479f (stale) Mirror 10: 0x18b39b68 (stale) Mirror 11: 0xb35d115b (stale) Mirror 12: 0xcc9aadeb (stale) CRC-32 checksum value for chunk [0, 0x32930): Mirror 1: 0x3742bdaa Mirror 13: 0x3742bdaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32930: skipped 000546 1779870663.007243 write 0x0300bd thru 0x03795d (0x78a1 bytes) 000547 1779870663.094580 read 0x021fc1 thru 0x02c697 (0xa6d7 bytes) 000548 1779870663.119625 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3795e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3795e): Mirror 9: 0x7cf61e9e (stale) Mirror 10: 0xcaaee42c (stale) Mirror 11: 0xd3132b29 (stale) Mirror 12: 0x155635d7 (stale) Mirror 13: 0xfb63bee (stale) CRC-32 checksum value for chunk [0, 0x3795e): Mirror 1: 0xc546b1fe Mirror 14: 0xc546b1fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3795e: skipped 000549 1779870663.819602 trunc from 0x03795e to 0x01b312 (0x1c64b bytes) 000551 1779870663.929957 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b313): Mirror 10: 0xac21180b (stale) Mirror 11: 0xd53a2d2f (stale) Mirror 12: 0x8357f3f7 (stale) Mirror 13: 0x8357f3f7 (stale) Mirror 14: 0x8357f3f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b313): Mirror 10: 0xac21180b (stale) Mirror 11: 0xd53a2d2f (stale) Mirror 12: 0x8357f3f7 (stale) Mirror 13: 0x8357f3f7 (stale) Mirror 14: 0x8357f3f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b313: skipped 000553 1779870664.311326 read 0x014080 thru 0x01b312 (0x7293 bytes) 000554 1779870664.320326 trunc from 0x01b313 to 0x031194 (0x15e82 bytes) 000556 1779870664.334525 read 0x01ced7 thru 0x020d19 (0x3e43 bytes) 000557 1779870664.347410 read 0x02652e thru 0x02c515 (0x5fe8 bytes) 000560 1779870664.350439 write 0x037d1d thru 0x03ffff (0x82e3 bytes) 000562 1779870664.359116 MIRROR_RESYNC 6 mirrors 000563 1779870665.479536 write 0x014e8e thru 0x018b6b (0x3cde bytes) 000565 1779870665.553288 read 0x038287 thru 0x039ade (0x1858 bytes) 000566 1779870665.558366 write 0x03a978 thru 0x03ffff (0x5688 bytes) 000567 1779870665.570507 read 0x032c39 thru 0x036765 (0x3b2d bytes) 000569 1779870665.588254 read 0x00b330 thru 0x01a165 (0xee36 bytes) 000570 1779870665.589277 read 0x01eb1d thru 0x025c4e (0x7132 bytes) 000574 1779870665.594189 read 0x00cf29 thru 0x00ddbc (0xe94 bytes) 000575 1779870665.595492 mapread 0x0215ad thru 0x030568 (0xefbc bytes) 000576 1779870665.597381 read 0x02fdad thru 0x03ecfb (0xef4f bytes) 000578 1779870665.599403 write 0x03b30f thru 0x03ffff (0x4cf1 bytes) 000580 1779870665.602260 write 0x007173 thru 0x00c1ea (0x5078 bytes) 000581 1779870665.607660 MIRROR_RESYNC 6 mirrors 000583 1779870666.329587 mapread 0x0202cf thru 0x02e5a5 (0xe2d7 bytes) 000585 1779870666.356974 mapwrite 0x025a60 thru 0x02d0e3 (0x7684 bytes) 000586 1779870666.496211 write 0x01bba8 thru 0x0287da (0xcc33 bytes) 000587 1779870666.514267 write 0x03a3a0 thru 0x03ffff (0x5c60 bytes) 000588 1779870666.534366 read 0x019eef thru 0x028ed0 (0xefe2 bytes) 000591 1779870666.537111 mapread 0x005653 thru 0x00e974 (0x9322 bytes) 000592 1779870666.552885 mapwrite 0x02e655 thru 0x0317f1 (0x319d bytes) 000593 1779870666.602075 mapread 0x02134b thru 0x022117 (0xdcd bytes) 000594 1779870666.603604 trunc from 0x040000 to 0x002e1e (0x3d1e1 bytes) 000595 1779870666.626774 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e1f): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e1f): Mirror 10: 0xe1665e53 (stale) Mirror 11: 0xe1665e53 (stale) Mirror 12: 0xe1665e53 (stale) Mirror 13: 0xe1665e53 (stale) Mirror 14: 0xe1665e53 (stale) CRC-32 checksum value for chunk [0, 0x2e1f): Mirror 1: 0xe1665e53 Mirror 15: 0xe1665e53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e1f: skipped 000596 1779870667.322960 write 0x00c172 thru 0x01a959 (0xe7e8 bytes) 000597 1779870667.387714 read 0x008976 thru 0x016ddd (0xe468 bytes) 000598 1779870667.392620 write 0x01c6dc thru 0x026d47 (0xa66c bytes) 000599 1779870667.395271 mapread 0x0251b0 thru 0x026d47 (0x1b98 bytes) 000601 1779870667.397588 mapwrite 0x01f738 thru 0x024aa8 (0x5371 bytes) 000602 1779870667.454317 mapwrite 0x01d47d thru 0x029d68 (0xc8ec bytes) 000604 1779870667.560271 mapread 0x01fb9d thru 0x0234a6 (0x390a bytes) 000605 1779870667.561665 read 0x007f24 thru 0x010ee5 (0x8fc2 bytes) 000606 1779870667.581440 mapwrite 0x02dcd2 thru 0x0327e3 (0x4b12 bytes) 000609 1779870667.691339 trunc from 0x0327e4 to 0x03cdaf (0xa5cc bytes) 000610 1779870667.708515 mapread 0x00862e thru 0x00c16f (0x3b42 bytes) 000612 1779870667.710655 trunc from 0x03cdb0 to 0x004d31 (0x3807e bytes) 000613 1779870667.725203 write 0x03c588 thru 0x03e724 (0x219d bytes) 000614 1779870667.727006 read 0x02a42d thru 0x02b31e (0xef2 bytes) 000619 1779870667.745501 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e725): Mirror 10: 0xfbfc3d5d (stale) Mirror 12: 0xfbfc3d5d (stale) Mirror 13: 0xfbfc3d5d (stale) Mirror 14: 0xfbfc3d5d (stale) Mirror 15: 0xa8376269 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e725): Mirror 10: 0xfbfc3d5d (stale) Mirror 12: 0xfbfc3d5d (stale) Mirror 13: 0xfbfc3d5d (stale) Mirror 14: 0xfbfc3d5d (stale) Mirror 15: 0xa8376269 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e725: skipped 000620 1779870668.108299 MIRROR_RESYNC 6 mirrors 000621 1779870668.927798 mapread 0x010e92 thru 0x0196fe (0x886d bytes) 000623 1779870668.944630 read 0x036820 thru 0x03e724 (0x7f05 bytes) 000626 1779870668.954593 read 0x024d00 thru 0x029700 (0x4a01 bytes) 000627 1779870668.965722 mapwrite 0x006be1 thru 0x012e3d (0xc25d bytes) 000628 1779870669.130042 trunc from 0x03e725 to 0x00ab0a (0x33c1a bytes) 000629 1779870669.147741 mapread 0x007c31 thru 0x00ab0a (0x2eda bytes) 000630 1779870669.151627 trunc from 0x00ab0b to 0x028671 (0x1db67 bytes) 000631 1779870669.173761 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28672): Mirror 12: 0xdf3ab71e (stale) Mirror 13: 0xdf3ab71e (stale) Mirror 14: 0xdf3ab71e (stale) Mirror 15: 0xdf3ab71e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28672): Mirror 12: 0xdf3ab71e (stale) Mirror 13: 0xdf3ab71e (stale) Mirror 14: 0xdf3ab71e (stale) Mirror 15: 0xdf3ab71e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28672: skipped 000632 1779870669.441013 mapwrite 0x00f8a7 thru 0x0165f9 (0x6d53 bytes) 000633 1779870669.504205 MIRROR_RESYNC 5 mirrors 000634 1779870670.228434 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000635 1779870670.512551 read 0x01a75a thru 0x02677b (0xc022 bytes) 000640 1779870670.530050 read 0x00fb98 thru 0x01fae9 (0xff52 bytes) 000643 1779870670.539177 mapread 0x00df8c thru 0x015c80 (0x7cf5 bytes) 000648 1779870670.542291 trunc from 0x028672 to 0x029584 (0xf13 bytes) 000649 1779870670.609427 write 0x006996 thru 0x014030 (0xd69b bytes) 000652 1779870670.621007 trunc from 0x029585 to 0x020620 (0x8f64 bytes) 000653 1779870670.644603 mapread 0x00f795 thru 0x01d74a (0xdfb6 bytes) 000654 1779870670.683344 read 0x0119b5 thru 0x01f6b1 (0xdcfd bytes) 000657 1779870670.693507 read 0x009b41 thru 0x009c7a (0x13a bytes) 000659 1779870670.694846 mapread 0x00d02d thru 0x016646 (0x961a bytes) 000660 1779870670.699875 MIRROR_RESYNC 4 mirrors 000661 1779870671.392930 read 0x0013ee thru 0x00af6e (0x9b81 bytes) 000662 1779870671.398666 write 0x02048d thru 0x027bae (0x7722 bytes) 000664 1779870671.445562 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27baf): Mirror 13: 0x36ba3373 (stale) Mirror 14: 0x36ba3373 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27baf): Mirror 13: 0x36ba3373 (stale) Mirror 14: 0x36ba3373 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27baf: skipped 000665 1779870671.654165 read 0x015046 thru 0x01677a (0x1735 bytes) 000666 1779870671.660593 trunc from 0x027baf to 0x030735 (0x8b87 bytes) 000668 1779870671.683586 MIRROR_RESYNC 3 mirrors 000670 1779870672.286188 mapread 0x025b76 thru 0x030735 (0xabc0 bytes) 000672 1779870672.320466 MIRROR_RESYNC 3 mirrors 000674 1779870672.606502 trunc from 0x030736 to 0x01bdef (0x14946 bytes) 000675 1779870672.682755 mapread 0x01b791 thru 0x01baf4 (0x364 bytes) 000676 1779870672.690735 write 0x0270a0 thru 0x02ff84 (0x8ee5 bytes) 000677 1779870672.694955 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ff85): Mirror 13: 0xa30eddc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff85): Mirror 13: 0xa30eddc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff85: skipped 000679 1779870672.989357 trunc from 0x02ff85 to 0x03cd2c (0xcda8 bytes) 000682 1779870673.032157 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000683 1779870673.160682 read 0x00dcb1 thru 0x0199f0 (0xbd40 bytes) 000684 1779870673.169079 read 0x02482e thru 0x02be96 (0x7669 bytes) 000686 1779870673.193376 mapread 0x03b344 thru 0x03cd2c (0x19e9 bytes) 000687 1779870673.194450 write 0x0147e9 thru 0x01f54b (0xad63 bytes) 000688 1779870673.198679 mapread 0x035836 thru 0x038cad (0x3478 bytes) 000690 1779870673.204506 write 0x029e54 thru 0x02be8d (0x203a bytes) 000691 1779870673.205444 read 0x021177 thru 0x02b677 (0xa501 bytes) 000692 1779870673.206227 trunc from 0x03cd2d to 0x02bc45 (0x110e7 bytes) truncating to largest ever: 0x3f827 000695 1779870673.229469 trunc from 0x02bc46 to 0x03f826 (0x13be1 bytes) 000696 1779870673.253419 read 0x021242 thru 0x02715a (0x5f19 bytes) 000697 1779870673.254540 trunc from 0x03f827 to 0x023a6e (0x1bdb8 bytes) 000698 1779870673.270240 trunc from 0x023a6f to 0x012731 (0x1133d bytes) 000701 1779870673.288695 MIRROR_EXTEND to 2 mirrors 000702 1779870673.725422 mapwrite 0x024e18 thru 0x02b8ce (0x6ab7 bytes) 000703 1779870673.933018 MIRROR_RESYNC 2 mirrors 000704 1779870674.392597 mapwrite 0x01b02b thru 0x0287ef (0xd7c5 bytes) 000708 1779870674.619578 mapwrite 0x01454e thru 0x01ae8b (0x693e bytes) 000709 1779870674.731953 write 0x039ff5 thru 0x03ffff (0x600b bytes) 000710 1779870674.734039 mapwrite 0x01f84b thru 0x023a79 (0x422f bytes) 000713 1779870674.781577 write 0x0014a0 thru 0x00fb9c (0xe6fd bytes) 000715 1779870674.819434 read 0x036eeb thru 0x03ffff (0x9115 bytes) 000717 1779870674.831509 mapread 0x021345 thru 0x030651 (0xf30d bytes) 000718 1779870674.850471 trunc from 0x040000 to 0x00a308 (0x35cf7 bytes) 000719 1779870674.886417 trunc from 0x00a309 to 0x0284c9 (0x1e1c1 bytes) 000721 1779870674.907377 mapwrite 0x01368a thru 0x021636 (0xdfad bytes) 000722 1779870675.001989 read 0x01ea88 thru 0x022746 (0x3cbf bytes) 000723 1779870675.002795 trunc from 0x0284ca to 0x004db7 (0x23712 bytes) 000725 1779870675.033547 mapwrite 0x021d68 thru 0x028744 (0x69dd bytes) 000726 1779870675.109986 mapwrite 0x0003d4 thru 0x002329 (0x1f56 bytes) 000728 1779870675.162957 MIRROR_RESYNC 2 mirrors 000730 1779870675.704991 read 0x0279a6 thru 0x027e1c (0x477 bytes) 000732 1779870675.708281 mapwrite 0x00847d thru 0x00cd1e (0x48a2 bytes) 000733 1779870675.831662 trunc from 0x028745 to 0x02588b (0x2eb9 bytes) 000735 1779870675.911792 MIRROR_RESYNC 2 mirrors 000736 1779870676.415326 MIRROR_RESYNC 2 mirrors 000744 1779870676.699302 MIRROR_EXTEND to 3 mirrors 000745 1779870677.364530 mapread 0x01cabb thru 0x023933 (0x6e79 bytes) 000746 1779870677.419554 mapread 0x0169e8 thru 0x01ecd6 (0x82ef bytes) 000747 1779870677.443553 MIRROR_RESYNC 3 mirrors 000748 1779870677.758143 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000751 1779870678.050514 MIRROR_RESYNC 2 mirrors 000752 1779870678.398225 mapread 0x0097f4 thru 0x013385 (0x9b92 bytes) 000754 1779870678.420716 trunc from 0x02588c to 0x0010a5 (0x247e6 bytes) 000756 1779870678.489598 trunc from 0x0010a6 to 0x02e59d (0x2d4f8 bytes) 000757 1779870678.503324 mapread 0x01641b thru 0x02583b (0xf421 bytes) 000758 1779870678.519369 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e59e): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e59e): Mirror 3: 0x824b9b44 (stale) CRC-32 checksum value for chunk [0, 0x2e59e): Mirror 1: 0x77864886 Mirror 4: 0x77864886 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e59e: skipped 000759 1779870679.167579 trunc from 0x02e59e to 0x012a19 (0x1bb84 bytes) 000761 1779870679.280881 mapread 0x004fe6 thru 0x00b326 (0x6341 bytes) 000762 1779870679.306005 MIRROR_RESYNC 3 mirrors 000764 1779870679.843049 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000765 1779870680.113092 write 0x03a23b thru 0x03ffff (0x5dc5 bytes) 000771 1779870680.184390 mapread 0x00162d thru 0x0078cc (0x62a0 bytes) 000773 1779870680.206990 trunc from 0x040000 to 0x00e252 (0x31dad bytes) 000774 1779870680.256894 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) 000775 1779870680.374746 mapread 0x00cb82 thru 0x00e252 (0x16d1 bytes) 000778 1779870680.399194 write 0x037d93 thru 0x03ae73 (0x30e1 bytes) 000780 1779870680.401752 read 0x01d1e0 thru 0x02c989 (0xf7aa bytes) 000781 1779870680.409289 write 0x024ca8 thru 0x03260e (0xd967 bytes) 000782 1779870680.415414 trunc from 0x03ae74 to 0x0233d0 (0x17aa3 bytes) 000786 1779870680.437004 read 0x014f51 thru 0x016ca5 (0x1d55 bytes) 000787 1779870680.446301 read 0x0021fe thru 0x0043b2 (0x21b5 bytes) 000789 1779870680.449282 trunc from 0x0233d1 to 0x01742f (0xbfa1 bytes) 000791 1779870680.467193 write 0x0295ef thru 0x02cbfe (0x3610 bytes) 000793 1779870680.469812 write 0x01d2dc thru 0x01ef77 (0x1c9c bytes) 000794 1779870680.476387 mapread 0x016f0a thru 0x017eef (0xfe6 bytes) 000798 1779870680.480139 read 0x00ecc2 thru 0x015165 (0x64a4 bytes) 000799 1779870680.480931 mapread 0x02c1bd thru 0x02cbfe (0xa42 bytes) 000800 1779870680.482381 trunc from 0x02cbff to 0x00c8fa (0x20304 bytes) 000801 1779870680.497727 mapread 0x0016a7 thru 0x00697f (0x52d9 bytes) 000803 1779870680.504197 trunc from 0x00c8fb to 0x0212b2 (0x149b8 bytes) 000805 1779870680.525494 mapread 0x01085d thru 0x01fb72 (0xf316 bytes) 000806 1779870680.537330 write 0x014496 thru 0x015a1e (0x1589 bytes) 000808 1779870680.539511 write 0x03bb90 thru 0x03ffff (0x4470 bytes) 000811 1779870680.542710 mapwrite 0x01bb9e thru 0x02789a (0xbcfd bytes) 000815 1779870680.609170 mapwrite 0x02c620 thru 0x03b565 (0xef46 bytes) 000819 1779870680.649178 mapread 0x020b24 thru 0x02615d (0x563a bytes) 000822 1779870680.652905 read 0x00480a thru 0x005cfb (0x14f2 bytes) 000823 1779870680.653998 mapread 0x03c804 thru 0x03ffff (0x37fc bytes) 000824 1779870680.659309 MIRROR_EXTEND to 2 mirrors 000825 1779870681.090382 mapread 0x00a64a thru 0x00c6f2 (0x20a9 bytes) 000826 1779870681.107484 read 0x00d256 thru 0x0176d8 (0xa483 bytes) 000828 1779870681.119639 write 0x03da00 thru 0x03fa54 (0x2055 bytes) 000829 1779870681.193518 mapread 0x03ee35 thru 0x03ffff (0x11cb bytes) 000830 1779870681.194782 trunc from 0x040000 to 0x006e1f (0x391e0 bytes) 000832 1779870681.245259 trunc from 0x006e20 to 0x0218ce (0x1aaaf bytes) 000835 1779870681.264440 mapread 0x0204e4 thru 0x0218ce (0x13eb bytes) 000837 1779870681.274916 mapread 0x0034a1 thru 0x00d5f5 (0xa155 bytes) 000838 1779870681.282507 mapread 0x01e8ee thru 0x0218ce (0x2fe1 bytes) 000839 1779870681.284041 trunc from 0x0218cf to 0x03801d (0x1674f bytes) 000841 1779870681.304083 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3801e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3801e): Mirror 2: 0x20760f2d (stale) CRC-32 checksum value for chunk [0, 0x3801e): Mirror 1: 0xef03e2fa Mirror 3: 0xef03e2fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3801e: skipped 000842 1779870681.975174 trunc from 0x03801e to 0x012aa4 (0x25579 bytes) 000843 1779870682.110641 trunc from 0x012aa5 to 0x02e681 (0x1bbdd bytes) 000844 1779870682.136014 read 0x00e6c8 thru 0x010c7b (0x25b4 bytes) 000848 1779870682.145383 mapread 0x023c69 thru 0x02e681 (0xaa19 bytes) 000853 1779870682.172558 MIRROR_RESYNC 3 mirrors 000854 1779870682.740574 write 0x02e3b5 thru 0x03876f (0xa3bb bytes) 000855 1779870682.803366 mapread 0x00e4ae thru 0x018e19 (0xa96c bytes) 000857 1779870682.829697 read 0x00fdf7 thru 0x0114ed (0x16f7 bytes) 000859 1779870682.831060 mapwrite 0x0112b3 thru 0x01d5fe (0xc34c bytes) 000861 1779870682.883427 trunc from 0x038770 to 0x00ead0 (0x29c9f bytes) 000862 1779870682.930096 mapread 0x00345a thru 0x00ead0 (0xb677 bytes) 000864 1779870682.940410 write 0x02b323 thru 0x0337bf (0x849d bytes) 000866 1779870682.946715 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x337c0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337c0): Mirror 2: 0x2c7f825e (stale) Mirror 3: 0x2c7f825e (stale) CRC-32 checksum value for chunk [0, 0x337c0): Mirror 1: 0x1dd4d193 Mirror 4: 0x1dd4d193 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337c0: skipped 000868 1779870683.744984 mapread 0x00a512 thru 0x011ccc (0x77bb bytes) 000871 1779870683.788086 mapread 0x018be3 thru 0x022635 (0x9a53 bytes) 000873 1779870683.821136 read 0x015e00 thru 0x01eaa3 (0x8ca4 bytes) 000876 1779870683.836491 mapwrite 0x03113c thru 0x03324d (0x2112 bytes) 000877 1779870683.971444 read 0x006100 thru 0x010c99 (0xab9a bytes) 000883 1779870683.985267 mapwrite 0x0067f1 thru 0x00f13b (0x894b bytes) 000884 1779870684.025125 mapwrite 0x034456 thru 0x03aa77 (0x6622 bytes) 000885 1779870684.102936 mapread 0x0067b2 thru 0x00fc51 (0x94a0 bytes) 000886 1779870684.104214 mapwrite 0x00e603 thru 0x015f1e (0x791c bytes) 000887 1779870684.128734 mapread 0x006d56 thru 0x012c39 (0xbee4 bytes) 000888 1779870684.129600 mapread 0x00a5e0 thru 0x014341 (0x9d62 bytes) 000889 1779870684.130454 write 0x023347 thru 0x02b842 (0x84fc bytes) 000890 1779870684.131276 read 0x015512 thru 0x024c50 (0xf73f bytes) 000891 1779870684.132269 trunc from 0x03aa78 to 0x0078fb (0x3317c bytes) 000893 1779870684.147268 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x78fc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x78fc): Mirror 2: 0x38466937 (stale) Mirror 3: 0x38466937 (stale) Mirror 4: 0x38466937 (stale) CRC-32 checksum value for chunk [0, 0x78fc): Mirror 1: 0xa1a715b5 Mirror 5: 0xa1a715b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x78fc: skipped 000894 1779870684.969543 trunc from 0x0078fc to 0x01651d (0xec22 bytes) 000895 1779870685.102072 MIRROR_RESYNC 5 mirrors 000896 1779870686.175341 trunc from 0x01651e to 0x022455 (0xbf38 bytes) 000904 1779870686.250704 write 0x02c973 thru 0x032756 (0x5de4 bytes) 000906 1779870686.261683 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32757): Mirror 2: 0x7bb21a70 (stale) Mirror 3: 0x7bb21a70 (stale) Mirror 5: 0x7bb21a70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32757): Mirror 2: 0x7bb21a70 (stale) Mirror 3: 0x7bb21a70 (stale) Mirror 5: 0x7bb21a70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32757: skipped 000907 1779870686.612933 mapwrite 0x022267 thru 0x025e82 (0x3c1c bytes) 000908 1779870686.673569 mapwrite 0x018f79 thru 0x01db9d (0x4c25 bytes) 000909 1779870686.732422 write 0x0287fd thru 0x036869 (0xe06d bytes) 000910 1779870686.735088 write 0x038997 thru 0x03ffff (0x7669 bytes) 000911 1779870686.737778 mapread 0x012e4c thru 0x019eb4 (0x7069 bytes) 000914 1779870686.742280 mapwrite 0x031607 thru 0x031dd6 (0x7d0 bytes) 000915 1779870686.806151 trunc from 0x040000 to 0x02300b (0x1cff4 bytes) 000916 1779870686.823619 mapread 0x00db1a thru 0x018ec1 (0xb3a8 bytes) 000917 1779870686.832731 mapwrite 0x02c369 thru 0x036139 (0x9dd1 bytes) 000920 1779870686.908617 trunc from 0x03613a to 0x006307 (0x2fe32 bytes) 000922 1779870686.928304 read 0x00511b thru 0x006307 (0x11ed bytes) 000925 1779870686.936050 mapwrite 0x027a1a thru 0x033d43 (0xc32a bytes) 000926 1779870686.998941 read 0x00f4c6 thru 0x01b980 (0xc4bb bytes) 000927 1779870687.019352 mapwrite 0x02ffe0 thru 0x03c512 (0xc533 bytes) 000928 1779870687.102202 trunc from 0x03c513 to 0x02a43d (0x120d5 bytes) 000929 1779870687.117371 read 0x00c3a3 thru 0x01c2a8 (0xff06 bytes) 000930 1779870687.118317 read 0x01f921 thru 0x02a43d (0xab1d bytes) 000931 1779870687.119609 mapwrite 0x01bb91 thru 0x021ae4 (0x5f54 bytes) 000932 1779870687.153037 mapread 0x01139f thru 0x017b70 (0x67d2 bytes) 000933 1779870687.153771 mapwrite 0x02caea thru 0x03416d (0x7684 bytes) 000934 1779870687.227113 MIRROR_RESYNC 4 mirrors 000935 1779870687.881366 MIRROR_EXTEND to 5 mirrors 000938 1779870688.383646 trunc from 0x03416e to 0x03e713 (0xa5a6 bytes) 000941 1779870688.471949 write 0x023519 thru 0x030602 (0xd0ea bytes) 000942 1779870688.480000 trunc from 0x03e714 to 0x00dcdc (0x30a37 bytes) 000943 1779870688.493290 write 0x0398f5 thru 0x03ffff (0x670b bytes) 000945 1779870688.495458 read 0x03fb4a thru 0x03ffff (0x4b6 bytes) 000947 1779870688.497052 mapread 0x02967c thru 0x03381c (0xa1a1 bytes) 000948 1779870688.512294 trunc from 0x040000 to 0x006b7b (0x39484 bytes) 000949 1779870688.524985 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6b7c): Mirror 2: 0x4879888e (stale) Mirror 5: 0x4879888e (stale) Mirror 6: 0x4879888e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b7c): Mirror 2: 0x4879888e (stale) Mirror 5: 0x4879888e (stale) Mirror 6: 0x4879888e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b7c: skipped 000952 1779870688.758432 trunc from 0x006b7c to 0x00f7fe (0x8c83 bytes) 000953 1779870688.766618 mapwrite 0x022f11 thru 0x031054 (0xe144 bytes) 000954 1779870688.821205 mapread 0x01caa7 thru 0x0260bb (0x9615 bytes) 000955 1779870688.828046 mapwrite 0x0003d3 thru 0x003cf9 (0x3927 bytes) 000957 1779870688.867218 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31055): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31055): Mirror 2: 0xa5ddd7a (stale) Mirror 5: 0xa5ddd7a (stale) Mirror 6: 0xa5ddd7a (stale) CRC-32 checksum value for chunk [0, 0x31055): Mirror 1: 0x9bf1403 Mirror 7: 0x9bf1403 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31055: skipped 000958 1779870689.616323 write 0x00f93a thru 0x0114e6 (0x1bad bytes) 000959 1779870689.698328 mapread 0x009d42 thru 0x014da1 (0xb060 bytes) 000960 1779870689.714648 mapread 0x0052ce thru 0x0073ec (0x211f bytes) 000961 1779870689.719176 MIRROR_RESYNC 5 mirrors 000962 1779870690.456515 mapwrite 0x0269dc thru 0x02c688 (0x5cad bytes) 000964 1779870690.586612 MIRROR_RESYNC 5 mirrors 000967 1779870691.228451 mapread 0x02aa72 thru 0x030739 (0x5cc8 bytes) 000969 1779870691.252062 trunc from 0x031055 to 0x03f6d6 (0xe682 bytes) 000970 1779870691.300150 read 0x00faea thru 0x0198ae (0x9dc5 bytes) 000972 1779870691.309740 mapread 0x01c1c2 thru 0x02adfc (0xec3b bytes) 000973 1779870691.323171 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 2: 0x3d8337ac (stale) Mirror 6: 0x3d8337ac (stale) Mirror 7: 0x3d8337ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 2: 0x3d8337ac (stale) Mirror 6: 0x3d8337ac (stale) Mirror 7: 0x3d8337ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6d7: skipped 000975 1779870691.621065 mapread 0x03d2df thru 0x03f6d6 (0x23f8 bytes) 000976 1779870691.653428 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 6: 0x3d8337ac (stale) Mirror 7: 0x3d8337ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 6: 0x3d8337ac (stale) Mirror 7: 0x3d8337ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6d7: skipped 000978 1779870691.931986 MIRROR_RESYNC 3 mirrors 000980 1779870692.571816 MIRROR_RESYNC 3 mirrors 000982 1779870692.834963 mapread 0x0336b4 thru 0x03cd06 (0x9653 bytes) 000985 1779870692.851051 mapwrite 0x012e4b thru 0x0154cf (0x2685 bytes) 000987 1779870692.993743 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 7: 0x3d8337ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 7: 0x3d8337ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6d7: skipped 000988 1779870693.233638 mapwrite 0x008bd0 thru 0x015ba3 (0xcfd4 bytes) 000992 1779870693.324932 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 7: 0x3d8337ac (stale) CRC-32 checksum value for chunk [0, 0x3f6d7): Mirror 1: 0x2fe15fe5 Mirror 8: 0x2fe15fe5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6d7: skipped 000993 1779870693.972272 write 0x038f74 thru 0x03ffff (0x708c bytes) 000995 1779870694.064093 MIRROR_RESYNC 3 mirrors 000996 1779870694.619977 write 0x01e8be thru 0x026847 (0x7f8a bytes) 000997 1779870694.676788 MIRROR_RESYNC 3 mirrors 000998 1779870695.262767 read 0x00fec2 thru 0x01da5c (0xdb9b bytes) 000999 1779870695.273435 trunc from 0x040000 to 0x02febb (0x10144 bytes) 001000 1779870695.363251 mapread 0x006a34 thru 0x010940 (0x9f0d bytes) All operations completed A-OK! PASS 70a (66s) == sanity-flr test 71: check mirror extend parent fid ==== 04:31:39 (1779870699) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 04:31:41 (1779870701) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x387:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237908 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x387:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x388:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994671 s, 10.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x387:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x388:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x387:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x388:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x389:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 04:31:47 (1779870707) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3697376 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7736320 .. Extending file size to 6349120 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7879680 .. Extending file size to 8105344 .. Extending file size to 8263488 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8194944 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8568448 .. Extending file size to 8446528 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9020000 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9118208 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8626816 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56971 56972 56974 56975 56976 failed 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) 192.168.202.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:33:07 (1779870787) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4535840 .. Extending file size to 3402528 .. Extending file size to 4060384 .. Extending file size to 6258624 .. Extending file size to 5003392 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8926016 .. Extending file size to 5991648 .. Extending file size to 8304736 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58728 58729 58730 58732 58736 done 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) 192.168.202.157@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:34:22 (1779870862) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0895554 s, 23.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227447 s, 9.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf7ff81f0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:34:36 (1779870876) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0851875 s, 24.6 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 04:34:42 (1779870882) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x263:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ba:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ba:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bb:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x39e:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:34:48 (1779870888) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.249908 s, 8.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:34:55 (1779870895) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140825 s, 14.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3be:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3be:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a2:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:35:01 (1779870901) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164998 s, 12.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a4:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:35:32 (1779870932) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg257-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119621 s, 8.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c1:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:36:28 (1779870988) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg257-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.243718 s, 8.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:37:26 (1779871046) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg257-server mds-ost sync done. Creating new pool test_204f oleg257-server: Pool lustre.test_204f created Adding targets to pool oleg257-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg257-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.254139 s, 8.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg257-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg257-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg257-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (72s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:38:39 (1779871119) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:38:44 (1779871124) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:38:49 (1779871129) oleg257-server: Pool lustre.test_206 created Adding targets to pool oleg257-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg257-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg257-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg257-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg257-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:39:12 (1779871152) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844589 s, 12.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047994 s, 21.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb84a09c9 Mirror 3: 0xb84a09c9 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:39:19 (1779871159) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:39:21 (1779871161) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:39:24 (1779871164) Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.116478 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2240000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1640000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1640000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1640000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xfd0000 Read this (4 bytes): BBBB 192.168.202.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg257-client.virtnet /mnt/lustre2 (opts:) PASS 209a (29s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:39:53 (1779871193) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0202361 s, 405 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:39:58 (1779871198) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356698 s, 29.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x288 lmm_fid: [0x200000404:0x288:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x288 lmm_fid: [0x200000404:0x288:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3b3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834568 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-27 04:39:59.000000000 -0400 Modify: 2026-05-27 04:39:59.000000000 -0400 Change: 2026-05-27 04:40:00.000000000 -0400 Birth: 2026-05-27 04:40:00.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:40:04 (1779871204) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:40:06 (1779871206) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.415564 s, 98.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0593685 s, 69.0 kB/s size after second write -rw-r--r-- 1 root root 4096 May 27 04:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x28a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 27 04:40 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:40:13 (1779871213) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:40:20 (1779871220) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1377 sec =========== 04:40:27 (1779871227) === sanity-flr: start cleanup 04:40:28 (1779871228) === === sanity-flr: finish cleanup 04:40:32 (1779871232) ===