-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 17:46:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg616-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 17:46:24 (1786743984) === oleg616-client.virtnet: executing check_config_client /mnt/lustre oleg616-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg616-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c174a543000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c174a543000.idle_timeout=debug disable quota as required oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 17:46:58 (1786744018) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7654] == sanity-flr test 0a: lfs mirror create with -N option == 17:47:00 (1786744020) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:47:18 (1786744038) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:47:21 (1786744041) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:47:24 (1786744044) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:47:46 (1786744066) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:47:49 (1786744069) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 17:47:52 (1786744072) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c174a543000.stats=clear osc.lustre-OST0001-osc-ffff9c174a543000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:48:00 (1786744080) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:48:09 (1786744089) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:48:17 (1786744097) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:48:21 (1786744101) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:48:28 (1786744108) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 17:48:36 (1786744116) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:48:42 (1786744122) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:48:46 (1786744126) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:48:53 (1786744133) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:49:03 (1786744143) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:49:09 (1786744149) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.95108 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261249 s, 4.0 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:49:21 (1786744161) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.228557 s, 36.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:49:30 (1786744170) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:49:41 (1786744181) Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:50:58 (1786744258) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.349882 s, 21.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:51:39 (1786744299) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:53:03 (1786744383) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.20827 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 17:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.112s user 0m0.001s sys 0m0.158s Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c174a543000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.621s user 0m0.001s sys 0m0.198s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.443s user 0m0.001s sys 0m0.170s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg616-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c174a543000.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:54:47 (1786744487) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:54:51 (1786744491) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:54:54 (1786744494) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 17:54:57 (1786744497) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 17:55:04 (1786744504) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.648285 s, 19.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:55:16 (1786744516) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408599 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529926 s, 19.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:55:23 (1786744523) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:55:31 (1786744531) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109347 s, 936 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102455 s, 999 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104598 s, 979 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138821 s, 738 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 17:55:41 (1786744541) create test files with size 15760064 .. bb7def97753dc921aa8c38666a26fee7 - e47503bd8d29d124651eccc1ec48c1bb - 56a96acc587ffc94e20c51fe75069ee9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 17:56:05 (1786744565) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:56:19 (1786744579) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:56:26 (1786744586) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880513 s, 11.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257653 s, 40.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 17:56:35 (1786744595) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107305 s, 19.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.595597 s, 7.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198156 s, 10.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 17:56:50 (1786744610) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 17:56:53 (1786744613) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:56:56 (1786744616) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281488 s, 37.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 17:57:04 (1786744624) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:57:07 (1786744627) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:57:16 (1786744636) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.293972 s, 35.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:57:27 (1786744647) llite.lustre-ffff9c17507c1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.274768 s, 38.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c1770e55000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:57:40 (1786744660) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:58:18 (1786744698) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:58:53 (1786744733) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 17:58:56 (1786744736) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.338595 s, 9.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.134933 s, 23.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf9b904ad (nosync,stale) Mirror 2: 0xf9b904ad (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7f15bdf5d3c9fd754e13bfb71a2c17d7 - mirror 1 checksum: d11850d72c772636b4dc1caec6df159f - mirror 2 checksum: 7f15bdf5d3c9fd754e13bfb71a2c17d7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T17:58:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T17:58:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7f15bdf5d3c9fd754e13bfb71a2c17d7 - mirror 1 checksum: d11850d72c772636b4dc1caec6df159f - mirror 2 checksum: d11850d72c772636b4dc1caec6df159f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T17:58:59-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T17:58:59-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:59:08 (1786744748) Failing mds1 on oleg616-server Stopping /mnt/lustre-mds1 (opts:) on oleg616-server 17:59:14 (1786744754) shut down facet: mds1 facet_host: oleg616-server facet_failover_host: oleg616-server Failover mds1 to oleg616-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-MDT0000 17:59:37 (1786744777) targets are mounted 17:59:37 (1786744777) facet_failover done oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476162 s, 22.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:59:53 (1786744793) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:59:58 (1786744798) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:00:02 (1786744802) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00490793 s, 3.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:00:09 (1786744809) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:00:12 (1786744812) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0212695 s, 193 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0955254 s, 42.9 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:00:23 (1786744823) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:00:45 (1786744845) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:01:15 (1786744875) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:01:40 (1786744900) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:02:06 (1786744926) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 18:02:13 (1786744933) mirror create pid 54355 mirror split pid 54356 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 54355 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54356 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 18:03:21 (1786745001) Seed set to 5004 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x39421 000005 1786745005.005116 trunc from 00000000 to 0x039420 (0x39421 bytes) 000006 1786745005.034570 read 0x023389 thru 0x028ca9 (0x5921 bytes) 000011 1786745005.078307 write 0x00b69b thru 0x0175dd (0xbf43 bytes) 000012 1786745005.108933 write 0x020f0e thru 0x02d9a7 (0xca9a bytes) 000017 1786745005.146018 mapwrite 0x02940a thru 0x033540 (0xa137 bytes) 000019 1786745005.371243 mapread 0x016b3b thru 0x01a556 (0x3a1c bytes) 000020 1786745005.394504 read 0x01341c thru 0x015bf6 (0x27db bytes) 000021 1786745005.398269 mapwrite 0x01bc8f thru 0x02b6f0 (0xfa62 bytes) 000024 1786745005.588108 read 0x027787 thru 0x031a00 (0xa27a bytes) 000025 1786745005.588730 read 0x015ba3 thru 0x01728e (0x16ec bytes) 000026 1786745005.605792 write 0x007ce9 thru 0x011127 (0x943f bytes) 000027 1786745005.628317 mapwrite 0x0312f5 thru 0x036a4f (0x575b bytes) 000028 1786745005.781964 mapread 0x01390b thru 0x016036 (0x272c bytes) 000029 1786745005.783322 trunc from 0x039421 to 0x007d7f (0x316a1 bytes) 000030 1786745005.817469 mapwrite 0x03e9ec thru 0x03ffff (0x1614 bytes) truncating to largest ever: 0x3ef77 000031 1786745005.920539 trunc from 0x040000 to 0x03ef76 (0x1089 bytes) 000033 1786745005.971782 write 0x035593 thru 0x03f388 (0x9df6 bytes) 000035 1786745005.991704 read 0x02f6d2 thru 0x035975 (0x62a4 bytes) 000036 1786745006.029008 mapwrite 0x00568c thru 0x00e92b (0x92a0 bytes) 000041 1786745006.194393 read 0x01c59c thru 0x01fcf7 (0x375c bytes) 000042 1786745006.195350 mapread 0x00d702 thru 0x0115f4 (0x3ef3 bytes) 000043 1786745006.209605 write 0x016051 thru 0x024669 (0xe619 bytes) 000044 1786745006.211025 mapread 0x007a99 thru 0x00c21b (0x4783 bytes) 000046 1786745006.224321 mapwrite 0x022b8b thru 0x02eece (0xc344 bytes) 000047 1786745006.347751 write 0x01133e thru 0x015cd7 (0x499a bytes) 000048 1786745006.349148 read 0x00fcb2 thru 0x018c9e (0x8fed bytes) 000049 1786745006.350277 read 0x02a80e thru 0x033d08 (0x94fb bytes) 000054 1786745006.362195 trunc from 0x03f389 to 0x02c3be (0x12fca bytes) 000058 1786745006.412629 trunc from 0x02c3bf to 0x00e790 (0x1dc2e bytes) 000060 1786745006.425739 mapread 0x00257f thru 0x00da1e (0xb4a0 bytes) 000062 1786745006.437190 trunc from 0x00e791 to 0x029491 (0x1ad01 bytes) 000063 1786745006.473081 trunc from 0x029492 to 0x01338d (0x16104 bytes) 000065 1786745006.504976 write 0x03094c thru 0x03f016 (0xe6cb bytes) 000066 1786745006.519340 read 0x034b47 thru 0x03a441 (0x58fb bytes) 000067 1786745006.520014 mapread 0x035967 thru 0x03aeeb (0x5585 bytes) 000072 1786745006.533274 trunc from 0x03f017 to 0x00b84f (0x337c7 bytes) 000073 1786745006.587360 mapwrite 0x03b95f thru 0x03ffff (0x46a1 bytes) 000074 1786745006.768159 mapwrite 0x004c93 thru 0x00b40f (0x677d bytes) 000077 1786745006.894516 mapwrite 0x02e453 thru 0x03548f (0x703d bytes) 000080 1786745007.010137 trunc from 0x040000 to 0x00a779 (0x35886 bytes) 000081 1786745007.068630 read 0x003c9f thru 0x00a779 (0x6adb bytes) 000082 1786745007.074308 write 0x000b04 thru 0x004ad3 (0x3fd0 bytes) 000085 1786745007.083462 trunc from 0x00a77a to 0x037600 (0x2ce87 bytes) 000086 1786745007.136080 write 0x01babf thru 0x026525 (0xaa67 bytes) 000088 1786745007.157986 mapread 0x021b9f thru 0x02d602 (0xba64 bytes) 000089 1786745007.162946 trunc from 0x037601 to 0x007655 (0x2ffab bytes) 000090 1786745007.217450 mapread 0x0054e1 thru 0x007655 (0x2175 bytes) 000091 1786745007.219548 mapwrite 0x03984f thru 0x03ffff (0x67b1 bytes) 000093 1786745007.337094 mapwrite 0x03976b thru 0x039be3 (0x479 bytes) 000094 1786745007.395169 read 0x003eb8 thru 0x00dc49 (0x9d92 bytes) 000095 1786745007.431115 mapwrite 0x026b99 thru 0x0338f7 (0xcd5f bytes) 000097 1786745007.520264 mapread 0x01fdf8 thru 0x02291a (0x2b23 bytes) 000098 1786745007.523230 read 0x0134e0 thru 0x0135b2 (0xd3 bytes) 000101 1786745007.534593 mapwrite 0x01620e thru 0x0171bd (0xfb0 bytes) 000102 1786745007.643649 mapread 0x005724 thru 0x013bb4 (0xe491 bytes) 000103 1786745007.653944 write 0x02c0ba thru 0x03a3a6 (0xe2ed bytes) 000105 1786745007.662663 read 0x027199 thru 0x030d71 (0x9bd9 bytes) 000108 1786745007.690788 MIRROR_EXTEND to 2 mirrors 000109 1786745009.016693 write 0x02db94 thru 0x035b8f (0x7ffc bytes) 000111 1786745009.127055 write 0x0163dc thru 0x018db1 (0x29d6 bytes) 000113 1786745009.144396 mapwrite 0x008ed9 thru 0x009df4 (0xf1c bytes) 000114 1786745009.222091 mapread 0x0218db thru 0x0275a3 (0x5cc9 bytes) 000115 1786745009.234634 mapread 0x020b34 thru 0x02b046 (0xa513 bytes) 000120 1786745009.245930 write 0x006277 thru 0x008f26 (0x2cb0 bytes) 000122 1786745009.255733 write 0x0268ce thru 0x03162a (0xad5d bytes) 000124 1786745009.258306 trunc from 0x040000 to 0x02ddfa (0x12205 bytes) 000126 1786745009.299053 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ddfb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddfb): Mirror 2: 0xbba2c3c6 (stale) CRC-32 checksum value for chunk [0, 0x2ddfb): Mirror 1: 0x12ae122a Mirror 3: 0x12ae122a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddfb: skipped 000127 1786745010.769168 read 0x026736 thru 0x026caf (0x57a bytes) 000128 1786745010.786650 mapread 0x028efa thru 0x02ddfa (0x4f01 bytes) 000129 1786745010.857036 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ddfb): Mirror 2: 0xbba2c3c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ddfb): Mirror 2: 0xbba2c3c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ddfb: skipped 000130 1786745011.152568 mapread 0x00be45 thru 0x01b03a (0xf1f6 bytes) 000132 1786745011.177979 mapwrite 0x030531 thru 0x03a457 (0x9f27 bytes) 000133 1786745011.269509 mapwrite 0x006b1b thru 0x007489 (0x96f bytes) 000135 1786745011.406544 trunc from 0x03a458 to 0x003edb (0x3657c bytes) 000138 1786745011.444888 trunc from 0x003edc to 0x012a41 (0xeb66 bytes) 000139 1786745011.495189 trunc from 0x012a42 to 0x005c71 (0xcdd0 bytes) 000140 1786745011.535709 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) 000141 1786745011.700336 write 0x020bb6 thru 0x02cd27 (0xc172 bytes) 000142 1786745011.704049 read 0x0128e7 thru 0x01435d (0x1a77 bytes) 000144 1786745011.725620 MIRROR_EXTEND to 2 mirrors 000145 1786745012.509656 read 0x01da44 thru 0x0289c9 (0xaf86 bytes) 000146 1786745012.547145 MIRROR_EXTEND to 3 mirrors 000148 1786745013.358375 mapread 0x01897c thru 0x01b60b (0x2c90 bytes) 000149 1786745013.376817 trunc from 0x02cd28 to 0x0210c3 (0xbc64 bytes) 000150 1786745013.489215 mapwrite 0x02024e thru 0x02f91c (0xf6cf bytes) 000151 1786745013.582115 write 0x00e6cc thru 0x01c9b3 (0xe2e8 bytes) 000152 1786745013.593311 write 0x03d7e2 thru 0x03ffff (0x281e bytes) 000154 1786745013.596112 write 0x02ba53 thru 0x0308ad (0x4e5b bytes) 000155 1786745013.607543 mapwrite 0x015ebb thru 0x023a41 (0xdb87 bytes) 000157 1786745013.729957 mapread 0x0321d0 thru 0x03321d (0x104e bytes) 000164 1786745013.768438 mapread 0x0109bd thru 0x01522a (0x486e bytes) 000168 1786745013.776277 mapwrite 0x01b60d thru 0x027124 (0xbb18 bytes) 000171 1786745013.940358 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xff3fee6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xff3fee6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000172 1786745014.273561 mapwrite 0x010cb1 thru 0x01722f (0x657f bytes) 000173 1786745014.387468 write 0x028ca5 thru 0x02b9f0 (0x2d4c bytes) 000174 1786745014.411699 mapread 0x0062a5 thru 0x0064c9 (0x225 bytes) 000175 1786745014.451360 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) 000178 1786745014.696478 mapread 0x01b552 thru 0x02014d (0x4bfc bytes) 000181 1786745014.727892 mapwrite 0x0341f2 thru 0x03ffff (0xbe0e bytes) 000183 1786745014.796139 mapread 0x0047b0 thru 0x00d688 (0x8ed9 bytes) 000184 1786745014.806300 write 0x00bc53 thru 0x019ad1 (0xde7f bytes) 000185 1786745014.808028 mapread 0x00cd5b thru 0x01b663 (0xe909 bytes) 000186 1786745014.810691 write 0x01c1df thru 0x01f5e1 (0x3403 bytes) 000187 1786745014.811911 mapwrite 0x009c4f thru 0x00dbc0 (0x3f72 bytes) 000188 1786745014.849123 read 0x0246f6 thru 0x02d0bf (0x89ca bytes) 000191 1786745014.858549 trunc from 0x040000 to 0x022f23 (0x1d0dc bytes) 000193 1786745014.880050 trunc from 0x022f24 to 0x0330c0 (0x1019d bytes) 000194 1786745014.900897 write 0x02b05f thru 0x02fc0b (0x4bad bytes) 000195 1786745014.909476 write 0x024931 thru 0x02c215 (0x78e5 bytes) 000196 1786745014.924233 mapwrite 0x0045ab thru 0x00e052 (0x9aa8 bytes) 000207 1786745014.981790 write 0x008edf thru 0x01855b (0xf67d bytes) 000208 1786745014.987309 read 0x010fe8 thru 0x016e98 (0x5eb1 bytes) 000212 1786745014.993024 write 0x037727 thru 0x03b7f4 (0x40ce bytes) 000213 1786745014.995211 trunc from 0x03b7f5 to 0x00253d (0x392b7 bytes) 000214 1786745015.024876 mapread 0x0016bf thru 0x00205a (0x99c bytes) 000215 1786745015.030678 read 0x0008b3 thru 0x00253d (0x1c8b bytes) 000216 1786745015.039925 trunc from 0x00253e to 0x02a9c0 (0x28483 bytes) 000217 1786745015.087337 mapread 0x00c953 thru 0x011743 (0x4df1 bytes) 000218 1786745015.103109 read 0x02883a thru 0x02a9c0 (0x2187 bytes) 000219 1786745015.104120 read 0x022ba7 thru 0x028359 (0x57b3 bytes) 000220 1786745015.105063 read 0x02279c thru 0x026854 (0x40b9 bytes) 000221 1786745015.106071 mapwrite 0x032764 thru 0x033e05 (0x16a2 bytes) 000222 1786745015.188528 trunc from 0x033e06 to 0x02c43d (0x79c8 bytes) 000223 1786745015.202336 MIRROR_EXTEND to 2 mirrors 000224 1786745015.994642 mapread 0x02a147 thru 0x02bca1 (0x1b5b bytes) 000225 1786745016.011294 mapwrite 0x0135aa thru 0x022378 (0xedcf bytes) 000226 1786745016.193253 write 0x03684e thru 0x03dff7 (0x77aa bytes) 000228 1786745016.198150 mapwrite 0x038ba8 thru 0x03ffff (0x7458 bytes) 000229 1786745016.281409 trunc from 0x040000 to 0x01a347 (0x25cb8 bytes) 000230 1786745016.307574 trunc from 0x01a348 to 0x0189fb (0x194c bytes) 000231 1786745016.341703 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x189fc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x189fc): Mirror 2: 0xd50b68f0 (stale) CRC-32 checksum value for chunk [0, 0x189fc): Mirror 1: 0xb82e7a14 Mirror 3: 0xb82e7a14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x189fc: skipped 000233 1786745017.751677 read 0x0016cb thru 0x010551 (0xee87 bytes) 000235 1786745017.790004 MIRROR_RESYNC 3 mirrors 000236 1786745018.522985 write 0x01861d thru 0x023fa9 (0xb98d bytes) 000239 1786745018.601072 mapwrite 0x039c58 thru 0x03ffff (0x63a8 bytes) 000240 1786745018.702183 read 0x02084e thru 0x02434f (0x3b02 bytes) 000241 1786745018.706597 mapread 0x00ac40 thru 0x012f42 (0x8303 bytes) 000245 1786745018.717797 read 0x0030ee thru 0x00548c (0x239f bytes) 000246 1786745018.722012 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7e1e2a16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7e1e2a16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000247 1786745018.989570 mapwrite 0x00b54b thru 0x01387b (0x8331 bytes) 000250 1786745019.054475 mapread 0x01319c thru 0x01a77a (0x75df bytes) 000252 1786745019.067141 mapread 0x031eab thru 0x03e1e4 (0xc33a bytes) 000253 1786745019.082598 write 0x024c5c thru 0x02acef (0x6094 bytes) 000256 1786745019.088899 mapread 0x01773b thru 0x021c90 (0xa556 bytes) 000257 1786745019.100605 mapwrite 0x027d76 thru 0x02a332 (0x25bd bytes) 000261 1786745019.143154 mapwrite 0x02c3e8 thru 0x02ce9c (0xab5 bytes) 000263 1786745019.189195 mapwrite 0x00749d thru 0x00f195 (0x7cf9 bytes) 000264 1786745019.230274 mapwrite 0x021e4c thru 0x025dde (0x3f93 bytes) 000265 1786745019.278061 mapread 0x0212cc thru 0x02396b (0x26a0 bytes) 000268 1786745019.282355 mapwrite 0x036b78 thru 0x0380f0 (0x1579 bytes) 000271 1786745019.325900 mapwrite 0x014b69 thru 0x0215bc (0xca54 bytes) 000273 1786745019.368615 write 0x03a71d thru 0x03b9df (0x12c3 bytes) 000276 1786745019.375570 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) 000277 1786745019.514566 mapwrite 0x03dcd6 thru 0x03e733 (0xa5e bytes) 000279 1786745019.569954 mapwrite 0x0167a7 thru 0x019c1c (0x3476 bytes) 000280 1786745019.632994 mapwrite 0x0306a3 thru 0x0329bf (0x231d bytes) 000284 1786745019.710979 trunc from 0x040000 to 0x0127e7 (0x2d818 bytes) 000285 1786745019.729996 mapread 0x00eb0b thru 0x0127e7 (0x3cdd bytes) 000286 1786745019.749354 write 0x02f2bb thru 0x039427 (0xa16d bytes) 000287 1786745019.752245 mapread 0x03218f thru 0x039427 (0x7299 bytes) 000288 1786745019.753535 write 0x008fc7 thru 0x01177b (0x87b5 bytes) 000289 1786745019.754749 write 0x01a434 thru 0x0203bb (0x5f88 bytes) 000291 1786745019.770962 trunc from 0x039428 to 0x035212 (0x4215 bytes) 000293 1786745019.787407 mapread 0x01ee79 thru 0x02991d (0xaaa5 bytes) 000294 1786745019.801025 trunc from 0x035213 to 0x031a7a (0x3798 bytes) 000297 1786745019.822980 read 0x0163ae thru 0x02147e (0xb0d1 bytes) 000301 1786745019.836555 mapread 0x0259d8 thru 0x031a7a (0xc0a3 bytes) 000303 1786745019.854535 write 0x01a993 thru 0x022132 (0x77a0 bytes) 000306 1786745019.862586 mapwrite 0x0167f3 thru 0x017ed4 (0x16e2 bytes) 000308 1786745019.924575 mapwrite 0x027072 thru 0x02fd09 (0x8c98 bytes) 000309 1786745020.021357 trunc from 0x031a7b to 0x03cf90 (0xb516 bytes) 000310 1786745020.045955 mapwrite 0x00ab92 thru 0x010a6e (0x5edd bytes) 000312 1786745020.106574 mapread 0x00c925 thru 0x011a9d (0x5179 bytes) 000313 1786745020.109094 write 0x038dbb thru 0x03ffff (0x7245 bytes) 000314 1786745020.119084 trunc from 0x040000 to 0x01a115 (0x25eea bytes) 000316 1786745020.149725 read 0x007caa thru 0x008b57 (0xeae bytes) 000317 1786745020.151298 trunc from 0x01a116 to 0x03ed75 (0x24c60 bytes) 000319 1786745020.185116 mapwrite 0x01284c thru 0x01d37e (0xab33 bytes) 000320 1786745020.331279 trunc from 0x03ed76 to 0x019ad0 (0x252a5 bytes) 000321 1786745020.376087 read 0x0151a8 thru 0x019ad0 (0x4929 bytes) 000322 1786745020.377945 mapwrite 0x034e48 thru 0x03ffff (0xb1b8 bytes) 000325 1786745020.580293 trunc from 0x040000 to 0x0098d1 (0x3672e bytes) 000326 1786745020.633734 write 0x01733f thru 0x020b43 (0x9805 bytes) 000329 1786745020.652789 MIRROR_EXTEND to 2 mirrors 000331 1786745021.597556 write 0x00b4b5 thru 0x01093a (0x5486 bytes) 000332 1786745021.680233 mapwrite 0x02cc10 thru 0x03b65e (0xea4f bytes) 000335 1786745021.782791 trunc from 0x03b65f to 0x032e41 (0x881d bytes) 000338 1786745021.817442 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32e42): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e42): Mirror 2: 0x71955f84 (stale) CRC-32 checksum value for chunk [0, 0x32e42): Mirror 1: 0x17a18ab9 Mirror 3: 0x17a18ab9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e42: skipped 000340 1786745022.754391 trunc from 0x032e42 to 0x02cba0 (0x62a1 bytes) 000341 1786745022.862282 read 0x00be9a thru 0x019a2c (0xdb93 bytes) 000342 1786745022.881161 trunc from 0x02cba1 to 0x0208fc (0xc2a4 bytes) 000343 1786745022.935169 mapwrite 0x013fb4 thru 0x019901 (0x594e bytes) 000344 1786745023.030728 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x208fd): Mirror 3: 0x5678c74d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x208fd): Mirror 3: 0x5678c74d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x208fd: skipped 000347 1786745023.302008 read 0x020770 thru 0x0208fc (0x18d bytes) 000348 1786745023.317057 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) 000349 1786745023.482998 write 0x03eaa5 thru 0x03ffff (0x155b bytes) 000350 1786745023.501994 MIRROR_EXTEND to 2 mirrors 000353 1786745024.048216 mapread 0x00ea57 thru 0x018e75 (0xa41f bytes) 000354 1786745024.074503 mapread 0x0250d1 thru 0x026e67 (0x1d97 bytes) 000355 1786745024.091468 trunc from 0x040000 to 0x003f63 (0x3c09c bytes) 000356 1786745024.172750 trunc from 0x003f64 to 0x03b719 (0x377b6 bytes) 000357 1786745024.201581 mapwrite 0x018d56 thru 0x01b2ae (0x2559 bytes) 000359 1786745024.271445 read 0x004c0a thru 0x0147e0 (0xfbd7 bytes) 000361 1786745024.288219 MIRROR_RESYNC 2 mirrors 000363 1786745025.310741 mapwrite 0x015744 thru 0x0201c1 (0xaa7e bytes) 000364 1786745025.569618 write 0x0167be thru 0x01bc06 (0x5449 bytes) 000367 1786745025.572798 write 0x0280c3 thru 0x028d74 (0xcb2 bytes) 000368 1786745025.578215 mapwrite 0x01eb8e thru 0x02945e (0xa8d1 bytes) 000370 1786745025.630434 mapread 0x02fffd thru 0x0356f2 (0x56f6 bytes) 000373 1786745025.642174 read 0x007083 thru 0x011cd2 (0xac50 bytes) 000375 1786745025.653787 mapread 0x00e214 thru 0x018320 (0xa10d bytes) 000376 1786745025.655281 read 0x029a06 thru 0x02ab17 (0x1112 bytes) 000377 1786745025.656579 read 0x00c0a0 thru 0x01b497 (0xf3f8 bytes) 000382 1786745025.662182 read 0x015bee thru 0x0246c8 (0xeadb bytes) 000383 1786745025.665059 read 0x02b710 thru 0x030941 (0x5232 bytes) 000385 1786745025.669918 mapwrite 0x022091 thru 0x026650 (0x45c0 bytes) 000386 1786745025.731540 trunc from 0x03b71a to 0x02fda3 (0xb976 bytes) 000387 1786745025.785794 write 0x037d27 thru 0x039478 (0x1752 bytes) 000388 1786745025.789134 trunc from 0x039479 to 0x016bd0 (0x228a8 bytes) 000389 1786745025.833605 mapread 0x00027c thru 0x009932 (0x96b7 bytes) 000390 1786745025.869276 read 0x00d0f4 thru 0x016bd0 (0x9add bytes) 000393 1786745025.875181 trunc from 0x016bd1 to 0x03bdd1 (0x25201 bytes) 000395 1786745025.931212 write 0x023931 thru 0x031170 (0xd840 bytes) 000397 1786745025.943066 write 0x0151e9 thru 0x01523b (0x53 bytes) 000399 1786745025.951163 MIRROR_RESYNC 2 mirrors 000400 1786745026.833680 write 0x01b8c0 thru 0x02a052 (0xe793 bytes) 000402 1786745026.933511 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bdd2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bdd2): Mirror 2: 0x125951d4 (stale) CRC-32 checksum value for chunk [0, 0x3bdd2): Mirror 1: 0x3c519431 Mirror 3: 0x3c519431 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bdd2: skipped 000407 1786745027.756509 read 0x000c51 thru 0x001977 (0xd27 bytes) 000408 1786745027.763423 mapread 0x015271 thru 0x019b0e (0x489e bytes) 000410 1786745027.789235 mapread 0x025f8c thru 0x02f595 (0x960a bytes) 000411 1786745027.809766 mapwrite 0x004bb6 thru 0x0132fe (0xe749 bytes) 000412 1786745027.981529 mapread 0x01fbe2 thru 0x02982d (0x9c4c bytes) 000414 1786745028.009575 read 0x02cd85 thru 0x031b9e (0x4e1a bytes) 000417 1786745028.020136 trunc from 0x03bdd2 to 0x0261ba (0x15c17 bytes) 000418 1786745028.045228 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x261bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261bb): Mirror 2: 0xe48aa1ad (stale) Mirror 3: 0xcb3a1e75 (stale) CRC-32 checksum value for chunk [0, 0x261bb): Mirror 1: 0x7efe84ec Mirror 4: 0x7efe84ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261bb: skipped 000421 1786745029.051723 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x261bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261bb): Mirror 2: 0xe48aa1ad (stale) Mirror 3: 0xcb3a1e75 (stale) CRC-32 checksum value for chunk [0, 0x261bb): Mirror 1: 0x7efe84ec Mirror 4: 0x7efe84ec Mirror 5: 0x7efe84ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261bb: skipped 000423 1786745030.069050 trunc from 0x0261bb to 0x03cf57 (0x16d9d bytes) 000424 1786745030.223876 mapread 0x01cc46 thru 0x02bad2 (0xee8d bytes) 000427 1786745030.313119 mapread 0x01b595 thru 0x029900 (0xe36c bytes) 000428 1786745030.319089 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cf58): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf58): Mirror 2: 0x25c5e728 (stale) Mirror 3: 0x7de53747 (stale) Mirror 4: 0x6c73acc5 (stale) Mirror 5: 0x6c73acc5 (stale) CRC-32 checksum value for chunk [0, 0x3cf58): Mirror 1: 0x6c73acc5 Mirror 6: 0x6c73acc5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf58: skipped 000429 1786745031.302449 mapwrite 0x03388d thru 0x03dee9 (0xa65d bytes) 000430 1786745031.553515 read 0x01b9b5 thru 0x02a773 (0xedbf bytes) 000431 1786745031.568080 mapread 0x00043a thru 0x00a640 (0xa207 bytes) 000435 1786745031.576703 write 0x022799 thru 0x02b82b (0x9093 bytes) 000436 1786745031.582001 mapread 0x033130 thru 0x033920 (0x7f1 bytes) 000437 1786745031.583389 trunc from 0x03deea to 0x0012cc (0x3cc1d bytes) 000438 1786745031.610954 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12cd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12cd): Mirror 2: 0xd71dc979 (stale) Mirror 3: 0xd71dc979 (stale) Mirror 4: 0xd71dc979 (stale) Mirror 5: 0xd71dc979 (stale) Mirror 6: 0xd71dc979 (stale) CRC-32 checksum value for chunk [0, 0x12cd): Mirror 1: 0xd71dc979 Mirror 7: 0xd71dc979 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12cd: skipped 000441 1786745032.417032 write 0x03511c thru 0x03f734 (0xa619 bytes) 000442 1786745032.507912 read 0x0076b4 thru 0x00d0f9 (0x5a46 bytes) 000443 1786745032.517315 read 0x031f4d thru 0x033ac5 (0x1b79 bytes) 000448 1786745032.533554 MIRROR_RESYNC 7 mirrors 000449 1786745033.916450 trunc from 0x03f735 to 0x00ae89 (0x348ab bytes) 000450 1786745033.984751 trunc from 0x00ae8a to 0x00ddc4 (0x2f3b bytes) 000451 1786745034.006920 mapwrite 0x036bb3 thru 0x03edcc (0x821a bytes) 000453 1786745034.119721 write 0x023a77 thru 0x024626 (0xbb0 bytes) 000455 1786745034.136850 read 0x034dcf thru 0x03dc00 (0x8e32 bytes) 000457 1786745034.138781 write 0x002afd thru 0x007ee2 (0x53e6 bytes) 000458 1786745034.150752 MIRROR_RESYNC 7 mirrors 000459 1786745035.325537 MIRROR_RESYNC 7 mirrors 000460 1786745035.823520 trunc from 0x03edcd to 0x030bb4 (0xe218 bytes) 000461 1786745035.969203 mapwrite 0x0392fa thru 0x03c4e2 (0x31e9 bytes) 000463 1786745036.064736 trunc from 0x03c4e3 to 0x017913 (0x24bcf bytes) 000464 1786745036.083257 mapread 0x00ae7f thru 0x00e553 (0x36d5 bytes) 000465 1786745036.094724 mapread 0x006017 thru 0x007668 (0x1652 bytes) 000466 1786745036.101826 MIRROR_RESYNC 7 mirrors 000467 1786745037.441146 read 0x00c3b1 thru 0x017913 (0xb563 bytes) 000469 1786745037.473508 MIRROR_RESYNC 7 mirrors 000471 1786745037.960203 trunc from 0x017914 to 0x021f08 (0xa5f5 bytes) 000472 1786745038.117047 trunc from 0x021f09 to 0x03b571 (0x19669 bytes) 000473 1786745038.139785 write 0x02bd42 thru 0x038c1f (0xcede bytes) 000475 1786745038.152458 mapread 0x021fcd thru 0x024f7d (0x2fb1 bytes) 000481 1786745038.180140 mapwrite 0x009f9c thru 0x014855 (0xa8ba bytes) 000482 1786745038.254509 trunc from 0x03b572 to 0x014bb0 (0x269c1 bytes) 000485 1786745038.274090 trunc from 0x014bb1 to 0x017190 (0x25e0 bytes) 000488 1786745038.290392 read 0x00ce7f thru 0x017190 (0xa312 bytes) 000490 1786745038.298452 mapread 0x0132fa thru 0x016a62 (0x3769 bytes) 000491 1786745038.299663 read 0x005e6b thru 0x013023 (0xd1b9 bytes) 000492 1786745038.301027 write 0x00b344 thru 0x00c59a (0x1257 bytes) 000494 1786745038.303499 read 0x0066c6 thru 0x009774 (0x30af bytes) 000495 1786745038.308217 MIRROR_RESYNC 7 mirrors 000497 1786745039.319225 mapread 0x011f9b thru 0x017190 (0x51f6 bytes) 000499 1786745039.341158 mapwrite 0x00db4e thru 0x018de0 (0xb293 bytes) 000500 1786745039.490753 mapread 0x0063ea thru 0x00739e (0xfb5 bytes) 000502 1786745039.493514 mapwrite 0x03199b thru 0x03339c (0x1a02 bytes) 000503 1786745039.646511 trunc from 0x03339d to 0x037227 (0x3e8b bytes) 000504 1786745039.666688 mapwrite 0x004907 thru 0x0122f5 (0xd9ef bytes) 000505 1786745039.739905 read 0x025e20 thru 0x03162d (0xb80e bytes) 000509 1786745039.749508 trunc from 0x037228 to 0x028d27 (0xe500 bytes) 000510 1786745039.765381 mapread 0x024416 thru 0x024e67 (0xa52 bytes) 000512 1786745039.777978 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x28d28): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d28): Mirror 2: 0x604c494e (stale) Mirror 3: 0x604c494e (stale) Mirror 4: 0x604c494e (stale) Mirror 5: 0x604c494e (stale) Mirror 6: 0x604c494e (stale) Mirror 7: 0x604c494e (stale) CRC-32 checksum value for chunk [0, 0x28d28): Mirror 1: 0x19f52d84 Mirror 8: 0x19f52d84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d28: skipped 000513 1786745040.725734 MIRROR_RESYNC 8 mirrors 000515 1786745042.059819 write 0x008efb thru 0x0119c5 (0x8acb bytes) 000516 1786745042.128991 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x28d28): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28d28): Mirror 2: 0x19f52d84 (stale) Mirror 3: 0x19f52d84 (stale) Mirror 4: 0x19f52d84 (stale) Mirror 5: 0x19f52d84 (stale) Mirror 6: 0x19f52d84 (stale) Mirror 7: 0x19f52d84 (stale) Mirror 8: 0x19f52d84 (stale) CRC-32 checksum value for chunk [0, 0x28d28): Mirror 1: 0x909ba722 Mirror 9: 0x909ba722 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28d28: skipped 000517 1786745043.122903 mapread 0x01507b thru 0x022407 (0xd38d bytes) 000519 1786745043.152958 trunc from 0x028d28 to 0x031526 (0x87ff bytes) 000520 1786745043.263784 write 0x029e43 thru 0x02cf8b (0x3149 bytes) 000523 1786745043.295377 mapread 0x005cac thru 0x00c20a (0x655f bytes) 000524 1786745043.315189 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31527): Mirror 2: 0xa695cec0 (stale) Mirror 3: 0xa695cec0 (stale) Mirror 4: 0xa695cec0 (stale) Mirror 6: 0xa695cec0 (stale) Mirror 7: 0xa695cec0 (stale) Mirror 8: 0xa695cec0 (stale) Mirror 9: 0xd43491f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31527): Mirror 2: 0xa695cec0 (stale) Mirror 3: 0xa695cec0 (stale) Mirror 4: 0xa695cec0 (stale) Mirror 6: 0xa695cec0 (stale) Mirror 7: 0xa695cec0 (stale) Mirror 8: 0xa695cec0 (stale) Mirror 9: 0xd43491f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31527: skipped 000525 1786745044.046405 trunc from 0x031527 to 0x027adf (0x9a47 bytes) 000527 1786745044.065352 trunc from 0x027ae0 to 0x02f24d (0x776e bytes) 000528 1786745044.079042 mapread 0x020afa thru 0x021b6e (0x1075 bytes) 000529 1786745044.088136 read 0x013872 thru 0x01b67c (0x7e0b bytes) 000530 1786745044.099812 MIRROR_RESYNC 8 mirrors 000531 1786745045.564061 write 0x03a68a thru 0x03ffff (0x5976 bytes) 000532 1786745045.676856 write 0x002f4b thru 0x00e0f6 (0xb1ac bytes) 000534 1786745045.690676 trunc from 0x040000 to 0x0193fb (0x26c04 bytes) 000537 1786745045.731635 write 0x03b993 thru 0x03ffff (0x466d bytes) 000540 1786745045.739011 trunc from 0x040000 to 0x013a41 (0x2c5be bytes) 000541 1786745045.798438 mapread 0x00867f thru 0x013a41 (0xb3c3 bytes) 000542 1786745045.851091 write 0x01f610 thru 0x023d82 (0x4773 bytes) 000545 1786745045.856716 read 0x00850b thru 0x015021 (0xcb17 bytes) 000546 1786745045.866026 mapread 0x00f855 thru 0x01b4d8 (0xbc84 bytes) 000549 1786745045.876907 write 0x03c4ca thru 0x03ffff (0x3b36 bytes) 000550 1786745045.878150 read 0x03a1a2 thru 0x03cdad (0x2c0c bytes) 000554 1786745045.887669 mapread 0x01f492 thru 0x027869 (0x83d8 bytes) 000555 1786745045.894140 mapread 0x03022b thru 0x03f39b (0xf171 bytes) 000556 1786745045.895456 read 0x009f9b thru 0x00a596 (0x5fc bytes) 000558 1786745045.901166 read 0x013ebb thru 0x021958 (0xda9e bytes) 000559 1786745045.902347 trunc from 0x040000 to 0x00992a (0x366d5 bytes) 000560 1786745045.921093 write 0x02b649 thru 0x036215 (0xabcd bytes) 000561 1786745045.924805 mapread 0x01b05c thru 0x01eaf0 (0x3a95 bytes) 000562 1786745045.932798 mapread 0x00d005 thru 0x00fc55 (0x2c51 bytes) 000564 1786745045.938183 trunc from 0x036216 to 0x0267f8 (0xfa1d bytes) 000565 1786745045.959081 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x267f9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x267f9): Mirror 2: 0xb763c4b0 (stale) Mirror 3: 0xb763c4b0 (stale) Mirror 4: 0xb763c4b0 (stale) Mirror 6: 0xb763c4b0 (stale) Mirror 7: 0xb763c4b0 (stale) Mirror 8: 0xb763c4b0 (stale) Mirror 9: 0xb763c4b0 (stale) CRC-32 checksum value for chunk [0, 0x267f9): Mirror 1: 0x422fd86f Mirror 10: 0x422fd86f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x267f9: skipped 000566 1786745046.991613 read 0x0218fe thru 0x0267f8 (0x4efb bytes) 000567 1786745047.004916 write 0x00d399 thru 0x01b0bd (0xdd25 bytes) 000568 1786745047.093050 read 0x005bcf thru 0x00a304 (0x4736 bytes) 000569 1786745047.107177 trunc from 0x0267f9 to 0x02ec0e (0x8416 bytes) 000572 1786745047.139316 write 0x00d1df thru 0x0104c9 (0x32eb bytes) 000573 1786745047.146068 mapwrite 0x023fd7 thru 0x0264bf (0x24e9 bytes) 000575 1786745047.228652 write 0x03a559 thru 0x03ffff (0x5aa7 bytes) 000576 1786745047.239245 MIRROR_RESYNC 9 mirrors 000577 1786745048.570911 MIRROR_RESYNC 9 mirrors 000578 1786745049.116845 mapread 0x035160 thru 0x03ef49 (0x9dea bytes) 000579 1786745049.137321 read 0x018e93 thru 0x01d1f2 (0x4360 bytes) 000580 1786745049.147758 write 0x02fd93 thru 0x035405 (0x5673 bytes) 000583 1786745049.316989 trunc from 0x040000 to 0x009111 (0x36eee bytes) 000585 1786745049.375975 trunc from 0x009112 to 0x018c9e (0xfb8d bytes) 000586 1786745049.442815 mapwrite 0x00fbe8 thru 0x016a2f (0x6e48 bytes) 000587 1786745049.576463 read 0x016a9b thru 0x018c61 (0x21c7 bytes) 000588 1786745049.612312 write 0x032a4b thru 0x03ffff (0xd5b5 bytes) 000589 1786745049.618942 write 0x00e4b7 thru 0x012c5c (0x47a6 bytes) 000590 1786745049.621044 trunc from 0x040000 to 0x037f61 (0x809e bytes) 000593 1786745049.657458 trunc from 0x037f62 to 0x03d740 (0x57df bytes) 000595 1786745049.704594 write 0x0242d9 thru 0x02bd1f (0x7a47 bytes) 000597 1786745049.717014 write 0x0361cb thru 0x03ffff (0x9e35 bytes) 000598 1786745049.729305 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe0499683 (stale) Mirror 3: 0xe0499683 (stale) Mirror 4: 0xe0499683 (stale) Mirror 6: 0xe0499683 (stale) Mirror 7: 0xe0499683 (stale) Mirror 8: 0xe0499683 (stale) Mirror 9: 0xe0499683 (stale) Mirror 10: 0xe0499683 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3cec2fab Mirror 11: 0x3cec2fab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1786745051.358311 mapwrite 0x01603c thru 0x01af85 (0x4f4a bytes) 000602 1786745051.644600 trunc from 0x040000 to 0x02baad (0x14552 bytes) 000603 1786745051.711719 read 0x01e6ee thru 0x024e96 (0x67a9 bytes) 000605 1786745051.719887 read 0x00d6bf thru 0x00d842 (0x184 bytes) 000606 1786745051.745641 mapread 0x029cbe thru 0x02baad (0x1df0 bytes) 000608 1786745051.747909 trunc from 0x02baae to 0x02ad14 (0xd99 bytes) 000610 1786745051.774206 trunc from 0x02ad15 to 0x00f727 (0x1b5ed bytes) 000613 1786745051.804269 write 0x00130e thru 0x001cae (0x9a1 bytes) 000614 1786745051.806820 mapread 0x00414c thru 0x00b73f (0x75f4 bytes) 000617 1786745051.820789 mapwrite 0x03d6c0 thru 0x03ffff (0x2940 bytes) 000619 1786745051.920627 mapread 0x00785a thru 0x01069e (0x8e45 bytes) 000620 1786745051.932450 MIRROR_RESYNC 10 mirrors 000621 1786745053.167444 MIRROR_EXTEND to 11 mirrors 000622 1786745054.220815 mapwrite 0x026963 thru 0x02f7c5 (0x8e63 bytes) 000624 1786745054.437981 mapwrite 0x014f6c thru 0x01f4d2 (0xa567 bytes) 000625 1786745054.512349 trunc from 0x040000 to 0x002e92 (0x3d16d bytes) 000629 1786745054.543273 trunc from 0x002e93 to 0x00e57e (0xb6ec bytes) 000631 1786745054.565370 write 0x02f4ab thru 0x0366fe (0x7254 bytes) 000633 1786745054.569284 read 0x00815b thru 0x0117f9 (0x969f bytes) 000634 1786745054.578768 mapread 0x002689 thru 0x004a9f (0x2417 bytes) 000635 1786745054.591862 mapread 0x02c39d thru 0x0366fe (0xa362 bytes) 000636 1786745054.602336 mapwrite 0x020c9e thru 0x029104 (0x8467 bytes) 000637 1786745054.683301 mapread 0x018bdf thru 0x022bcc (0x9fee bytes) 000639 1786745054.731393 trunc from 0x0366ff to 0x0110b8 (0x25646 bytes) 000642 1786745054.787393 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x110b9): Mirror 2: 0xd3aab527 (stale) Mirror 3: 0xd3aab527 (stale) Mirror 4: 0xd3aab527 (stale) Mirror 6: 0xd3aab527 (stale) Mirror 8: 0xd3aab527 (stale) Mirror 9: 0xd3aab527 (stale) Mirror 10: 0xd3aab527 (stale) Mirror 11: 0xd3aab527 (stale) Mirror 12: 0xd3aab527 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x110b9): Mirror 2: 0xd3aab527 (stale) Mirror 3: 0xd3aab527 (stale) Mirror 4: 0xd3aab527 (stale) Mirror 6: 0xd3aab527 (stale) Mirror 8: 0xd3aab527 (stale) Mirror 9: 0xd3aab527 (stale) Mirror 10: 0xd3aab527 (stale) Mirror 11: 0xd3aab527 (stale) Mirror 12: 0xd3aab527 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x110b9: skipped 000643 1786745055.409767 mapread 0x00e906 thru 0x0110b8 (0x27b3 bytes) 000646 1786745055.447648 trunc from 0x0110b9 to 0x025d17 (0x14c5f bytes) 000647 1786745055.478148 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25d18): Mirror 2: 0x2ded6521 (stale) Mirror 3: 0x2ded6521 (stale) Mirror 4: 0x2ded6521 (stale) Mirror 6: 0x2ded6521 (stale) Mirror 8: 0x2ded6521 (stale) Mirror 9: 0x2ded6521 (stale) Mirror 10: 0x2ded6521 (stale) Mirror 12: 0x2ded6521 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25d18): Mirror 2: 0x2ded6521 (stale) Mirror 3: 0x2ded6521 (stale) Mirror 4: 0x2ded6521 (stale) Mirror 6: 0x2ded6521 (stale) Mirror 8: 0x2ded6521 (stale) Mirror 9: 0x2ded6521 (stale) Mirror 10: 0x2ded6521 (stale) Mirror 12: 0x2ded6521 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25d18: skipped 000649 1786745056.123597 read 0x0111e0 thru 0x01f7e1 (0xe602 bytes) 000651 1786745056.142245 trunc from 0x025d18 to 0x02d7b8 (0x7aa1 bytes) 000652 1786745056.172017 mapread 0x005cd0 thru 0x0074ea (0x181b bytes) 000653 1786745056.216560 mapwrite 0x03990e thru 0x03ffff (0x66f2 bytes) 000654 1786745056.360234 write 0x021327 thru 0x029d95 (0x8a6f bytes) 000655 1786745056.395928 write 0x039acc thru 0x03ffff (0x6534 bytes) 000656 1786745056.401419 mapread 0x02d507 thru 0x035e0e (0x8908 bytes) 000660 1786745056.441922 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbaafa682 (stale) Mirror 3: 0xbaafa682 (stale) Mirror 4: 0xbaafa682 (stale) Mirror 6: 0xbaafa682 (stale) Mirror 8: 0xbaafa682 (stale) Mirror 9: 0xbaafa682 (stale) Mirror 10: 0xbaafa682 (stale) Mirror 12: 0xbaafa682 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9ea508b Mirror 13: 0xd9ea508b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000663 1786745057.515000 mapread 0x022a78 thru 0x03054e (0xdad7 bytes) 000666 1786745057.546588 mapread 0x0303bc thru 0x03ab67 (0xa7ac bytes) 000668 1786745057.556123 mapwrite 0x00dba3 thru 0x01d8c7 (0xfd25 bytes) 000669 1786745057.718228 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbaafa682 (stale) Mirror 3: 0xbaafa682 (stale) Mirror 4: 0xbaafa682 (stale) Mirror 6: 0xbaafa682 (stale) Mirror 8: 0xbaafa682 (stale) Mirror 9: 0xbaafa682 (stale) Mirror 10: 0xbaafa682 (stale) Mirror 12: 0xbaafa682 (stale) Mirror 13: 0xd9ea508b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f1089d0 Mirror 14: 0x6f1089d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000670 1786745059.002951 read 0x007bfb thru 0x009d22 (0x2128 bytes) 000674 1786745059.054973 trunc from 0x040000 to 0x002a89 (0x3d576 bytes) 000675 1786745059.201259 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2a8a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a8a): Mirror 2: 0x86578d33 (stale) Mirror 3: 0x86578d33 (stale) Mirror 4: 0x86578d33 (stale) Mirror 6: 0x86578d33 (stale) Mirror 8: 0x86578d33 (stale) Mirror 9: 0x86578d33 (stale) Mirror 10: 0x86578d33 (stale) Mirror 12: 0x86578d33 (stale) Mirror 13: 0x86578d33 (stale) Mirror 14: 0x86578d33 (stale) CRC-32 checksum value for chunk [0, 0x2a8a): Mirror 1: 0x86578d33 Mirror 15: 0x86578d33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a8a: skipped 000676 1786745060.541770 mapwrite 0x017b25 thru 0x024bbe (0xd09a bytes) 000679 1786745060.730056 read 0x00c0ff thru 0x0176a5 (0xb5a7 bytes) 000681 1786745060.747939 MIRROR_RESYNC 12 mirrors 000683 1786745062.350821 write 0x02b269 thru 0x02daae (0x2846 bytes) 000684 1786745062.421541 mapwrite 0x026590 thru 0x033fea (0xda5b bytes) 000685 1786745062.517169 mapwrite 0x00f134 thru 0x0186c6 (0x9593 bytes) 000687 1786745062.656941 MIRROR_RESYNC 12 mirrors 000690 1786745064.765918 mapread 0x019265 thru 0x01e02d (0x4dc9 bytes) 000691 1786745064.801032 MIRROR_EXTEND to 13 mirrors 000695 1786745065.720494 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) 000697 1786745066.145291 read 0x018ff9 thru 0x01bb21 (0x2b29 bytes) 000698 1786745066.151150 read 0x0083d3 thru 0x009fda (0x1c08 bytes) 000700 1786745066.163520 read 0x02e728 thru 0x033fea (0x58c3 bytes) 000704 1786745066.166924 mapread 0x024f3c thru 0x0297e6 (0x48ab bytes) 000706 1786745066.173541 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) 000707 1786745066.640321 write 0x030aee thru 0x03b8a8 (0xadbb bytes) 000708 1786745066.830875 mapwrite 0x02f8a2 thru 0x03e376 (0xead5 bytes) 000711 1786745066.990949 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e377): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e377): Mirror 2: 0x4198b9b0 (stale) Mirror 3: 0x4198b9b0 (stale) Mirror 4: 0x4198b9b0 (stale) Mirror 6: 0x4198b9b0 (stale) Mirror 8: 0x4198b9b0 (stale) Mirror 9: 0x4198b9b0 (stale) Mirror 10: 0x4198b9b0 (stale) Mirror 13: 0x4198b9b0 (stale) Mirror 15: 0x4198b9b0 (stale) Mirror 16: 0x4198b9b0 (stale) CRC-32 checksum value for chunk [0, 0x3e377): Mirror 1: 0xc4c6e7c8 Mirror 17: 0xc4c6e7c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e377: skipped 000712 1786745068.282774 mapwrite 0x00716a thru 0x0107a0 (0x9637 bytes) 000714 1786745068.640044 trunc from 0x03e377 to 0x01300e (0x2b368 bytes) 000716 1786745068.663539 write 0x00efef thru 0x013447 (0x4459 bytes) 000718 1786745068.671568 read 0x005d43 thru 0x013123 (0xd3e1 bytes) 000721 1786745068.673907 mapread 0x0042e4 thru 0x00bf7f (0x7c9c bytes) 000723 1786745068.676418 trunc from 0x013448 to 0x039075 (0x25c2e bytes) 000724 1786745068.694157 trunc from 0x039076 to 0x0068a5 (0x327d0 bytes) 000727 1786745068.716617 mapwrite 0x030a0c thru 0x032c26 (0x221b bytes) 000729 1786745068.787911 MIRROR_RESYNC 12 mirrors 000730 1786745070.807383 read 0x002199 thru 0x00d257 (0xb0bf bytes) 000731 1786745070.838682 write 0x03a816 thru 0x03ffff (0x57ea bytes) 000732 1786745070.917274 mapread 0x020826 thru 0x02322f (0x2a0a bytes) 000733 1786745070.940796 write 0x03ea0e thru 0x03ffff (0x15f2 bytes) 000734 1786745070.942416 mapread 0x005b44 thru 0x006b2a (0xfe7 bytes) 000737 1786745070.961815 read 0x00bca0 thru 0x0110a3 (0x5404 bytes) 000739 1786745070.972290 mapwrite 0x017cc5 thru 0x0249d7 (0xcd13 bytes) 000740 1786745071.040893 mapread 0x010239 thru 0x014421 (0x41e9 bytes) 000742 1786745071.043925 mapread 0x0257a3 thru 0x02b6ac (0x5f0a bytes) 000743 1786745071.045372 read 0x02f65e thru 0x0397dd (0xa180 bytes) 000745 1786745071.047814 read 0x013428 thru 0x01b813 (0x83ec bytes) 000747 1786745071.049600 write 0x01bafd thru 0x023f0f (0x8413 bytes) 000749 1786745071.052325 read 0x034a2a thru 0x03fd41 (0xb318 bytes) 000751 1786745071.054440 read 0x03160b thru 0x03dab5 (0xc4ab bytes) 000752 1786745071.055386 trunc from 0x040000 to 0x011a11 (0x2e5ee bytes) 000753 1786745071.086553 mapread 0x00c287 thru 0x011a11 (0x578b bytes) 000754 1786745071.088876 mapread 0x00f154 thru 0x011a11 (0x28be bytes) 000755 1786745071.091496 mapwrite 0x005f89 thru 0x006e24 (0xe9c bytes) 000756 1786745071.132807 trunc from 0x011a12 to 0x010c39 (0xdd8 bytes) 000757 1786745071.151323 trunc from 0x010c3a to 0x03dda5 (0x2d16c bytes) 000758 1786745071.177713 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3dda6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dda6): Mirror 2: 0xe6fd29c (stale) Mirror 3: 0xe6fd29c (stale) Mirror 4: 0xe6fd29c (stale) Mirror 6: 0xe6fd29c (stale) Mirror 8: 0xe6fd29c (stale) Mirror 9: 0xe6fd29c (stale) Mirror 10: 0xe6fd29c (stale) Mirror 13: 0xe6fd29c (stale) Mirror 15: 0xe6fd29c (stale) Mirror 16: 0xe6fd29c (stale) Mirror 17: 0xe6fd29c (stale) CRC-32 checksum value for chunk [0, 0x3dda6): Mirror 1: 0x765d4980 Mirror 18: 0x765d4980 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dda6: skipped 000759 1786745072.575935 mapwrite 0x024a77 thru 0x02898d (0x3f17 bytes) 000760 1786745072.834316 read 0x00fc85 thru 0x01d949 (0xdcc5 bytes) 000761 1786745072.849714 trunc from 0x03dda6 to 0x01cdca (0x20fdb bytes) 000762 1786745072.875857 write 0x03291d thru 0x03d5c2 (0xaca6 bytes) 000764 1786745072.881861 trunc from 0x03d5c3 to 0x02c12d (0x11495 bytes) 000765 1786745072.904391 write 0x03f8f2 thru 0x03ffff (0x70e bytes) 000766 1786745072.910178 trunc from 0x040000 to 0x02b7cc (0x14833 bytes) 000767 1786745072.953269 mapread 0x021fa5 thru 0x028768 (0x67c4 bytes) 000768 1786745072.992514 mapwrite 0x02cc1d thru 0x0305dc (0x39c0 bytes) 000770 1786745073.083530 mapwrite 0x02ea90 thru 0x036979 (0x7eea bytes) 000771 1786745073.155643 mapread 0x0064a4 thru 0x014250 (0xddad bytes) 000772 1786745073.173630 mapwrite 0x02cdec thru 0x038582 (0xb797 bytes) 000775 1786745073.287158 trunc from 0x038583 to 0x027582 (0x11000 bytes) 000778 1786745073.319603 MIRROR_RESYNC 13 mirrors 000783 1786745075.313802 mapwrite 0x0111e7 thru 0x01e1b1 (0xcfcb bytes) 000784 1786745075.535895 write 0x014636 thru 0x01ec42 (0xa60d bytes) 000787 1786745075.543975 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27583): Mirror 3: 0x2a266d5c (stale) Mirror 4: 0x2a266d5c (stale) Mirror 6: 0x2a266d5c (stale) Mirror 8: 0x2a266d5c (stale) Mirror 9: 0x2a266d5c (stale) Mirror 10: 0x2a266d5c (stale) Mirror 13: 0x2a266d5c (stale) Mirror 15: 0x2a266d5c (stale) Mirror 16: 0x2a266d5c (stale) Mirror 17: 0x2a266d5c (stale) Mirror 18: 0x2a266d5c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27583): Mirror 3: 0x2a266d5c (stale) Mirror 4: 0x2a266d5c (stale) Mirror 6: 0x2a266d5c (stale) Mirror 8: 0x2a266d5c (stale) Mirror 9: 0x2a266d5c (stale) Mirror 10: 0x2a266d5c (stale) Mirror 13: 0x2a266d5c (stale) Mirror 15: 0x2a266d5c (stale) Mirror 16: 0x2a266d5c (stale) Mirror 17: 0x2a266d5c (stale) Mirror 18: 0x2a266d5c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27583: skipped 000788 1786745076.207909 write 0x020aab thru 0x024979 (0x3ecf bytes) 000789 1786745076.220785 mapread 0x0125fc thru 0x020ee1 (0xe8e6 bytes) 000794 1786745076.248312 write 0x01b81d thru 0x01faf3 (0x42d7 bytes) 000795 1786745076.250039 mapwrite 0x0278af thru 0x0301fc (0x894e bytes) 000797 1786745076.356478 MIRROR_RESYNC 12 mirrors 000798 1786745078.282085 trunc from 0x0301fd to 0x01d541 (0x12cbb bytes) 000799 1786745078.410082 mapread 0x008280 thru 0x00e8ec (0x666d bytes) 000800 1786745078.462110 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d542): Mirror 3: 0xaa6a0 (stale) Mirror 4: 0xaa6a0 (stale) Mirror 6: 0xaa6a0 (stale) Mirror 8: 0xaa6a0 (stale) Mirror 9: 0xaa6a0 (stale) Mirror 13: 0xaa6a0 (stale) Mirror 15: 0xaa6a0 (stale) Mirror 16: 0xaa6a0 (stale) Mirror 17: 0xaa6a0 (stale) Mirror 18: 0xaa6a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d542): Mirror 3: 0xaa6a0 (stale) Mirror 4: 0xaa6a0 (stale) Mirror 6: 0xaa6a0 (stale) Mirror 8: 0xaa6a0 (stale) Mirror 9: 0xaa6a0 (stale) Mirror 13: 0xaa6a0 (stale) Mirror 15: 0xaa6a0 (stale) Mirror 16: 0xaa6a0 (stale) Mirror 17: 0xaa6a0 (stale) Mirror 18: 0xaa6a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d542: skipped 000801 1786745079.046900 mapread 0x0009cc thru 0x0028a6 (0x1edb bytes) 000802 1786745079.060895 mapwrite 0x02a63c thru 0x02e65c (0x4021 bytes) 000803 1786745079.201821 mapwrite 0x0044e5 thru 0x0132bf (0xeddb bytes) 000804 1786745079.358968 mapwrite 0x022dc3 thru 0x02a65a (0x7898 bytes) 000805 1786745079.475598 mapwrite 0x0055d1 thru 0x0125bf (0xcfef bytes) 000806 1786745079.588104 mapread 0x006856 thru 0x00eab8 (0x8263 bytes) 000810 1786745079.593703 mapread 0x026c64 thru 0x02e65c (0x79f9 bytes) 000812 1786745079.595786 trunc from 0x02e65d to 0x008b4a (0x25b12 bytes) 000813 1786745079.622221 write 0x003c91 thru 0x00f3fd (0xb76d bytes) 000814 1786745079.625874 write 0x039018 thru 0x03ffff (0x6fe8 bytes) 000816 1786745079.639153 MIRROR_RESYNC 11 mirrors 000819 1786745081.252496 mapread 0x028819 thru 0x02ca78 (0x4260 bytes) 000822 1786745081.285112 read 0x01a46d thru 0x01c68d (0x2221 bytes) 000823 1786745081.295098 MIRROR_EXTEND to 12 mirrors 000824 1786745082.410389 read 0x02ceb3 thru 0x02d76a (0x8b8 bytes) 000825 1786745082.416653 trunc from 0x040000 to 0x00bb41 (0x344be bytes) 000826 1786745082.510409 trunc from 0x00bb42 to 0x00b9e5 (0x15c bytes) 000827 1786745082.532614 mapread 0x004ac2 thru 0x007fdf (0x351e bytes) 000829 1786745082.550735 trunc from 0x00b9e6 to 0x019554 (0xdb6f bytes) 000834 1786745082.565776 trunc from 0x019555 to 0x00e691 (0xaec3 bytes) 000837 1786745082.588332 trunc from 0x00e692 to 0x023443 (0x14db2 bytes) 000840 1786745082.606943 read 0x01a2ff thru 0x023443 (0x9145 bytes) 000841 1786745082.623814 MIRROR_RESYNC 12 mirrors 000842 1786745084.947450 read 0x00335d thru 0x012b00 (0xf7a4 bytes) 000849 1786745084.964991 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) 000852 1786745085.344216 mapwrite 0x0301f5 thru 0x03af5a (0xad66 bytes) 000853 1786745085.524797 write 0x03d846 thru 0x03e807 (0xfc2 bytes) 000854 1786745085.526958 read 0x036c31 thru 0x038534 (0x1904 bytes) 000855 1786745085.528263 mapwrite 0x036035 thru 0x03ffff (0x9fcb bytes) 000857 1786745085.605283 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6765165a (stale) Mirror 6: 0x6765165a (stale) Mirror 8: 0x6765165a (stale) Mirror 9: 0x6765165a (stale) Mirror 13: 0x6765165a (stale) Mirror 15: 0x6765165a (stale) Mirror 16: 0x6765165a (stale) Mirror 17: 0x6765165a (stale) Mirror 18: 0x6765165a (stale) Mirror 19: 0x6765165a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x195dfd31 Mirror 20: 0x195dfd31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000860 1786745086.598595 read 0x01959a thru 0x01f77c (0x61e3 bytes) 000862 1786745086.612634 mapread 0x023d37 thru 0x028956 (0x4c20 bytes) 000863 1786745086.628557 trunc from 0x040000 to 0x004523 (0x3badc bytes) 000864 1786745086.728671 write 0x03cb8f thru 0x03ffff (0x3471 bytes) 000865 1786745086.733093 mapread 0x02abbf thru 0x032e70 (0x82b2 bytes) 000866 1786745086.753868 MIRROR_RESYNC 12 mirrors 000868 1786745088.765285 read 0x022a16 thru 0x02a101 (0x76ec bytes) 000869 1786745088.774022 read 0x002676 thru 0x00b1c9 (0x8b54 bytes) 000870 1786745088.786092 trunc from 0x040000 to 0x0047d1 (0x3b82e bytes) 000871 1786745088.881612 mapwrite 0x01738e thru 0x026ffd (0xfc70 bytes) 000872 1786745089.053115 write 0x00fa20 thru 0x013a93 (0x4074 bytes) 000873 1786745089.087917 MIRROR_RESYNC 12 mirrors 000874 1786745090.996675 mapread 0x0210f0 thru 0x026ffd (0x5f0e bytes) 000875 1786745091.017635 write 0x01fc85 thru 0x02508f (0x540b bytes) 000876 1786745091.128769 read 0x0192a3 thru 0x01acd3 (0x1a31 bytes) 000878 1786745091.137523 trunc from 0x026ffe to 0x00c431 (0x1abcc bytes) 000879 1786745091.172709 mapwrite 0x03640c thru 0x03ffff (0x9bf4 bytes) 000880 1786745091.284680 write 0x015aad thru 0x02328c (0xd7e0 bytes) 000881 1786745091.306734 mapwrite 0x0329a1 thru 0x03ffff (0xd65f bytes) 000886 1786745091.410612 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x444962cf (stale) Mirror 6: 0x444962cf (stale) Mirror 8: 0x444962cf (stale) Mirror 9: 0x444962cf (stale) Mirror 13: 0x444962cf (stale) Mirror 15: 0x444962cf (stale) Mirror 16: 0x444962cf (stale) Mirror 17: 0x444962cf (stale) Mirror 18: 0x444962cf (stale) Mirror 20: 0x444962cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x444962cf (stale) Mirror 6: 0x444962cf (stale) Mirror 8: 0x444962cf (stale) Mirror 9: 0x444962cf (stale) Mirror 13: 0x444962cf (stale) Mirror 15: 0x444962cf (stale) Mirror 16: 0x444962cf (stale) Mirror 17: 0x444962cf (stale) Mirror 18: 0x444962cf (stale) Mirror 20: 0x444962cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1786745092.307703 mapread 0x00e004 thru 0x019063 (0xb060 bytes) 000889 1786745092.382235 mapwrite 0x02c1e4 thru 0x02f250 (0x306d bytes) 000890 1786745092.474502 mapread 0x035c31 thru 0x03e602 (0x89d2 bytes) 000891 1786745092.495678 trunc from 0x040000 to 0x0233c7 (0x1cc38 bytes) 000892 1786745092.518541 MIRROR_RESYNC 11 mirrors 000894 1786745093.971185 read 0x0117f7 thru 0x011d1a (0x524 bytes) 000895 1786745093.978087 mapwrite 0x02a3bf thru 0x02a668 (0x2aa bytes) 000897 1786745094.163575 read 0x0072d6 thru 0x007dec (0xb17 bytes) 000898 1786745094.193888 mapread 0x0038d4 thru 0x00f2a8 (0xb9d5 bytes) 000900 1786745094.203479 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a669): Mirror 6: 0x67177fab (stale) Mirror 8: 0x67177fab (stale) Mirror 9: 0x67177fab (stale) Mirror 13: 0x67177fab (stale) Mirror 15: 0x67177fab (stale) Mirror 16: 0x67177fab (stale) Mirror 17: 0x67177fab (stale) Mirror 18: 0x67177fab (stale) Mirror 20: 0x67177fab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a669): Mirror 6: 0x67177fab (stale) Mirror 8: 0x67177fab (stale) Mirror 9: 0x67177fab (stale) Mirror 13: 0x67177fab (stale) Mirror 15: 0x67177fab (stale) Mirror 16: 0x67177fab (stale) Mirror 17: 0x67177fab (stale) Mirror 18: 0x67177fab (stale) Mirror 20: 0x67177fab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a669: skipped 000901 1786745094.665817 mapwrite 0x02c0b4 thru 0x032aa0 (0x69ed bytes) 000903 1786745094.785204 mapwrite 0x03f091 thru 0x03ffff (0xf6f bytes) truncating to largest ever: 0x3f033 000906 1786745094.841525 trunc from 0x040000 to 0x03f032 (0xfcd bytes) 000908 1786745094.861501 mapread 0x03e935 thru 0x03f032 (0x6fe bytes) 000910 1786745094.864555 mapread 0x01bb4a thru 0x0258e4 (0x9d9b bytes) 000911 1786745094.881823 MIRROR_RESYNC 10 mirrors 000915 1786745096.336611 MIRROR_EXTEND to 11 mirrors 000916 1786745097.369267 trunc from 0x03f033 to 0x005d24 (0x3930e bytes) 000918 1786745097.468789 trunc from 0x005d25 to 0x0293e9 (0x236c5 bytes) 000920 1786745097.504292 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x293ea): Mirror 6: 0xf77fa8c1 (stale) Mirror 9: 0xf77fa8c1 (stale) Mirror 13: 0xf77fa8c1 (stale) Mirror 15: 0xf77fa8c1 (stale) Mirror 16: 0xf77fa8c1 (stale) Mirror 17: 0xf77fa8c1 (stale) Mirror 18: 0xf77fa8c1 (stale) Mirror 20: 0xf77fa8c1 (stale) Mirror 21: 0xf77fa8c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293ea): Mirror 6: 0xf77fa8c1 (stale) Mirror 9: 0xf77fa8c1 (stale) Mirror 13: 0xf77fa8c1 (stale) Mirror 15: 0xf77fa8c1 (stale) Mirror 16: 0xf77fa8c1 (stale) Mirror 17: 0xf77fa8c1 (stale) Mirror 18: 0xf77fa8c1 (stale) Mirror 20: 0xf77fa8c1 (stale) Mirror 21: 0xf77fa8c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293ea: skipped 000921 1786745098.011602 trunc from 0x0293ea to 0x039a5c (0x10673 bytes) 000922 1786745098.037995 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a5d): Mirror 6: 0x6bfcc0e3 (stale) Mirror 13: 0x6bfcc0e3 (stale) Mirror 15: 0x6bfcc0e3 (stale) Mirror 16: 0x6bfcc0e3 (stale) Mirror 17: 0x6bfcc0e3 (stale) Mirror 18: 0x6bfcc0e3 (stale) Mirror 20: 0x6bfcc0e3 (stale) Mirror 21: 0x6bfcc0e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a5d): Mirror 6: 0x6bfcc0e3 (stale) Mirror 13: 0x6bfcc0e3 (stale) Mirror 15: 0x6bfcc0e3 (stale) Mirror 16: 0x6bfcc0e3 (stale) Mirror 17: 0x6bfcc0e3 (stale) Mirror 18: 0x6bfcc0e3 (stale) Mirror 20: 0x6bfcc0e3 (stale) Mirror 21: 0x6bfcc0e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a5d: skipped 000923 1786745098.466621 read 0x003a57 thru 0x004575 (0xb1f bytes) 000924 1786745098.471682 read 0x01e155 thru 0x02c538 (0xe3e4 bytes) 000925 1786745098.480132 mapwrite 0x0093a5 thru 0x00b32e (0x1f8a bytes) 000926 1786745098.542547 write 0x0290e5 thru 0x02bedb (0x2df7 bytes) 000929 1786745098.550692 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a5d): Mirror 6: 0x6bfcc0e3 (stale) Mirror 13: 0x6bfcc0e3 (stale) Mirror 15: 0x6bfcc0e3 (stale) Mirror 16: 0x6bfcc0e3 (stale) Mirror 18: 0x6bfcc0e3 (stale) Mirror 20: 0x6bfcc0e3 (stale) Mirror 21: 0x6bfcc0e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a5d): Mirror 6: 0x6bfcc0e3 (stale) Mirror 13: 0x6bfcc0e3 (stale) Mirror 15: 0x6bfcc0e3 (stale) Mirror 16: 0x6bfcc0e3 (stale) Mirror 18: 0x6bfcc0e3 (stale) Mirror 20: 0x6bfcc0e3 (stale) Mirror 21: 0x6bfcc0e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a5d: skipped 000930 1786745099.103367 MIRROR_RESYNC 8 mirrors 000931 1786745100.788758 mapwrite 0x008d11 thru 0x016c4f (0xdf3f bytes) 000932 1786745100.989492 read 0x01fd37 thru 0x0296c4 (0x998e bytes) 000933 1786745101.002124 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39a5d): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a5d): Mirror 6: 0x520e6142 (stale) Mirror 13: 0x520e6142 (stale) Mirror 15: 0x520e6142 (stale) Mirror 16: 0x520e6142 (stale) Mirror 18: 0x520e6142 (stale) Mirror 20: 0x520e6142 (stale) Mirror 21: 0x520e6142 (stale) CRC-32 checksum value for chunk [0, 0x39a5d): Mirror 1: 0x38e147c7 Mirror 22: 0x38e147c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a5d: skipped 000935 1786745101.833696 write 0x02234e thru 0x02ae7b (0x8b2e bytes) 000939 1786745101.938473 write 0x0385fa thru 0x03b2ac (0x2cb3 bytes) 000940 1786745101.945294 write 0x002580 thru 0x008bc9 (0x664a bytes) 000941 1786745101.956376 mapread 0x039027 thru 0x03b076 (0x2050 bytes) 000942 1786745101.957492 read 0x0324f4 thru 0x03b2ac (0x8db9 bytes) 000943 1786745101.966473 mapread 0x02d88a thru 0x037d35 (0xa4ac bytes) 000946 1786745101.986546 write 0x02855e thru 0x037912 (0xf3b5 bytes) 000947 1786745101.989329 mapwrite 0x008cc5 thru 0x016e4a (0xe186 bytes) 000948 1786745102.039431 mapread 0x0009f2 thru 0x0076d1 (0x6ce0 bytes) 000949 1786745102.043934 write 0x028377 thru 0x0345e4 (0xc26e bytes) 000950 1786745102.045493 write 0x01ad66 thru 0x022113 (0x73ae bytes) 000951 1786745102.047279 trunc from 0x03b2ad to 0x030571 (0xad3b bytes) 000953 1786745102.065069 mapwrite 0x00ac74 thru 0x015ae1 (0xae6e bytes) 000954 1786745102.121154 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30572): Mirror 6: 0x9786abc7 (stale) Mirror 13: 0x9786abc7 (stale) Mirror 15: 0x9786abc7 (stale) Mirror 16: 0x9786abc7 (stale) Mirror 18: 0x9786abc7 (stale) Mirror 21: 0x9786abc7 (stale) Mirror 22: 0xf6bc268f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30572): Mirror 6: 0x9786abc7 (stale) Mirror 13: 0x9786abc7 (stale) Mirror 15: 0x9786abc7 (stale) Mirror 16: 0x9786abc7 (stale) Mirror 18: 0x9786abc7 (stale) Mirror 21: 0x9786abc7 (stale) Mirror 22: 0xf6bc268f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30572: skipped 000955 1786745102.685009 trunc from 0x030572 to 0x025a1f (0xab52 bytes) 000956 1786745102.702200 trunc from 0x025a20 to 0x00a964 (0x1b0bb bytes) 000958 1786745102.729085 mapread 0x0008af thru 0x00a964 (0xa0b6 bytes) 000959 1786745102.758740 mapread 0x006f89 thru 0x00a964 (0x39dc bytes) 000961 1786745102.763326 read 0x000011 thru 0x000fd8 (0xfc8 bytes) 000962 1786745102.765464 mapwrite 0x005def thru 0x006a8d (0xc9f bytes) 000965 1786745102.838740 write 0x0255ca thru 0x030886 (0xb2bd bytes) 000966 1786745102.849451 read 0x015d5c thru 0x016357 (0x5fc bytes) 000967 1786745102.882232 MIRROR_RESYNC 8 mirrors 000969 1786745104.328408 mapread 0x017a81 thru 0x0191a0 (0x1720 bytes) 000971 1786745104.370037 trunc from 0x030887 to 0x01bd55 (0x14b31 bytes) 000972 1786745104.599738 write 0x00b232 thru 0x0151e8 (0x9fb7 bytes) 000975 1786745104.619481 mapwrite 0x02b1b5 thru 0x0357d9 (0xa625 bytes) 000976 1786745104.740587 mapwrite 0x00346f thru 0x00bbfe (0x8790 bytes) 000977 1786745104.790285 trunc from 0x0357da to 0x033eaa (0x192f bytes) 000978 1786745104.804797 write 0x02d5b2 thru 0x039b3b (0xc58a bytes) 000979 1786745104.810661 read 0x00ffaa thru 0x01c138 (0xc18f bytes) 000980 1786745104.822802 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 6: 0x8eeafacd (stale) Mirror 13: 0x8eeafacd (stale) Mirror 15: 0x8eeafacd (stale) Mirror 16: 0x8eeafacd (stale) Mirror 18: 0x8eeafacd (stale) Mirror 21: 0x8eeafacd (stale) Mirror 22: 0x8eeafacd (stale) CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 1: 0x720234ed Mirror 23: 0x720234ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b3c: skipped 000981 1786745105.879530 MIRROR_RESYNC 9 mirrors 000982 1786745107.635901 write 0x010847 thru 0x01ba9a (0xb254 bytes) 000984 1786745107.756353 write 0x012dd2 thru 0x013f9b (0x11ca bytes) 000985 1786745107.758080 mapread 0x02921a thru 0x02c29a (0x3081 bytes) 000986 1786745107.804923 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 13: 0x720234ed (stale) Mirror 15: 0x720234ed (stale) Mirror 16: 0x720234ed (stale) Mirror 18: 0x720234ed (stale) Mirror 21: 0x720234ed (stale) Mirror 22: 0x720234ed (stale) Mirror 23: 0x720234ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 13: 0x720234ed (stale) Mirror 15: 0x720234ed (stale) Mirror 16: 0x720234ed (stale) Mirror 18: 0x720234ed (stale) Mirror 21: 0x720234ed (stale) Mirror 22: 0x720234ed (stale) Mirror 23: 0x720234ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b3c: skipped 000987 1786745108.496767 mapread 0x0009ff thru 0x0060e6 (0x56e8 bytes) 000990 1786745108.553949 mapwrite 0x000752 thru 0x0036d8 (0x2f87 bytes) 000991 1786745108.639560 write 0x022fa7 thru 0x029349 (0x63a3 bytes) 000994 1786745108.655087 mapread 0x02ad0a thru 0x034092 (0x9389 bytes) 000996 1786745108.692462 read 0x001344 thru 0x0049c5 (0x3682 bytes) 000997 1786745108.703648 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 13: 0x720234ed (stale) Mirror 15: 0x720234ed (stale) Mirror 16: 0x720234ed (stale) Mirror 18: 0x720234ed (stale) Mirror 21: 0x720234ed (stale) Mirror 23: 0x720234ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b3c): Mirror 13: 0x720234ed (stale) Mirror 15: 0x720234ed (stale) Mirror 16: 0x720234ed (stale) Mirror 18: 0x720234ed (stale) Mirror 21: 0x720234ed (stale) Mirror 23: 0x720234ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b3c: skipped 000999 1786745109.337728 mapread 0x016e46 thru 0x023c69 (0xce24 bytes) 001000 1786745109.385985 read 0x024a7f thru 0x02eda0 (0xa322 bytes) All operations completed A-OK! PASS 70a (113s) == sanity-flr test 71: check mirror extend parent fid ==== 18:05:14 (1786745114) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 18:05:17 (1786745117) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.296143 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187815 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 18:05:26 (1786745126) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3837632 .. Extending file size to 3151040 .. done Extending file size to 5296640 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8559040 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5980064 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8620000 .. Extending file size to 7867392 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8287168 .. Extending file size to 8891936 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8423072 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8429312 .. Extending file size to 8903136 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8658400 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8769376 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57462 57464 57465 57466 57467 done 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) 192.168.206.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:06:51 (1786745211) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2654272 .. Extending file size to 4874912 .. Extending file size to 5437920 .. Extending file size to 8284064 .. Extending file size to 5398688 .. Extending file size to 8823744 .. Extending file size to 9301248 .. Extending file size to 8955776 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9159136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59092 59093 59095 59096 59099 failed 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) 192.168.206.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:08:08 (1786745288) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139512 s, 15.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.461176 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbc2a699e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:08:25 (1786745305) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0771996 s, 27.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 18:08:32 (1786745312) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x213:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:08:39 (1786745319) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.273741 s, 7.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:08:47 (1786745327) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220304 s, 9.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:08:55 (1786745335) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144975 s, 14.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:09:29 (1786745369) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171418 s, 6.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:10:24 (1786745424) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.550067 s, 3.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:11:22 (1786745482) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. Creating new pool test_204f oleg616-server: Pool lustre.test_204f created Adding targets to pool oleg616-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg616-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.140776 s, 14.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg616-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg616-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg616-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:12:40 (1786745560) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:12:49 (1786745569) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:12:57 (1786745577) oleg616-server: Pool lustre.test_206 created Adding targets to pool oleg616-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg616-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg616-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg616-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg616-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:13:20 (1786745600) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485862 s, 21.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345717 s, 30.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xebc9547a Mirror 3: 0xebc9547a PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:13:28 (1786745608) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:13:32 (1786745612) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:13:35 (1786745615) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.180817 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1490000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xdb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xdb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xdb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x950000 Read this (4 bytes): BBBB 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:14:11 (1786745651) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.026192 s, 313 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:14:18 (1786745658) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056063 s, 18.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834488 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 18:14:20.000000000 -0400 Modify: 2026-08-14 18:14:20.000000000 -0400 Change: 2026-08-14 18:14:22.000000000 -0400 Birth: 2026-08-14 18:14:21.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:14:28 (1786745668) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:14:32 (1786745672) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.599836 s, 68.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0632611 s, 64.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 18:14 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 18:14 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:14:41 (1786745681) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:14:50 (1786745690) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1733 sec =========== 18:15:00 (1786745700) === sanity-flr: start cleanup 18:15:02 (1786745702) === === sanity-flr: finish cleanup 18:15:08 (1786745708) ===