-----============= acceptance-small: sanity-flr ============----- Sun Nov 16 20:21:01 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg157-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:21:18 (1763342478) === oleg157-client.virtnet: executing check_config_client /mnt/lustre oleg157-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg157-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98e858cee000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98e858cee000.idle_timeout=debug disable quota as required oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:21:43 (1763342503) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7549] == sanity-flr test 0a: lfs mirror create with -N option == 20:21:45 (1763342505) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:21:59 (1763342519) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:22:02 (1763342522) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:22:05 (1763342525) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:22:24 (1763342544) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:22:27 (1763342547) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:22:30 (1763342550) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98e858cee000.stats=clear osc.lustre-OST0001-osc-ffff98e858cee000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:22:37 (1763342557) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:22:46 (1763342566) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:22:53 (1763342573) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:22:55 (1763342575) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 20:23:02 (1763342582) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:23:08 (1763342588) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:23:11 (1763342591) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:23:17 (1763342597) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:23:25 (1763342605) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:23:30 (1763342610) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.7047 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166309 s, 6.3 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:23:40 (1763342620) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.253572 s, 45.5 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:23:47 (1763342627) Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg157-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (54s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:24:42 (1763342682) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.413722 s, 20.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (26s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:25:08 (1763342708) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg157-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (57s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:26:05 (1763342765) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.44803 s, 15.2 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 20:26 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.829s user 0m0.000s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98e858cee000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.051s user 0m0.000s sys 0m0.211s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg157-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg157-server oleg157-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.089s user 0m0.002s sys 0m0.159s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg157-server: oleg157-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg157-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98e858cee000.ost_server_uuid in FULL state after 0 sec PASS 33b (70s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:27:15 (1763342835) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:27:18 (1763342838) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:27:21 (1763342841) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:27:24 (1763342844) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 20:27:31 (1763342851) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.633894 s, 19.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:27:42 (1763342862) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214449 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393641 s, 26.6 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:27:48 (1763342868) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:27:56 (1763342876) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110626 s, 926 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110797 s, 924 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000868642 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0016247 s, 630 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 20:28:07 (1763342887) create test files with size 15747654 .. b11e6f45c0f3771d4afb33701a1bff35 - a38ed5f690b357b4e3f94e22207922ac - 44bab847dad52da9a6942e1758372290 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg157-client.virtnet /mnt/lustre (opts:) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 20:28:30 (1763342910) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:28:41 (1763342921) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:28:47 (1763342927) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106768 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367289 s, 28.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 20:28:55 (1763342935) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207237 s, 10.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.306884 s, 13.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30644 s, 6.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 20:29:10 (1763342950) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 20:29:14 (1763342954) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:29:18 (1763342958) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213407 s, 49.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 20:29:27 (1763342967) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:29:31 (1763342971) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:29:41 (1763342981) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.188916 s, 55.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg157-client.virtnet /mnt/lustre (opts:) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:29:52 (1763342992) llite.lustre-ffff98e8490cf000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.233834 s, 44.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg157-client.virtnet /mnt/lustre (opts:) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff98e846c2b800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:30:04 (1763343004) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:30:34 (1763343034) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (28s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:31:02 (1763343062) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:31:05 (1763343065) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.304231 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.193747 s, 16.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7c085aa5 (stale) Mirror 2: 0x7c085aa5 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: f9e8034cab3e8b43f956b08907897fe5 - mirror 1 checksum: 6d5ab33677ac5e7b4362dd0ac42951c9 - mirror 2 checksum: f9e8034cab3e8b43f956b08907897fe5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 20:31:08 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 20:31:08 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f9e8034cab3e8b43f956b08907897fe5 - mirror 1 checksum: 6d5ab33677ac5e7b4362dd0ac42951c9 - mirror 2 checksum: 6d5ab33677ac5e7b4362dd0ac42951c9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 20:31:08 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 20:31:08 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:31:16 (1763343076) Failing mds1 on oleg157-server Stopping /mnt/lustre-mds1 (opts:) on oleg157-server 20:31:21 (1763343081) shut down facet: mds1 facet_host: oleg157-server facet_failover_host: oleg157-server Failover mds1 to oleg157-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 20:31:41 (1763343101) targets are mounted 20:31:41 (1763343101) facet_failover done oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226722 s, 46.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (37s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:31:53 (1763343113) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:31:55 (1763343115) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:31:57 (1763343117) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00612151 s, 2.7 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:32:02 (1763343122) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:32:04 (1763343124) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:32:06 (1763343126) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:32:26 (1763343146) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:32:53 (1763343173) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:33:17 (1763343197) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:33:40 (1763343220) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 20:33:46 (1763343226) mirror create pid 52249 mirror split pid 52251 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52249 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52251 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 20:34:52 (1763343292) Seed set to 3293 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1763343294.123120 mapwrite 0x03e780 thru 0x03ffff (0x1880 bytes) truncating to largest ever: 0x215d2 000003 1763343294.256012 trunc from 0x040000 to 0x0215d1 (0x1ea2e bytes) 000005 1763343294.270484 mapwrite 0x019216 thru 0x0291ca (0xffb5 bytes) 000010 1763343294.384636 read 0x01a5ed thru 0x022539 (0x7f4d bytes) 000012 1763343294.387176 write 0x012e97 thru 0x01fafd (0xcc67 bytes) 000013 1763343294.388517 trunc from 0x0291cb to 0x019ea9 (0xf321 bytes) 000015 1763343294.411769 write 0x033cac thru 0x0391e2 (0x5537 bytes) 000016 1763343294.413918 read 0x01e67e thru 0x025e1c (0x779f bytes) 000018 1763343294.430038 trunc from 0x0391e3 to 0x009f74 (0x2f26e bytes) 000019 1763343294.457525 trunc from 0x009f75 to 0x0103f1 (0x647d bytes) 000020 1763343294.481881 write 0x01709e thru 0x0235ad (0xc510 bytes) 000021 1763343294.485617 mapwrite 0x01e2aa thru 0x0231be (0x4f15 bytes) 000022 1763343294.546610 trunc from 0x0235ae to 0x00b350 (0x1825d bytes) 000026 1763343294.576786 trunc from 0x00b351 to 0x00466c (0x6ce4 bytes) 000027 1763343294.612958 mapread 0x0041a3 thru 0x00466c (0x4ca bytes) 000028 1763343294.613877 mapread 0x0031eb thru 0x00466c (0x1482 bytes) 000029 1763343294.614652 write 0x002cc1 thru 0x00bf12 (0x9252 bytes) 000032 1763343294.617152 read 0x00989f thru 0x00bf12 (0x2674 bytes) 000033 1763343294.617723 trunc from 0x00bf13 to 0x00771a (0x47f8 bytes) 000034 1763343294.630810 mapread 0x002291 thru 0x00771a (0x548a bytes) 000036 1763343294.632255 write 0x0306e7 thru 0x038148 (0x7a62 bytes) 000037 1763343294.633654 trunc from 0x038149 to 0x00904d (0x2f0fb bytes) 000039 1763343294.649511 write 0x0178ce thru 0x020713 (0x8e46 bytes) 000040 1763343294.650857 mapread 0x00d728 thru 0x0142bc (0x6b95 bytes) 000042 1763343294.660699 mapread 0x010258 thru 0x0193c9 (0x9172 bytes) 000046 1763343294.662781 mapwrite 0x027cf0 thru 0x02cd5b (0x506c bytes) 000047 1763343294.757603 trunc from 0x02cd5c to 0x00d896 (0x1f4c5 bytes) 000048 1763343294.785360 MIRROR_EXTEND to 2 mirrors 000051 1763343295.464799 mapwrite 0x006f61 thru 0x00dba7 (0x6c47 bytes) 000052 1763343295.664026 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) 000055 1763343295.868175 mapwrite 0x03d98e thru 0x03ffff (0x2672 bytes) 000056 1763343295.985435 read 0x03b537 thru 0x03ffff (0x4ac9 bytes) 000059 1763343295.986958 read 0x002f3a thru 0x00e799 (0xb860 bytes) 000060 1763343296.012756 mapwrite 0x03a3e3 thru 0x03ffff (0x5c1d bytes) 000061 1763343296.076767 mapwrite 0x0066a3 thru 0x00ca3d (0x639b bytes) 000062 1763343296.127723 MIRROR_EXTEND to 2 mirrors 000063 1763343296.730009 write 0x0366b8 thru 0x03ffff (0x9948 bytes) 000065 1763343296.807014 mapwrite 0x02fbea thru 0x031d67 (0x217e bytes) 000066 1763343296.888714 write 0x012de9 thru 0x01a645 (0x785d bytes) 000067 1763343296.906754 read 0x033233 thru 0x03d4c2 (0xa290 bytes) 000068 1763343296.914220 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2747f890 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3e579256 Mirror 3: 0x3e579256 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000070 1763343297.977985 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2747f890 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2747f890 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000071 1763343298.338979 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2747f890 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3e579256 Mirror 3: 0x3e579256 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000072 1763343299.390510 write 0x022fc2 thru 0x030a66 (0xdaa5 bytes) 000073 1763343299.501891 read 0x0333ae thru 0x03911d (0x5d70 bytes) 000076 1763343299.512326 mapread 0x0202cf thru 0x02fee3 (0xfc15 bytes) 000077 1763343299.533828 trunc from 0x040000 to 0x004e49 (0x3b1b6 bytes) 000079 1763343299.590681 mapread 0x00413b thru 0x004e49 (0xd0f bytes) 000081 1763343299.603188 read 0x000a8e thru 0x004e49 (0x43bc bytes) 000083 1763343299.616549 read 0x000174 thru 0x004e49 (0x4cd6 bytes) 000085 1763343299.618709 read 0x001e23 thru 0x004e49 (0x3027 bytes) 000087 1763343299.619794 mapwrite 0x009278 thru 0x01626f (0xcff8 bytes) 000088 1763343299.776651 mapwrite 0x035731 thru 0x03ffff (0xa8cf bytes) 000091 1763343299.947718 mapwrite 0x03a52e thru 0x03ffff (0x5ad2 bytes) 000095 1763343300.030143 write 0x037d36 thru 0x03fc62 (0x7f2d bytes) 000096 1763343300.031208 trunc from 0x040000 to 0x015047 (0x2afb8 bytes) 000097 1763343300.063851 mapwrite 0x017520 thru 0x01e902 (0x73e3 bytes) 000098 1763343300.249488 write 0x03439d thru 0x03ffff (0xbc63 bytes) 000100 1763343300.251307 mapwrite 0x03cc5b thru 0x03ffff (0x33a5 bytes) 000103 1763343300.320315 mapwrite 0x037014 thru 0x03ffff (0x8fec bytes) 000107 1763343300.415931 read 0x01ffc3 thru 0x02146b (0x14a9 bytes) 000111 1763343300.441156 trunc from 0x040000 to 0x00e5fc (0x31a03 bytes) 000112 1763343300.470259 mapread 0x00b870 thru 0x00e5fc (0x2d8d bytes) 000114 1763343300.472806 write 0x02bd2e thru 0x037e9b (0xc16e bytes) 000115 1763343300.474159 read 0x01c9b5 thru 0x028ae4 (0xc130 bytes) 000116 1763343300.487820 mapread 0x034baa thru 0x037701 (0x2b58 bytes) 000117 1763343300.488730 write 0x0202cf thru 0x028bdb (0x890d bytes) 000118 1763343300.494679 MIRROR_RESYNC 3 mirrors 000119 1763343301.330395 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) 000121 1763343301.663614 read 0x01d223 thru 0x01e818 (0x15f6 bytes) 000122 1763343301.672560 read 0x01b33d thru 0x024d2e (0x99f2 bytes) 000124 1763343301.684656 write 0x00f2f0 thru 0x01c763 (0xd474 bytes) 000126 1763343301.795828 read 0x026d45 thru 0x02b9f5 (0x4cb1 bytes) 000127 1763343301.825145 mapread 0x0277c6 thru 0x029b0c (0x2347 bytes) 000128 1763343301.826167 write 0x000607 thru 0x00ebcc (0xe5c6 bytes) 000130 1763343301.835249 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) 000131 1763343302.031051 trunc from 0x037e9c to 0x002ceb (0x351b0 bytes) 000132 1763343302.072076 read 0x002bd2 thru 0x002ceb (0x11a bytes) 000134 1763343302.086070 write 0x005510 thru 0x012a77 (0xd568 bytes) truncating to largest ever: 0x25f70 000136 1763343302.088243 trunc from 0x012a78 to 0x025f6f (0x134f8 bytes) 000137 1763343302.129250 MIRROR_EXTEND to 2 mirrors 000139 1763343302.779942 mapread 0x00454f thru 0x00ba08 (0x74ba bytes) 000140 1763343302.804174 MIRROR_EXTEND to 3 mirrors 000142 1763343303.510799 trunc from 0x025f70 to 0x00de61 (0x1810e bytes) 000143 1763343303.615751 mapread 0x0010d6 thru 0x0095d1 (0x84fc bytes) 000146 1763343303.636166 mapread 0x009654 thru 0x00de61 (0x480e bytes) 000147 1763343303.637142 read 0x0005b4 thru 0x0086c9 (0x8116 bytes) 000149 1763343303.643787 trunc from 0x00de62 to 0x00502f (0x8e32 bytes) 000151 1763343303.671931 read 0x004103 thru 0x00502f (0xf2d bytes) 000152 1763343303.672544 mapread 0x00435e thru 0x00502f (0xcd2 bytes) 000153 1763343303.673396 write 0x036c71 thru 0x03ffff (0x938f bytes) 000155 1763343303.676621 mapread 0x0339aa thru 0x03fe7e (0xc4d5 bytes) 000156 1763343303.695606 mapwrite 0x00f257 thru 0x01d5bc (0xe366 bytes) 000157 1763343303.753040 mapwrite 0x00267d thru 0x00dc34 (0xb5b8 bytes) 000158 1763343303.813204 read 0x00b115 thru 0x019dd6 (0xecc2 bytes) 000159 1763343303.814080 trunc from 0x040000 to 0x0070c7 (0x38f38 bytes) 000160 1763343303.839501 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x70c8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x70c8): Mirror 2: 0x3a14c780 (stale) Mirror 3: 0x3a14c780 (stale) CRC-32 checksum value for chunk [0, 0x70c8): Mirror 1: 0x5fa9d812 Mirror 4: 0x5fa9d812 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x70c8: skipped 000161 1763343305.026928 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x70c8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x70c8): Mirror 2: 0x3a14c780 (stale) Mirror 3: 0x3a14c780 (stale) CRC-32 checksum value for chunk [0, 0x70c8): Mirror 1: 0x5fa9d812 Mirror 4: 0x5fa9d812 Mirror 5: 0x5fa9d812 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x70c8: skipped 000162 1763343305.869834 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x70c8): Mirror 2: 0x3a14c780 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x70c8): Mirror 2: 0x3a14c780 (stale) CRC-32 checksum value for chunk [0, 0x70c8): Mirror 1: 0x5fa9d812 Mirror 4: 0x5fa9d812 Mirror 5: 0x5fa9d812 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x70c8: skipped truncating to largest ever: 0x2d913 000165 1763343306.416835 trunc from 0x0070c8 to 0x02d912 (0x2684b bytes) 000166 1763343306.509138 trunc from 0x02d913 to 0x014cff (0x18c13 bytes) 000168 1763343306.536596 mapread 0x0025a6 thru 0x004021 (0x1a7c bytes) 000170 1763343306.557417 write 0x0209ab thru 0x024d47 (0x439d bytes) 000172 1763343306.573860 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24d48): Mirror 2: 0xa2251022 (stale) Mirror 4: 0x231c2462 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d48): Mirror 2: 0xa2251022 (stale) Mirror 4: 0x231c2462 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d48: skipped 000173 1763343306.912825 write 0x0218a6 thru 0x0296aa (0x7e05 bytes) 000175 1763343306.925163 read 0x00e912 thru 0x019d7c (0xb46b bytes) 000177 1763343306.954086 write 0x00eac1 thru 0x01654c (0x7a8c bytes) 000178 1763343306.955124 mapwrite 0x00f3da thru 0x014b7d (0x57a4 bytes) 000179 1763343307.025767 read 0x00263a thru 0x00b48b (0x8e52 bytes) 000180 1763343307.047592 read 0x017304 thru 0x02350b (0xc208 bytes) 000181 1763343307.050992 mapwrite 0x034aa1 thru 0x03cf60 (0x84c0 bytes) 000186 1763343307.162958 trunc from 0x03cf61 to 0x013e8d (0x290d3 bytes) 000187 1763343307.183041 MIRROR_RESYNC 3 mirrors 000190 1763343307.863658 mapwrite 0x02f9fc thru 0x03dfbc (0xe5c1 bytes) 000191 1763343308.085096 mapwrite 0x024f23 thru 0x02eae1 (0x9bbf bytes) 000192 1763343308.146045 mapread 0x0064f3 thru 0x0076a1 (0x11af bytes) 000194 1763343308.167821 MIRROR_RESYNC 3 mirrors 000195 1763343308.831899 read 0x037a17 thru 0x03dfbc (0x65a6 bytes) 000197 1763343308.841391 trunc from 0x03dfbd to 0x020bd0 (0x1d3ec bytes) 000198 1763343308.960606 mapwrite 0x00c1a1 thru 0x00fcd7 (0x3b37 bytes) 000199 1763343309.042335 read 0x002701 thru 0x00313b (0xa3b bytes) 000201 1763343309.072374 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20bd1): Mirror 4: 0xf6c7a73a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20bd1): Mirror 4: 0xf6c7a73a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20bd1: skipped 000202 1763343309.384543 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20bd1): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20bd1): Mirror 4: 0xf6c7a73a (stale) CRC-32 checksum value for chunk [0, 0x20bd1): Mirror 1: 0xa900f492 Mirror 5: 0xa900f492 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20bd1: skipped 000203 1763343310.175729 mapwrite 0x033957 thru 0x03ffff (0xc6a9 bytes) truncating to largest ever: 0x36a4f 000204 1763343310.348949 trunc from 0x040000 to 0x036a4e (0x95b1 bytes) 000206 1763343310.376222 trunc from 0x036a4f to 0x0161c3 (0x2088b bytes) 000207 1763343310.403859 read 0x0052ec thru 0x00ae0a (0x5b1f bytes) 000212 1763343310.432100 MIRROR_RESYNC 3 mirrors 000213 1763343311.089721 mapread 0x01448c thru 0x0161c3 (0x1d38 bytes) 000215 1763343311.113797 MIRROR_EXTEND to 4 mirrors 000216 1763343311.748461 write 0x00f490 thru 0x019934 (0xa4a5 bytes) 000220 1763343311.838879 mapread 0x00d7d2 thru 0x01122d (0x3a5c bytes) 000221 1763343311.853868 write 0x035054 thru 0x03ffff (0xafac bytes) truncating to largest ever: 0x38121 000223 1763343311.856119 trunc from 0x040000 to 0x038120 (0x7edf bytes) 000225 1763343311.891691 write 0x0375aa thru 0x03d95a (0x63b1 bytes) 000226 1763343311.892925 trunc from 0x03d95b to 0x02639e (0x175bc bytes) 000227 1763343311.922266 write 0x00755a thru 0x017321 (0xfdc8 bytes) 000228 1763343311.923843 read 0x007ba1 thru 0x00a493 (0x28f3 bytes) 000230 1763343311.926771 write 0x03b8b5 thru 0x03ffff (0x474b bytes) 000232 1763343311.937138 trunc from 0x040000 to 0x024c76 (0x1b389 bytes) 000234 1763343311.967472 mapread 0x0122f4 thru 0x0178a9 (0x55b6 bytes) 000235 1763343311.969608 read 0x001dba thru 0x00e4c4 (0xc70b bytes) 000236 1763343311.992150 write 0x02fed9 thru 0x03b9c3 (0xbaeb bytes) 000237 1763343311.994231 trunc from 0x03b9c4 to 0x024f8d (0x16a36 bytes) 000238 1763343312.019268 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24f8e): Mirror 5: 0x213f8cf9 (stale) Mirror 6: 0x213f8cf9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f8e): Mirror 5: 0x213f8cf9 (stale) Mirror 6: 0x213f8cf9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f8e: skipped 000240 1763343312.355197 mapwrite 0x02fc7d thru 0x031866 (0x1bea bytes) 000241 1763343312.480551 mapread 0x01bad4 thru 0x026e69 (0xb396 bytes) 000243 1763343312.521425 mapwrite 0x0006d4 thru 0x0095fe (0x8f2b bytes) 000245 1763343312.632605 mapwrite 0x00a363 thru 0x00fcd8 (0x5976 bytes) 000246 1763343312.717867 write 0x003924 thru 0x011038 (0xd715 bytes) 000247 1763343312.747067 trunc from 0x031867 to 0x030a90 (0xdd6 bytes) 000248 1763343312.783532 write 0x0227e0 thru 0x02dc5a (0xb47b bytes) 000249 1763343312.785444 mapread 0x00bcf5 thru 0x0132f0 (0x75fc bytes) 000250 1763343312.811954 write 0x00b792 thru 0x015b79 (0xa3e8 bytes) 000251 1763343312.819810 MIRROR_RESYNC 3 mirrors 000252 1763343313.619187 MIRROR_RESYNC 3 mirrors 000253 1763343314.006321 trunc from 0x030a91 to 0x00279f (0x2e2f1 bytes) 000254 1763343314.118271 mapread 0x000994 thru 0x00279f (0x1e0c bytes) 000256 1763343314.141798 mapwrite 0x0090f4 thru 0x012c83 (0x9b90 bytes) 000258 1763343314.253476 MIRROR_RESYNC 3 mirrors 000259 1763343314.919160 mapread 0x010025 thru 0x012c83 (0x2c5f bytes) 000260 1763343314.955170 MIRROR_EXTEND to 4 mirrors truncating to largest ever: 0x3f781 000261 1763343315.623515 trunc from 0x012c84 to 0x03f780 (0x2cafd bytes) 000262 1763343315.707118 write 0x03aa3c thru 0x03dd9a (0x335f bytes) 000264 1763343315.719892 trunc from 0x03f781 to 0x032a26 (0xcd5a bytes) 000265 1763343315.752383 write 0x03f772 thru 0x03ffff (0x88e bytes) 000266 1763343315.753445 read 0x034d35 thru 0x03ffff (0xb2cb bytes) 000267 1763343315.777215 MIRROR_RESYNC 4 mirrors 000268 1763343316.678358 mapread 0x028755 thru 0x036ec2 (0xe76e bytes) 000269 1763343316.708869 read 0x00024c thru 0x001697 (0x144c bytes) 000272 1763343316.714894 mapread 0x008e35 thru 0x00e4a9 (0x5675 bytes) 000273 1763343316.724526 mapread 0x03979e thru 0x03ffff (0x6862 bytes) 000275 1763343316.732052 read 0x03a311 thru 0x03c7e1 (0x24d1 bytes) 000276 1763343316.732438 trunc from 0x040000 to 0x03095f (0xf6a0 bytes) 000277 1763343316.829631 write 0x028a27 thru 0x02bdf7 (0x33d1 bytes) 000279 1763343316.848725 mapwrite 0x03a720 thru 0x03ffff (0x58e0 bytes) 000281 1763343316.960353 mapwrite 0x007ca5 thru 0x00d172 (0x54ce bytes) 000282 1763343317.033571 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x43688ea0 (stale) Mirror 6: 0x43688ea0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x43688ea0 (stale) Mirror 6: 0x43688ea0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000284 1763343317.386382 mapread 0x01233c thru 0x012362 (0x27 bytes) 000285 1763343317.393181 mapwrite 0x02425e thru 0x0272f2 (0x3095 bytes) 000286 1763343317.468376 read 0x03c001 thru 0x03ffff (0x3fff bytes) 000289 1763343317.479715 read 0x02eade thru 0x02ed34 (0x257 bytes) 000290 1763343317.493699 mapwrite 0x0171c0 thru 0x0224a7 (0xb2e8 bytes) 000293 1763343317.548700 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x43688ea0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x43688ea0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1763343317.913609 mapwrite 0x020a61 thru 0x020bbb (0x15b bytes) 000297 1763343317.980982 MIRROR_RESYNC 2 mirrors 000300 1763343318.657991 write 0x01086c thru 0x018bbf (0x8354 bytes) 000301 1763343318.789140 mapwrite 0x026bb3 thru 0x027dde (0x122c bytes) 000302 1763343318.858886 write 0x000f3a thru 0x002ad9 (0x1ba0 bytes) 000303 1763343318.871408 mapwrite 0x00c894 thru 0x01774e (0xaebb bytes) 000304 1763343318.980053 trunc from 0x040000 to 0x03d71e (0x28e1 bytes) 000307 1763343319.031481 mapread 0x016b90 thru 0x020af8 (0x9f69 bytes) 000308 1763343319.046913 mapwrite 0x0233c7 thru 0x030b4b (0xd785 bytes) 000310 1763343319.111424 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) 000312 1763343319.263640 mapwrite 0x0076bd thru 0x009e47 (0x278b bytes) 000314 1763343319.343603 mapwrite 0x007ec7 thru 0x00bc83 (0x3dbd bytes) 000315 1763343319.404061 write 0x02a621 thru 0x03005c (0x5a3c bytes) 000316 1763343319.419996 mapread 0x008097 thru 0x012e69 (0xadd3 bytes) 000317 1763343319.433546 write 0x005a24 thru 0x006706 (0xce3 bytes) 000318 1763343319.434490 trunc from 0x03d71f to 0x026b99 (0x16b85 bytes) 000320 1763343319.461379 mapwrite 0x021e31 thru 0x02bcf9 (0x9ec9 bytes) 000322 1763343319.547452 write 0x03964d thru 0x03c628 (0x2fdc bytes) 000327 1763343319.557830 MIRROR_EXTEND to 2 mirrors 000329 1763343320.213246 read 0x02f5a4 thru 0x032a4b (0x34a8 bytes) 000331 1763343320.223057 write 0x03b9eb thru 0x03ffff (0x4615 bytes) 000332 1763343320.296394 write 0x02c4a5 thru 0x038444 (0xbfa0 bytes) 000333 1763343320.319629 mapwrite 0x013109 thru 0x017a4e (0x4946 bytes) 000334 1763343320.376211 mapwrite 0x001e8e thru 0x00a81e (0x8991 bytes) 000335 1763343320.438506 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf1d70200 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb886ce47 Mirror 3: 0xb886ce47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000336 1763343321.123171 MIRROR_RESYNC 3 mirrors 000339 1763343321.806524 mapwrite 0x02188d thru 0x025057 (0x37cb bytes) 000340 1763343322.020049 mapwrite 0x006090 thru 0x00c11c (0x608d bytes) 000342 1763343322.083263 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb886ce47 (stale) Mirror 3: 0xb886ce47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x925e0ad4 Mirror 4: 0x925e0ad4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000343 1763343322.786669 write 0x014990 thru 0x01ff84 (0xb5f5 bytes) 000344 1763343322.902186 mapwrite 0x033400 thru 0x03ffff (0xcc00 bytes) 000347 1763343323.024908 mapread 0x02f529 thru 0x03839c (0x8e74 bytes) 000348 1763343323.027722 write 0x01a7d9 thru 0x023fea (0x9812 bytes) 000349 1763343323.059351 MIRROR_RESYNC 4 mirrors 000350 1763343323.920747 read 0x0040f0 thru 0x0137c9 (0xf6da bytes) 000353 1763343323.936940 mapwrite 0x036887 thru 0x03ea67 (0x81e1 bytes) 000356 1763343324.162770 mapwrite 0x034f04 thru 0x03ffff (0xb0fc bytes) 000357 1763343324.235631 mapwrite 0x03b8bf thru 0x03f9c2 (0x4104 bytes) 000358 1763343324.298195 mapread 0x005e29 thru 0x00c613 (0x67eb bytes) 000359 1763343324.319958 MIRROR_RESYNC 4 mirrors 000360 1763343325.078548 MIRROR_EXTEND to 5 mirrors 000361 1763343325.756360 mapwrite 0x027685 thru 0x02ec16 (0x7592 bytes) 000363 1763343325.948213 mapwrite 0x03abe3 thru 0x03ffff (0x541d bytes) 000364 1763343326.014629 mapread 0x002002 thru 0x0062f0 (0x42ef bytes) 000366 1763343326.038070 mapread 0x00f932 thru 0x017cce (0x839d bytes) 000367 1763343326.071085 MIRROR_RESYNC 5 mirrors 000368 1763343326.994390 mapread 0x01f255 thru 0x026b90 (0x793c bytes) 000372 1763343327.025101 trunc from 0x040000 to 0x01ba08 (0x245f7 bytes) 000373 1763343327.114012 trunc from 0x01ba09 to 0x035b67 (0x1a15f bytes) 000374 1763343327.150272 trunc from 0x035b68 to 0x00772a (0x2e43d bytes) 000376 1763343327.175647 write 0x022750 thru 0x029719 (0x6fca bytes) 000377 1763343327.177630 write 0x0133bd thru 0x020187 (0xcdcb bytes) 000379 1763343327.198438 mapwrite 0x0122d6 thru 0x01264b (0x376 bytes) 000380 1763343327.247552 trunc from 0x02971a to 0x02b82f (0x2116 bytes) 000381 1763343327.270797 trunc from 0x02b830 to 0x021ea3 (0x998c bytes) 000382 1763343327.291172 read 0x011dcf thru 0x01a36e (0x85a0 bytes) 000383 1763343327.291840 trunc from 0x021ea4 to 0x00cfb4 (0x14eef bytes) 000384 1763343327.311584 read 0x008d92 thru 0x00cfb4 (0x4223 bytes) 000385 1763343327.323454 trunc from 0x00cfb5 to 0x0123a1 (0x53ed bytes) 000386 1763343327.343701 trunc from 0x0123a2 to 0x0228c8 (0x10527 bytes) 000387 1763343327.375839 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x228c9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x228c9): Mirror 2: 0xae0b8a72 (stale) Mirror 3: 0xae0b8a72 (stale) Mirror 4: 0xae0b8a72 (stale) Mirror 5: 0xae0b8a72 (stale) CRC-32 checksum value for chunk [0, 0x228c9): Mirror 1: 0x18b40201 Mirror 6: 0x18b40201 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x228c9: skipped 000388 1763343328.041239 mapread 0x00c1fe thru 0x019849 (0xd64c bytes) 000389 1763343328.082977 mapread 0x014e94 thru 0x0221ec (0xd359 bytes) 000390 1763343328.090085 read 0x01227d thru 0x01ad2a (0x8aae bytes) 000392 1763343328.091878 read 0x00878d thru 0x00e058 (0x58cc bytes) 000393 1763343328.098267 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x228c9): Mirror 2: 0xae0b8a72 (stale) Mirror 3: 0xae0b8a72 (stale) Mirror 4: 0xae0b8a72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x228c9): Mirror 2: 0xae0b8a72 (stale) Mirror 3: 0xae0b8a72 (stale) Mirror 4: 0xae0b8a72 (stale) CRC-32 checksum value for chunk [0, 0x228c9): Mirror 1: 0x18b40201 Mirror 6: 0x18b40201 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x228c9: skipped 000394 1763343328.724589 mapread 0x00bd51 thru 0x00efd1 (0x3281 bytes) 000396 1763343328.779144 mapwrite 0x01180f thru 0x016b0a (0x52fc bytes) 000399 1763343328.962127 write 0x031c89 thru 0x037369 (0x56e1 bytes) 000400 1763343328.963239 write 0x004c7f thru 0x0094d7 (0x4859 bytes) 000401 1763343328.978670 mapwrite 0x0318f7 thru 0x03bbce (0xa2d8 bytes) 000402 1763343329.092814 mapread 0x02bf74 thru 0x03bb88 (0xfc15 bytes) 000403 1763343329.109207 mapwrite 0x005aed thru 0x00d19b (0x76af bytes) 000404 1763343329.166217 MIRROR_RESYNC 5 mirrors 000406 1763343330.290269 mapwrite 0x018534 thru 0x019185 (0xc52 bytes) 000407 1763343330.483483 read 0x01cbf6 thru 0x01ec31 (0x203c bytes) 000408 1763343330.489943 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bbcf): Mirror 2: 0x4006bb2d (stale) Mirror 3: 0x4006bb2d (stale) Mirror 6: 0x4006bb2d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bbcf): Mirror 2: 0x4006bb2d (stale) Mirror 3: 0x4006bb2d (stale) Mirror 6: 0x4006bb2d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bbcf: skipped 000409 1763343330.908153 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bbcf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bbcf): Mirror 2: 0x4006bb2d (stale) Mirror 3: 0x4006bb2d (stale) Mirror 6: 0x4006bb2d (stale) CRC-32 checksum value for chunk [0, 0x3bbcf): Mirror 1: 0x7c3eb7b2 Mirror 7: 0x7c3eb7b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bbcf: skipped 000410 1763343331.841186 mapwrite 0x039d97 thru 0x03ffff (0x6269 bytes) 000411 1763343332.019802 read 0x02bf57 thru 0x03b133 (0xf1dd bytes) 000413 1763343332.030841 write 0x02a929 thru 0x02f1a7 (0x487f bytes) 000416 1763343332.042026 read 0x03b19c thru 0x03ffff (0x4e64 bytes) 000418 1763343332.044868 trunc from 0x040000 to 0x01d4fd (0x22b02 bytes) 000419 1763343332.078014 write 0x026164 thru 0x02d840 (0x76dd bytes) 000420 1763343332.079350 trunc from 0x02d841 to 0x031b97 (0x4357 bytes) 000422 1763343332.099661 trunc from 0x031b98 to 0x008a0f (0x29188 bytes) 000423 1763343332.125869 mapwrite 0x0115b9 thru 0x0187ca (0x7212 bytes) 000424 1763343332.205057 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x187cb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x187cb): Mirror 2: 0xb5885de5 (stale) Mirror 3: 0xb5885de5 (stale) Mirror 6: 0xb5885de5 (stale) Mirror 7: 0x74c89d5c (stale) CRC-32 checksum value for chunk [0, 0x187cb): Mirror 1: 0x52282bf6 Mirror 8: 0x52282bf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x187cb: skipped 000427 1763343333.150095 read 0x0146bf thru 0x017de4 (0x3726 bytes) 000428 1763343333.169571 mapread 0x00f168 thru 0x0187ca (0x9663 bytes) 000432 1763343333.206909 read 0x009c14 thru 0x00ddd9 (0x41c6 bytes) 000433 1763343333.221921 mapread 0x01591d thru 0x0187ca (0x2eae bytes) 000437 1763343333.225392 mapread 0x010149 thru 0x017a4e (0x7906 bytes) 000441 1763343333.226861 read 0x0089ec thru 0x017855 (0xee6a bytes) 000443 1763343333.229218 mapread 0x00aa0f thru 0x012294 (0x7886 bytes) 000444 1763343333.230834 write 0x0271b9 thru 0x0343b3 (0xd1fb bytes) 000448 1763343333.360204 mapwrite 0x0168ee thru 0x02394d (0xd060 bytes) 000449 1763343333.471594 mapwrite 0x033ae0 thru 0x03ffff (0xc520 bytes) 000450 1763343333.624765 mapread 0x0334e4 thru 0x03ffff (0xcb1c bytes) 000451 1763343333.625829 mapwrite 0x00e9c3 thru 0x01540d (0x6a4b bytes) 000455 1763343333.751988 read 0x01b8a6 thru 0x027ffe (0xc759 bytes) 000456 1763343333.753154 read 0x02f5bd thru 0x036bee (0x7632 bytes) 000459 1763343333.754948 mapwrite 0x03a6ff thru 0x03f764 (0x5066 bytes) 000462 1763343333.843188 mapread 0x03130b thru 0x03f8ac (0xe5a2 bytes) 000465 1763343333.852659 MIRROR_RESYNC 6 mirrors 000467 1763343334.975826 read 0x02c252 thru 0x034bd9 (0x8988 bytes) 000468 1763343334.983584 write 0x008b0f thru 0x00ef38 (0x642a bytes) 000470 1763343335.082763 MIRROR_RESYNC 6 mirrors 000472 1763343336.234958 read 0x028d42 thru 0x02954a (0x809 bytes) 000476 1763343336.241428 mapread 0x03e7ec thru 0x03ffff (0x1814 bytes) 000477 1763343336.251941 write 0x022482 thru 0x02e303 (0xbe82 bytes) 000479 1763343336.343235 write 0x0383d4 thru 0x03ffff (0x7c2c bytes) 000480 1763343336.359209 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x53c91c83 (stale) Mirror 3: 0x53c91c83 (stale) Mirror 6: 0x53c91c83 (stale) Mirror 7: 0x53c91c83 (stale) Mirror 8: 0x53c91c83 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ba70f80 Mirror 9: 0x2ba70f80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000481 1763343337.449995 read 0x016c89 thru 0x01c405 (0x577d bytes) 000482 1763343337.487103 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x53c91c83 (stale) Mirror 6: 0x53c91c83 (stale) Mirror 7: 0x53c91c83 (stale) Mirror 8: 0x53c91c83 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x53c91c83 (stale) Mirror 6: 0x53c91c83 (stale) Mirror 7: 0x53c91c83 (stale) Mirror 8: 0x53c91c83 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ba70f80 Mirror 9: 0x2ba70f80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000483 1763343338.089770 write 0x01606d thru 0x01cd28 (0x6cbc bytes) 000485 1763343338.185649 trunc from 0x040000 to 0x00786e (0x38791 bytes) 000488 1763343338.229597 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x786f): Mirror 2: 0x14eb8cd1 (stale) Mirror 6: 0x14eb8cd1 (stale) Mirror 7: 0x14eb8cd1 (stale) Mirror 9: 0x14eb8cd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x786f): Mirror 2: 0x14eb8cd1 (stale) Mirror 6: 0x14eb8cd1 (stale) Mirror 7: 0x14eb8cd1 (stale) Mirror 9: 0x14eb8cd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x786f: skipped 000489 1763343338.621004 MIRROR_RESYNC 5 mirrors 000490 1763343339.588711 write 0x01fe46 thru 0x02d677 (0xd832 bytes) 000491 1763343339.661016 mapread 0x00a13d thru 0x018e8c (0xed50 bytes) 000494 1763343339.687705 trunc from 0x02d678 to 0x02e9c1 (0x134a bytes) 000495 1763343339.708948 trunc from 0x02e9c2 to 0x0283a9 (0x6618 bytes) 000497 1763343339.730123 mapwrite 0x02925d thru 0x0325a0 (0x9344 bytes) 000498 1763343339.914326 mapread 0x01384c thru 0x01953d (0x5cf2 bytes) 000500 1763343339.915826 write 0x00555e thru 0x0104fb (0xaf9e bytes) 000504 1763343339.918968 mapwrite 0x037555 thru 0x03953a (0x1fe6 bytes) 000505 1763343340.017844 write 0x035da9 thru 0x03ffff (0xa257 bytes) 000506 1763343340.019403 trunc from 0x040000 to 0x0241b3 (0x1be4c bytes) 000508 1763343340.042857 read 0x001a86 thru 0x0029d1 (0xf4c bytes) 000509 1763343340.056242 mapwrite 0x01e534 thru 0x021fd9 (0x3aa6 bytes) 000510 1763343340.110381 trunc from 0x0241b4 to 0x01eeaa (0x5309 bytes) 000511 1763343340.145999 read 0x01393b thru 0x01eeaa (0xb570 bytes) 000512 1763343340.146787 mapwrite 0x024c4c thru 0x028bfe (0x3fb3 bytes) 000515 1763343340.242638 trunc from 0x028bff to 0x003563 (0x2569b bytes) 000516 1763343340.270335 read 0x0017dd thru 0x003563 (0x1d87 bytes) 000518 1763343340.271814 read 0x0000bf thru 0x003563 (0x34a5 bytes) 000520 1763343340.272707 write 0x01e04f thru 0x025f6a (0x7f1c bytes) 000521 1763343340.274647 write 0x029f2c thru 0x038555 (0xe62a bytes) 000522 1763343340.284633 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38556): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38556): Mirror 2: 0xd12f502d (stale) Mirror 6: 0xd12f502d (stale) Mirror 7: 0xd12f502d (stale) Mirror 9: 0xd12f502d (stale) CRC-32 checksum value for chunk [0, 0x38556): Mirror 1: 0xf456734e Mirror 10: 0xf456734e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38556: skipped 000523 1763343341.158631 mapread 0x02abae thru 0x03499c (0x9def bytes) 000524 1763343341.181515 mapwrite 0x03c5ba thru 0x03f0cd (0x2b14 bytes) 000525 1763343341.421265 MIRROR_RESYNC 6 mirrors 000527 1763343342.720533 write 0x013252 thru 0x021886 (0xe635 bytes) 000529 1763343342.808144 mapread 0x03dcf5 thru 0x03f0cd (0x13d9 bytes) 000530 1763343342.817396 mapwrite 0x035b99 thru 0x03ffff (0xa467 bytes) 000533 1763343342.922776 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e914b75 (stale) Mirror 6: 0x1e914b75 (stale) Mirror 7: 0x1e914b75 (stale) Mirror 9: 0x1e914b75 (stale) Mirror 10: 0x1e914b75 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1893cc96 Mirror 11: 0x1893cc96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000535 1763343343.947800 write 0x0075c3 thru 0x00e84d (0x728b bytes) 000536 1763343344.041007 MIRROR_RESYNC 7 mirrors 000537 1763343345.195993 mapwrite 0x02ac4d thru 0x02fc09 (0x4fbd bytes) 000540 1763343345.435234 mapread 0x014518 thru 0x01d7d0 (0x92b9 bytes) 000541 1763343345.451898 mapread 0x00c4b3 thru 0x01a940 (0xe48e bytes) 000542 1763343345.468877 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9442aace (stale) Mirror 7: 0x9442aace (stale) Mirror 9: 0x9442aace (stale) Mirror 10: 0x9442aace (stale) Mirror 11: 0x9442aace (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9442aace (stale) Mirror 7: 0x9442aace (stale) Mirror 9: 0x9442aace (stale) Mirror 10: 0x9442aace (stale) Mirror 11: 0x9442aace (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000543 1763343345.921580 trunc from 0x040000 to 0x029582 (0x16a7d bytes) 000545 1763343345.936996 mapread 0x014d80 thru 0x01946f (0x46f0 bytes) 000546 1763343345.963509 read 0x024f43 thru 0x029582 (0x4640 bytes) 000548 1763343345.982517 MIRROR_RESYNC 6 mirrors 000550 1763343347.059187 write 0x03d711 thru 0x03e84e (0x113e bytes) 000551 1763343347.163577 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 6: 0x4592c7f0 (stale) Mirror 7: 0x4592c7f0 (stale) Mirror 9: 0x4592c7f0 (stale) Mirror 10: 0x4592c7f0 (stale) Mirror 11: 0x4592c7f0 (stale) CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 1: 0x50480fd4 Mirror 12: 0x50480fd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e84f: skipped 000552 1763343348.307169 mapread 0x018ae1 thru 0x025b97 (0xd0b7 bytes) 000554 1763343348.371576 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 6: 0x4592c7f0 (stale) Mirror 7: 0x4592c7f0 (stale) Mirror 10: 0x4592c7f0 (stale) Mirror 11: 0x4592c7f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 6: 0x4592c7f0 (stale) Mirror 7: 0x4592c7f0 (stale) Mirror 10: 0x4592c7f0 (stale) Mirror 11: 0x4592c7f0 (stale) CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 1: 0x50480fd4 Mirror 12: 0x50480fd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e84f: skipped 000555 1763343349.191076 mapwrite 0x037fd2 thru 0x03aa56 (0x2a85 bytes) 000556 1763343349.440082 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 6: 0x4592c7f0 (stale) Mirror 7: 0x4592c7f0 (stale) Mirror 10: 0x4592c7f0 (stale) Mirror 11: 0x4592c7f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e84f): Mirror 6: 0x4592c7f0 (stale) Mirror 7: 0x4592c7f0 (stale) Mirror 10: 0x4592c7f0 (stale) Mirror 11: 0x4592c7f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e84f: skipped 000557 1763343349.914679 trunc from 0x03e84f to 0x008628 (0x36226 bytes) 000559 1763343349.933175 mapwrite 0x03e821 thru 0x03ffff (0x17df bytes) 000560 1763343350.008997 trunc from 0x040000 to 0x0246ea (0x1b915 bytes) 000561 1763343350.026947 mapwrite 0x0057b8 thru 0x00b323 (0x5b6c bytes) 000563 1763343350.087444 write 0x0315cf thru 0x03c733 (0xb165 bytes) 000565 1763343350.090410 write 0x03f310 thru 0x03ffff (0xcf0 bytes) 000567 1763343350.091803 mapwrite 0x006acb thru 0x00a0be (0x35f4 bytes) 000568 1763343350.158118 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb1eb495d (stale) Mirror 7: 0xb1eb495d (stale) Mirror 10: 0xb1eb495d (stale) Mirror 11: 0xb1eb495d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd60b2e66 Mirror 12: 0xd60b2e66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000570 1763343350.930098 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb1eb495d (stale) Mirror 10: 0xb1eb495d (stale) Mirror 11: 0xb1eb495d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb1eb495d (stale) Mirror 10: 0xb1eb495d (stale) Mirror 11: 0xb1eb495d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd60b2e66 Mirror 12: 0xd60b2e66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000572 1763343351.565637 trunc from 0x040000 to 0x00d569 (0x32a96 bytes) 000573 1763343351.704168 read 0x000309 thru 0x007c01 (0x78f9 bytes) 000574 1763343351.715539 trunc from 0x00d56a to 0x018c88 (0xb71f bytes) 000578 1763343351.736692 trunc from 0x018c89 to 0x02edc2 (0x1613a bytes) 000579 1763343351.762612 MIRROR_RESYNC 5 mirrors 000581 1763343352.799885 read 0x000777 thru 0x005a73 (0x52fd bytes) 000584 1763343352.819567 mapread 0x00d9aa thru 0x01766f (0x9cc6 bytes) 000585 1763343352.848246 trunc from 0x02edc3 to 0x035fdc (0x721a bytes) 000587 1763343352.943873 trunc from 0x035fdd to 0x0051d2 (0x30e0a bytes) 000589 1763343352.969529 trunc from 0x0051d3 to 0x037b4c (0x3297a bytes) 000592 1763343352.991809 write 0x0368ea thru 0x03ffff (0x9716 bytes) 000593 1763343353.000080 read 0x00c185 thru 0x0101b4 (0x4030 bytes) 000596 1763343353.019076 mapread 0x000519 thru 0x00ec21 (0xe709 bytes) 000600 1763343353.022434 write 0x0391ee thru 0x03ffff (0x6e12 bytes) 000604 1763343353.025740 trunc from 0x040000 to 0x016024 (0x29fdb bytes) 000606 1763343353.047827 mapwrite 0x037759 thru 0x03ffff (0x88a7 bytes) 000607 1763343353.177976 mapread 0x03e2ec thru 0x03ffff (0x1d14 bytes) 000608 1763343353.178848 mapread 0x01e287 thru 0x02ac66 (0xc9e0 bytes) 000610 1763343353.256502 mapread 0x021cb7 thru 0x0267c1 (0x4b0b bytes) 000611 1763343353.257350 read 0x006e95 thru 0x00e858 (0x79c4 bytes) 000616 1763343353.259826 read 0x02179f thru 0x0262a3 (0x4b05 bytes) 000617 1763343353.261106 mapread 0x03baf5 thru 0x03ffff (0x450b bytes) 000622 1763343353.271122 MIRROR_RESYNC 5 mirrors 000624 1763343354.132109 MIRROR_RESYNC 5 mirrors 000626 1763343354.565987 mapread 0x02c37b thru 0x038b0e (0xc794 bytes) 000628 1763343354.598261 mapread 0x028e0d thru 0x02ad00 (0x1ef4 bytes) 000629 1763343354.600335 mapread 0x013681 thru 0x019fbc (0x693c bytes) 000631 1763343354.616266 read 0x00bf3d thru 0x01b1e6 (0xf2aa bytes) 000633 1763343354.621212 mapwrite 0x02e3ac thru 0x033205 (0x4e5a bytes) 000635 1763343354.772904 trunc from 0x040000 to 0x01efb5 (0x2104a bytes) 000638 1763343354.830865 trunc from 0x01efb6 to 0x013b2e (0xb487 bytes) 000639 1763343354.852172 mapread 0x0062f3 thru 0x0075bb (0x12c9 bytes) 000640 1763343354.868771 MIRROR_RESYNC 5 mirrors 000641 1763343355.916918 write 0x0269ff thru 0x0345d7 (0xdbd9 bytes) 000642 1763343356.007897 trunc from 0x0345d8 to 0x0388ef (0x4318 bytes) 000643 1763343356.041171 read 0x018e21 thru 0x01ac2c (0x1e0c bytes) 000645 1763343356.050645 trunc from 0x0388f0 to 0x02be09 (0xcae6 bytes) 000646 1763343356.074104 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2be0a): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be0a): Mirror 7: 0xebd043e6 (stale) Mirror 10: 0xebd043e6 (stale) Mirror 11: 0xebd043e6 (stale) Mirror 12: 0xebd043e6 (stale) CRC-32 checksum value for chunk [0, 0x2be0a): Mirror 1: 0x6a56e549 Mirror 13: 0x6a56e549 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be0a: skipped 000647 1763343356.785605 MIRROR_RESYNC 6 mirrors 000648 1763343357.778652 read 0x02bcd4 thru 0x02be09 (0x136 bytes) 000649 1763343357.783842 read 0x0248a5 thru 0x02be09 (0x7565 bytes) 000651 1763343357.801845 mapread 0x016bfe thru 0x016f99 (0x39c bytes) 000652 1763343357.802627 mapread 0x0049d0 thru 0x00fb6f (0xb1a0 bytes) 000653 1763343357.803348 trunc from 0x02be0a to 0x018e95 (0x12f74 bytes) 000654 1763343357.888221 mapread 0x008563 thru 0x01667a (0xe118 bytes) 000656 1763343357.975366 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18e96): Mirror 10: 0x12506eb6 (stale) Mirror 11: 0x12506eb6 (stale) Mirror 12: 0x12506eb6 (stale) Mirror 13: 0x12506eb6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18e96): Mirror 10: 0x12506eb6 (stale) Mirror 11: 0x12506eb6 (stale) Mirror 12: 0x12506eb6 (stale) Mirror 13: 0x12506eb6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18e96: skipped 000657 1763343358.329337 mapwrite 0x026c41 thru 0x03259a (0xb95a bytes) 000659 1763343358.441590 mapwrite 0x005ec4 thru 0x01479c (0xe8d9 bytes) 000660 1763343358.529571 write 0x01f446 thru 0x02c244 (0xcdff bytes) 000661 1763343358.554392 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3259b): Mirror 11: 0x68b5e2e6 (stale) Mirror 12: 0x68b5e2e6 (stale) Mirror 13: 0x68b5e2e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3259b): Mirror 11: 0x68b5e2e6 (stale) Mirror 12: 0x68b5e2e6 (stale) Mirror 13: 0x68b5e2e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3259b: skipped 000662 1763343358.940947 mapwrite 0x024336 thru 0x02dace (0x9799 bytes) 000665 1763343359.010575 trunc from 0x03259b to 0x014527 (0x1e073 bytes) 000668 1763343359.030819 mapread 0x00feb6 thru 0x014527 (0x4672 bytes) 000670 1763343359.042572 mapwrite 0x012c0e thru 0x0145ce (0x19c1 bytes) 000673 1763343359.125835 read 0x011c8d thru 0x0145ce (0x2942 bytes) 000674 1763343359.126594 mapwrite 0x03f493 thru 0x03ffff (0xb6d bytes) 000677 1763343359.240196 trunc from 0x040000 to 0x021d69 (0x1e296 bytes) 000678 1763343359.262346 trunc from 0x021d6a to 0x007f9d (0x19dcc bytes) 000680 1763343359.292761 write 0x000f42 thru 0x007f23 (0x6fe2 bytes) 000681 1763343359.307869 mapread 0x006a67 thru 0x007f9d (0x1537 bytes) 000683 1763343359.309237 trunc from 0x007f9e to 0x01930b (0x1136e bytes) 000684 1763343359.331093 read 0x00828b thru 0x018108 (0xfe7e bytes) 000686 1763343359.342153 trunc from 0x01930c to 0x039bd8 (0x208cd bytes) 000687 1763343359.363672 trunc from 0x039bd9 to 0x024a57 (0x15181 bytes) 000691 1763343359.384681 mapwrite 0x02f569 thru 0x03941e (0x9eb6 bytes) 000692 1763343359.481978 mapwrite 0x02b523 thru 0x036f1d (0xb9fb bytes) 000693 1763343359.553681 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3941f): Mirror 12: 0x5beb9a7 (stale) Mirror 13: 0x5beb9a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3941f): Mirror 12: 0x5beb9a7 (stale) Mirror 13: 0x5beb9a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3941f: skipped 000694 1763343359.915270 mapread 0x003bca thru 0x00e1e3 (0xa61a bytes) 000696 1763343359.949808 trunc from 0x03941f to 0x028750 (0x10cce bytes) 000700 1763343359.972782 mapwrite 0x00995e thru 0x00a4bc (0xb5f bytes) 000702 1763343360.022623 mapwrite 0x02d321 thru 0x033af5 (0x67d5 bytes) 000703 1763343360.121636 trunc from 0x033af6 to 0x03ddd0 (0xa2db bytes) 000706 1763343360.137630 mapwrite 0x001a3b thru 0x00d44b (0xba11 bytes) 000707 1763343360.208312 trunc from 0x03ddd1 to 0x03e589 (0x7b9 bytes) 000711 1763343360.237172 trunc from 0x03e58a to 0x015ac5 (0x28ac4 bytes) 000712 1763343360.258189 write 0x03ae0f thru 0x03fe1d (0x500f bytes) 000715 1763343360.261803 read 0x00ec43 thru 0x0101d2 (0x1590 bytes) 000719 1763343360.276506 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fe1e): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe1e): Mirror 12: 0x2226f86e (stale) Mirror 13: 0x2226f86e (stale) CRC-32 checksum value for chunk [0, 0x3fe1e): Mirror 1: 0x3a63d706 Mirror 14: 0x3a63d706 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe1e: skipped 000720 1763343361.197738 trunc from 0x03fe1e to 0x00bd34 (0x340e9 bytes) 000721 1763343361.294967 mapread 0x006a80 thru 0x00bd34 (0x52b5 bytes) 000722 1763343361.341042 write 0x02153d thru 0x0303ca (0xee8e bytes) 000724 1763343361.343465 mapwrite 0x01e28c thru 0x0232a6 (0x501b bytes) 000725 1763343361.422046 mapread 0x01eec3 thru 0x02cfd9 (0xe117 bytes) 000728 1763343361.424089 write 0x010e53 thru 0x01f907 (0xeab5 bytes) 000731 1763343361.435727 mapread 0x0251f6 thru 0x0303ca (0xb1d5 bytes) 000733 1763343361.437544 read 0x0186e5 thru 0x025651 (0xcf6d bytes) 000734 1763343361.446550 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x303cb): Mirror 12: 0x5bd9b821 (stale) Mirror 14: 0xcab86297 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303cb): Mirror 12: 0x5bd9b821 (stale) Mirror 14: 0xcab86297 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303cb: skipped 000735 1763343361.846605 read 0x0088ca thru 0x0121ac (0x98e3 bytes) 000740 1763343361.860380 trunc from 0x0303cb to 0x03a7d8 (0xa40e bytes) 000742 1763343361.877245 mapread 0x016732 thru 0x01b1a5 (0x4a74 bytes) 000745 1763343361.900073 read 0x02afac thru 0x038028 (0xd07d bytes) 000747 1763343361.913270 write 0x02bde2 thru 0x039998 (0xdbb7 bytes) 000748 1763343361.914607 mapread 0x02f962 thru 0x036e90 (0x752f bytes) 000749 1763343361.915472 trunc from 0x03a7d9 to 0x00683a (0x33f9e bytes) 000750 1763343361.943250 mapwrite 0x0164b7 thru 0x01834d (0x1e97 bytes) 000752 1763343362.068354 mapwrite 0x02a532 thru 0x035a1e (0xb4ed bytes) 000753 1763343362.281302 mapread 0x022b6a thru 0x02b5ef (0x8a86 bytes) 000754 1763343362.332557 trunc from 0x035a1f to 0x03e240 (0x8822 bytes) 000755 1763343362.360890 MIRROR_RESYNC 3 mirrors 000758 1763343363.236713 trunc from 0x03e241 to 0x015689 (0x28bb7 bytes) 000760 1763343363.336661 mapwrite 0x01a5de thru 0x0239d2 (0x93f5 bytes) 000761 1763343363.442855 mapwrite 0x00f7ea thru 0x01d7ae (0xdfc5 bytes) 000764 1763343363.513218 trunc from 0x0239d3 to 0x031635 (0xdc63 bytes) 000768 1763343363.543135 write 0x02c4dd thru 0x038365 (0xbe89 bytes) 000769 1763343363.564131 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38366): Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38366): Mirror 12: 0x458adacb (stale) Mirror 14: 0x458adacb (stale) CRC-32 checksum value for chunk [0, 0x38366): Mirror 1: 0xcb078ca9 Mirror 15: 0xcb078ca9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38366: skipped 000770 1763343364.479108 trunc from 0x038366 to 0x0214f4 (0x16e71 bytes) 000771 1763343364.587499 write 0x0088f5 thru 0x00ab0d (0x2219 bytes) 000772 1763343364.621542 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x214f5): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214f5): Mirror 12: 0xa1bdef41 (stale) Mirror 14: 0xa1bdef41 (stale) Mirror 15: 0x789aa6a6 (stale) CRC-32 checksum value for chunk [0, 0x214f5): Mirror 1: 0x347daec7 Mirror 16: 0x347daec7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214f5: skipped 000775 1763343365.552120 mapwrite 0x0122c3 thru 0x0219ad (0xf6eb bytes) 000776 1763343365.721731 write 0x02ff86 thru 0x03b113 (0xb18e bytes) 000777 1763343365.727575 mapwrite 0x00700e thru 0x009626 (0x2619 bytes) 000778 1763343365.795929 mapread 0x03643e thru 0x03a7b2 (0x4375 bytes) 000779 1763343365.797160 mapwrite 0x009042 thru 0x01192e (0x88ed bytes) 000782 1763343365.848235 read 0x036ee9 thru 0x037ae0 (0xbf8 bytes) 000783 1763343365.848915 write 0x013cd5 thru 0x01cb3c (0x8e68 bytes) 000784 1763343365.850468 write 0x008d7b thru 0x012630 (0x98b6 bytes) 000786 1763343365.857950 MIRROR_RESYNC 5 mirrors 000787 1763343367.002752 mapread 0x00757a thru 0x009d65 (0x27ec bytes) 000788 1763343367.036688 mapread 0x02a20d thru 0x02e8f5 (0x46e9 bytes) 000789 1763343367.050751 write 0x00c3bb thru 0x01057e (0x41c4 bytes) 000791 1763343367.151268 write 0x01183a thru 0x0152c3 (0x3a8a bytes) 000792 1763343367.163070 mapread 0x025551 thru 0x02f2c0 (0x9d70 bytes) 000794 1763343367.202374 MIRROR_RESYNC 5 mirrors 000795 1763343368.122409 MIRROR_EXTEND to 6 mirrors 000798 1763343368.881651 mapwrite 0x01c45f thru 0x01fa05 (0x35a7 bytes) 000800 1763343369.098773 MIRROR_RESYNC 6 mirrors 000802 1763343370.098191 mapread 0x00c67c thru 0x0178dd (0xb262 bytes) 000804 1763343370.123198 mapread 0x00bf11 thru 0x0140fc (0x81ec bytes) 000805 1763343370.124624 mapwrite 0x0010cd thru 0x00970a (0x863e bytes) 000806 1763343370.303719 MIRROR_RESYNC 6 mirrors 000812 1763343371.266605 mapread 0x001224 thru 0x00123b (0x18 bytes) 000816 1763343371.277218 write 0x02ad43 thru 0x0361fe (0xb4bc bytes) 000818 1763343371.370853 mapread 0x035a43 thru 0x03a9d2 (0x4f90 bytes) 000820 1763343371.379068 write 0x0284b4 thru 0x02e373 (0x5ec0 bytes) 000823 1763343371.394168 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: 0x324d0b13 (stale) Mirror 14: 0x324d0b13 (stale) Mirror 16: 0x324d0b13 (stale) Mirror 17: 0x324d0b13 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: 0x324d0b13 (stale) Mirror 14: 0x324d0b13 (stale) Mirror 16: 0x324d0b13 (stale) Mirror 17: 0x324d0b13 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b114: skipped 000824 1763343371.734813 write 0x02baac thru 0x032f00 (0x7455 bytes) 000827 1763343371.755895 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: 0x324d0b13 (stale) Mirror 14: 0x324d0b13 (stale) Mirror 16: 0x324d0b13 (stale) Mirror 17: 0x324d0b13 (stale) CRC-32 checksum value for chunk [0, 0x3b114): Mirror 1: 0xb838402e Mirror 18: 0xb838402e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b114: skipped 000832 1763343372.646877 mapread 0x0001ca thru 0x00dd0e (0xdb45 bytes) 000833 1763343372.683684 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: 0x324d0b13 (stale) Mirror 14: 0x324d0b13 (stale) Mirror 16: 0x324d0b13 (stale) Mirror 17: 0x324d0b13 (stale) CRC-32 checksum value for chunk [0, 0x3b114): Mirror 1: 0xb838402e Mirror 18: 0xb838402e Mirror 19: 0xb838402e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b114: skipped 000835 1763343373.624455 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: 0x324d0b13 (stale) Mirror 14: 0x324d0b13 (stale) Mirror 16: 0x324d0b13 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b114): Mirror 12: 0x324d0b13 (stale) Mirror 14: 0x324d0b13 (stale) Mirror 16: 0x324d0b13 (stale) CRC-32 checksum value for chunk [0, 0x3b114): Mirror 1: 0xb838402e Mirror 18: 0xb838402e Mirror 19: 0xb838402e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b114: skipped 000838 1763343374.317626 write 0x02f66d thru 0x030e33 (0x17c7 bytes) 000842 1763343374.424457 mapwrite 0x02f1be thru 0x02f354 (0x197 bytes) 000843 1763343374.490828 trunc from 0x03b114 to 0x025a1a (0x156f9 bytes) 000846 1763343374.535072 mapread 0x009a00 thru 0x00b728 (0x1d29 bytes) 000850 1763343374.571635 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x25a1b): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a1b): Mirror 12: 0x1988f526 (stale) Mirror 14: 0x1988f526 (stale) Mirror 16: 0x1988f526 (stale) Mirror 18: 0x1988f526 (stale) Mirror 19: 0x1988f526 (stale) CRC-32 checksum value for chunk [0, 0x25a1b): Mirror 1: 0x1988f526 Mirror 20: 0x1988f526 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a1b: skipped 000852 1763343375.295674 write 0x032c8c thru 0x034884 (0x1bf9 bytes) 000854 1763343375.386918 write 0x028d74 thru 0x02da10 (0x4c9d bytes) 000855 1763343375.397684 write 0x02e141 thru 0x038efd (0xadbd bytes) 000856 1763343375.409417 read 0x011909 thru 0x0166e1 (0x4dd9 bytes) 000857 1763343375.418013 mapwrite 0x0289f6 thru 0x0370f5 (0xe700 bytes) 000858 1763343375.500521 mapwrite 0x00e570 thru 0x018929 (0xa3ba bytes) 000860 1763343375.608309 mapread 0x009345 thru 0x0157ef (0xc4ab bytes) 000861 1763343375.609251 trunc from 0x038efe to 0x01c664 (0x1c899 bytes) 000862 1763343375.667872 mapread 0x008de8 thru 0x013686 (0xa89f bytes) 000863 1763343375.668849 trunc from 0x01c665 to 0x000796 (0x1bece bytes) 000865 1763343375.685962 trunc from 0x000797 to 0x01aaee (0x1a358 bytes) 000866 1763343375.702180 read 0x010317 thru 0x01aaee (0xa7d8 bytes) 000868 1763343375.721222 read 0x009f69 thru 0x017f18 (0xdfb0 bytes) 000869 1763343375.730362 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1aaef): Mirror 12: 0x9c92b439 (stale) Mirror 14: 0x9c92b439 (stale) Mirror 18: 0x9c92b439 (stale) Mirror 19: 0x9c92b439 (stale) Mirror 20: 0x9c92b439 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aaef): Mirror 12: 0x9c92b439 (stale) Mirror 14: 0x9c92b439 (stale) Mirror 18: 0x9c92b439 (stale) Mirror 19: 0x9c92b439 (stale) Mirror 20: 0x9c92b439 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aaef: skipped 000873 1763343376.168046 trunc from 0x01aaef to 0x03e00b (0x2351d bytes) 000876 1763343376.212080 trunc from 0x03e00c to 0x02a7f6 (0x13815 bytes) 000877 1763343376.232534 read 0x009a60 thru 0x016776 (0xcd17 bytes) 000879 1763343376.246219 mapread 0x0037ad thru 0x00e6c4 (0xaf18 bytes) 000881 1763343376.264931 read 0x011412 thru 0x013485 (0x2074 bytes) 000884 1763343376.266347 trunc from 0x02a7f7 to 0x004ec4 (0x25932 bytes) 000885 1763343376.283902 write 0x001987 thru 0x007147 (0x57c1 bytes) 000886 1763343376.285058 trunc from 0x007148 to 0x027588 (0x20441 bytes) 000887 1763343376.322176 mapwrite 0x026194 thru 0x02aa48 (0x48b5 bytes) 000888 1763343376.449785 mapread 0x01a97e thru 0x01be59 (0x14dc bytes) 000891 1763343376.463407 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aa49): Mirror 12: 0xc9d3f638 (stale) Mirror 14: 0xc9d3f638 (stale) Mirror 19: 0x7211ff6d (stale) Mirror 20: 0xc9d3f638 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa49): Mirror 12: 0xc9d3f638 (stale) Mirror 14: 0xc9d3f638 (stale) Mirror 19: 0x7211ff6d (stale) Mirror 20: 0xc9d3f638 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa49: skipped 000892 1763343376.903115 write 0x030e8e thru 0x036434 (0x55a7 bytes) 000894 1763343376.904905 mapread 0x032afd thru 0x036434 (0x3938 bytes) 000895 1763343376.906952 trunc from 0x036435 to 0x013297 (0x2319d bytes) 000896 1763343376.928844 read 0x005c20 thru 0x012b7e (0xcf5f bytes) 000897 1763343376.939120 trunc from 0x013298 to 0x031af9 (0x1e862 bytes) 000898 1763343376.966502 read 0x02728f thru 0x030847 (0x95b9 bytes) 000899 1763343376.989776 trunc from 0x031afa to 0x01d037 (0x14ac2 bytes) 000900 1763343377.011719 write 0x0345ec thru 0x03c34d (0x7d62 bytes) 000901 1763343377.015819 mapread 0x02c1de thru 0x03278c (0x65af bytes) 000905 1763343377.062003 write 0x029462 thru 0x02c9bb (0x355a bytes) 000908 1763343377.077163 mapread 0x00607f thru 0x00b37e (0x5300 bytes) 000909 1763343377.078118 read 0x005d9f thru 0x00a3cd (0x462f bytes) 000910 1763343377.078785 read 0x004439 thru 0x00d398 (0x8f60 bytes) 000911 1763343377.089506 MIRROR_RESYNC 5 mirrors 000912 1763343378.047265 trunc from 0x03c34e to 0x03a784 (0x1bc9 bytes) 000913 1763343378.233968 write 0x0241c5 thru 0x032705 (0xe541 bytes) 000915 1763343378.248210 mapwrite 0x00ac89 thru 0x012865 (0x7bdd bytes) 000916 1763343378.327362 write 0x02c04f thru 0x02e5c7 (0x2579 bytes) 000917 1763343378.328253 write 0x02d404 thru 0x03a6f8 (0xd2f5 bytes) 000919 1763343378.336904 read 0x00dea5 thru 0x0177bb (0x9917 bytes) 000920 1763343378.347961 trunc from 0x03a785 to 0x00f373 (0x2b411 bytes) 000921 1763343378.390146 write 0x008c7f thru 0x014f49 (0xc2cb bytes) 000922 1763343378.391589 mapread 0x001146 thru 0x00a5ba (0x9475 bytes) 000923 1763343378.408131 mapread 0x00db01 thru 0x0105d1 (0x2ad1 bytes) 000924 1763343378.414014 write 0x032bbc thru 0x03ffff (0xd444 bytes) 000925 1763343378.421941 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ee:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf7702960 (stale) Mirror 19: 0xf7702960 (stale) Mirror 20: 0xf7702960 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf7702960 (stale) Mirror 19: 0xf7702960 (stale) Mirror 20: 0xf7702960 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000927 1763343378.828146 write 0x006a2d thru 0x011a95 (0xb069 bytes) 000931 1763343378.845585 mapread 0x03e0c4 thru 0x03ffff (0x1f3c bytes) 000934 1763343378.856162 mapread 0x01f860 thru 0x029757 (0x9ef8 bytes) 000936 1763343378.872797 mapread 0x010219 thru 0x014b6e (0x4956 bytes) 000937 1763343378.881975 read 0x00af03 thru 0x01369d (0x879b bytes) 000938 1763343378.883664 read 0x01c02d thru 0x02a934 (0xe908 bytes) 000942 1763343378.890596 trunc from 0x040000 to 0x0120ea (0x2df15 bytes) 000943 1763343378.921619 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x120eb): Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x120eb): Mirror 12: 0xe476c7c1 (stale) Mirror 19: 0xe476c7c1 (stale) Mirror 20: 0xe476c7c1 (stale) CRC-32 checksum value for chunk [0, 0x120eb): Mirror 1: 0xd10090bd Mirror 21: 0xd10090bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x120eb: skipped 000944 1763343379.724337 write 0x019033 thru 0x02767f (0xe64d bytes) 000945 1763343379.821678 write 0x00ef3e thru 0x014e2e (0x5ef1 bytes) 000946 1763343379.831625 write 0x03a680 thru 0x03ffff (0x5980 bytes) 000948 1763343379.833464 write 0x025884 thru 0x033251 (0xd9ce bytes) 000949 1763343379.839969 mapwrite 0x03b856 thru 0x03d6f0 (0x1e9b bytes) 000950 1763343379.967803 mapwrite 0x036415 thru 0x03ffff (0x9beb bytes) 000953 1763343380.071094 read 0x02e839 thru 0x0352d8 (0x6aa0 bytes) 000955 1763343380.072760 mapread 0x030906 thru 0x038fda (0x86d5 bytes) 000957 1763343380.080185 MIRROR_RESYNC 5 mirrors 000960 1763343381.067488 mapread 0x001042 thru 0x007bd5 (0x6b94 bytes) 000962 1763343381.087428 write 0x01d971 thru 0x0279b7 (0xa047 bytes) 000963 1763343381.170448 mapwrite 0x02cb71 thru 0x032ef0 (0x6380 bytes) 000964 1763343381.239338 mapwrite 0x00986e thru 0x018bc0 (0xf353 bytes) 000965 1763343381.297453 mapwrite 0x01dbaa thru 0x02cfbc (0xf413 bytes) 000966 1763343381.354203 write 0x028a16 thru 0x02e391 (0x597c bytes) 000969 1763343381.357266 mapwrite 0x03048e thru 0x03911b (0x8c8e bytes) 000971 1763343381.424460 read 0x00e71e thru 0x01d426 (0xed09 bytes) 000973 1763343381.425541 read 0x037914 thru 0x037c6b (0x358 bytes) 000975 1763343381.431191 MIRROR_RESYNC 5 mirrors 000976 1763343382.484006 read 0x015424 thru 0x01e9f2 (0x95cf bytes) 000977 1763343382.502929 mapread 0x030163 thru 0x038c2d (0x8acb bytes) 000980 1763343382.528595 write 0x031872 thru 0x0370c7 (0x5856 bytes) 000981 1763343382.586215 write 0x0185a5 thru 0x01fe09 (0x7865 bytes) 000983 1763343382.612579 MIRROR_RESYNC 5 mirrors 000984 1763343383.556186 write 0x0045a5 thru 0x011a63 (0xd4bf bytes) 000985 1763343383.646544 mapwrite 0x00b2ba thru 0x010e87 (0x5bce bytes) 000990 1763343383.699124 trunc from 0x040000 to 0x01c2ce (0x23d31 bytes) 000992 1763343383.741013 read 0x013f3c thru 0x0143d2 (0x497 bytes) 000994 1763343383.747484 mapwrite 0x01e929 thru 0x0239ef (0x50c7 bytes) 000996 1763343383.822547 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x239f0): Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x239f0): Mirror 12: 0x17981dd2 (stale) Mirror 19: 0x17981dd2 (stale) Mirror 20: 0x17981dd2 (stale) Mirror 21: 0x17981dd2 (stale) CRC-32 checksum value for chunk [0, 0x239f0): Mirror 1: 0x113eb363 Mirror 22: 0x113eb363 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x239f0: skipped 000997 1763343384.806364 mapwrite 0x02520f thru 0x02fce6 (0xaad8 bytes) 001000 1763343385.002818 read 0x02c2c5 thru 0x02fce6 (0x3a22 bytes) All operations completed A-OK! PASS 70a (97s) == sanity-flr test 71: check mirror extend parent fid ==== 20:36:29 (1763343389) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 200a: stress test ===================== 20:36:33 (1763343393) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4713856 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7445472 .. Extending file size to 7971872 .. Extending file size to 8177248 .. Extending file size to 8137920 .. failed Extending file size to 8986752 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8668352 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9268704 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 54912 54913 54917 54918 54919 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) 192.168.201.157@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (74s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:37:48 (1763343468) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3901408 .. Extending file size to 8419392 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5770528 .. Extending file size to 7894112 .. Extending file size to 8087904 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8556768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9148128 .. Extending file size to 8921408 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56817 56818 56820 56821 56824 done 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) 192.168.201.157@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (72s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:39:00 (1763343540) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100499 s, 20.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x50e9c09d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:39:15 (1763343555) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104509 s, 20.1 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 20:39:23 (1763343563) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x239:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:39:29 (1763343569) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171599 s, 12.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:39:37 (1763343577) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235436 s, 8.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:39:44 (1763343584) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265924 s, 7.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:40:15 (1763343615) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg157-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13312 s, 7.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:41:11 (1763343671) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg157-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.180779 s, 11.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:42:12 (1763343732) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg157-server mds-ost sync done. Creating new pool test_204f oleg157-server: Pool lustre.test_204f created Adding targets to pool oleg157-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg157-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165975 s, 12.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg157-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg157-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg157-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:43:26 (1763343806) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:43:34 (1763343814) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:43:41 (1763343821) oleg157-server: Pool lustre.test_206 created Adding targets to pool oleg157-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg157-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg157-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg157-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg157-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:44:05 (1763343845) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12682 s, 8.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670658 s, 15.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x71a0e63a Mirror 3: 0x71a0e63a PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:44:17 (1763343857) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:44:21 (1763343861) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:44:24 (1763343864) Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.111528 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ef0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ef0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ef0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd30000 Read this (4 bytes): BBBB 192.168.201.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg157-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:44:57 (1763343897) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0316825 s, 259 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:45:03 (1763343903) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443048 s, 23.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x25e lmm_fid: [0x200000404:0x25e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x25e lmm_fid: [0x200000404:0x25e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834526 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 20:45:05.000000000 -0500 Modify: 2025-11-16 20:45:05.000000000 -0500 Change: 2025-11-16 20:45:06.000000000 -0500 Birth: 2025-11-16 20:45:05.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:45:11 (1763343911) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:45:14 (1763343914) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.610909 s, 67.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0729799 s, 56.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 16 20:45 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x260:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 16 20:45 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:45:21 (1763343921) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:45:29 (1763343929) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1474 sec =========== 20:45:37 (1763343937) === sanity-flr: start cleanup 20:45:39 (1763343939) === === sanity-flr: finish cleanup 20:45:46 (1763343946) ===