-----============= acceptance-small: sanity-flr ============----- Fri May 8 20:29:07 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg428-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:29:26 (1778286566) === oleg428-client.virtnet: executing check_config_client /mnt/lustre oleg428-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg428-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88f38929e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88f38929e800.idle_timeout=debug disable quota as required oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:29:59 (1778286599) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7553] == sanity-flr test 0a: lfs mirror create with -N option == 20:30:01 (1778286601) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:30:18 (1778286618) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:30:22 (1778286622) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:30:26 (1778286626) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:30:50 (1778286650) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:30:53 (1778286653) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:30:56 (1778286656) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88f38929e800.stats=clear osc.lustre-OST0001-osc-ffff88f38929e800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:31:03 (1778286663) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:31:13 (1778286673) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:31:23 (1778286683) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:31:26 (1778286686) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:31:33 (1778286693) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 20:31:43 (1778286703) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:31:50 (1778286710) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:31:53 (1778286713) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:32:00 (1778286720) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:32:08 (1778286728) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:32:14 (1778286734) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.885938 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996224 s, 10.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:32:25 (1778286745) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.288405 s, 43.6 MB/s PASS 22 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:32:34 (1778286754) Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (63s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:33:37 (1778286817) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.57804 s, 20.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:34:15 (1778286855) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:35:27 (1778286927) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.26703 s, 12.3 MB/s -rw-r--r-- 1 root root 52428800 May 8 20:35 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.105s user 0m0.001s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88f38929e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.233s user 0m0.000s sys 0m0.164s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg428-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg428-server oleg428-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.434s user 0m0.000s sys 0m0.204s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg428-server: oleg428-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg428-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88f38929e800.ost_server_uuid in FULL state after 0 sec PASS 33b (86s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:36:52 (1778287012) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:36:55 (1778287015) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:36:59 (1778287019) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:37:01 (1778287021) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 20:37:08 (1778287028) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.98497 s, 12.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (19s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:37:27 (1778287047) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713631 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395867 s, 26.5 MB/s PASS 36b (12s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:37:39 (1778287059) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:37:50 (1778287070) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00355422 s, 288 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0198994 s, 51.5 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0110586 s, 92.6 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00203816 s, 502 kB/s flv=10 olv=10 PASS 36d (17s) == sanity-flr test 37: mirror I/O API verification ======= 20:38:07 (1778287087) create test files with size 15742757 .. 8e952e8518bb289f0e42ea41918d7f81 - cb7b6b301391171938f0bba7902fec90 - 8a9c4b9c6471af51d149204d57ec3524 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 20:38:29 (1778287109) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:38:43 (1778287123) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:38:49 (1778287129) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124668 s, 8.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x50:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371505 s, 28.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x51:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 20:38:57 (1778287137) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240793 s, 8.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.358348 s, 11.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.329897 s, 6.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 20:39:14 (1778287154) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:39:17 (1778287157) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:39:20 (1778287160) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308662 s, 34.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 20:39:28 (1778287168) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:39:31 (1778287171) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x55:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:39:40 (1778287180) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.240214 s, 43.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:39:51 (1778287191) llite.lustre-ffff88f382df8000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296896 s, 35.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg428-client.virtnet /mnt/lustre (opts:) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff88f385e9d000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:40:05 (1778287205) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:40:49 (1778287249) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:41:23 (1778287283) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:41:26 (1778287286) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.28319 s, 11.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.13528 s, 23.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd7dc10e6 (stale) Mirror 2: 0xd7dc10e6 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 96c79535c05e53a473dd558c71bbe2de - mirror 1 checksum: baae903da52a84c4004e6c68c3060315 - mirror 2 checksum: 96c79535c05e53a473dd558c71bbe2de - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 8 20:41:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 8 20:41:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 96c79535c05e53a473dd558c71bbe2de - mirror 1 checksum: baae903da52a84c4004e6c68c3060315 - mirror 2 checksum: baae903da52a84c4004e6c68c3060315 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 8 20:41:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 8 20:41:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6c:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:41:38 (1778287298) Failing mds1 on oleg428-server Stopping /mnt/lustre-mds1 (opts:) on oleg428-server 20:41:44 (1778287304) shut down facet: mds1 facet_host: oleg428-server facet_failover_host: oleg428-server Failover mds1 to oleg428-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 20:42:07 (1778287327) targets are mounted 20:42:07 (1778287327) facet_failover done oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032308 s, 32.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6d:0x0] } PASS 50A (43s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:42:21 (1778287341) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:42:24 (1778287344) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:42:28 (1778287348) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0126576 s, 1.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:42:35 (1778287355) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:42:38 (1778287358) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:42:41 (1778287361) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:43:06 (1778287386) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:43:34 (1778287414) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:43:57 (1778287437) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:44:23 (1778287463) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 20:44:29 (1778287469) mirror create pid 52873 mirror split pid 52874 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52873 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52874 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 20:45:37 (1778287537) Seed set to 7539 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005 1778287539.418541 mapwrite 0x01004c thru 0x0140cf (0x4084 bytes) 000007 1778287539.521083 mapread 0x004417 thru 0x005149 (0xd33 bytes) 000008 1778287539.533110 write 0x02d286 thru 0x0336aa (0x6425 bytes) 000009 1778287539.534767 read 0x0134b2 thru 0x0228bf (0xf40e bytes) 000010 1778287539.545745 mapread 0x00692f thru 0x00f69e (0x8d70 bytes) 000012 1778287539.548297 mapwrite 0x02f8fc thru 0x03b75a (0xbe5f bytes) truncating to largest ever: 0x1c41 000015 1778287539.712350 trunc from 0x03b75b to 0x001c40 (0x39b1a bytes) 000020 1778287539.770185 mapwrite 0x03f9e3 thru 0x03ffff (0x61d bytes) 000021 1778287539.892519 read 0x0071a8 thru 0x011bbe (0xaa17 bytes) 000022 1778287539.966447 MIRROR_EXTEND to 2 mirrors 000023 1778287540.969038 mapwrite 0x034a45 thru 0x0353ec (0x9a8 bytes) truncating to largest ever: 0x13a35 000024 1778287541.257157 trunc from 0x040000 to 0x013a34 (0x2c5cb bytes) 000026 1778287541.306974 read 0x002e8c thru 0x0095ef (0x6764 bytes) 000027 1778287541.328305 mapread 0x000542 thru 0x00db79 (0xd638 bytes) 000028 1778287541.352185 read 0x00d450 thru 0x013a34 (0x65e5 bytes) 000029 1778287541.367287 mapread 0x010d36 thru 0x013a34 (0x2cff bytes) 000030 1778287541.371781 mapwrite 0x023ae0 thru 0x02bdd2 (0x82f3 bytes) 000031 1778287541.547185 mapread 0x021f17 thru 0x026cf0 (0x4dda bytes) 000033 1778287541.550391 read 0x017f02 thru 0x0276a5 (0xf7a4 bytes) 000037 1778287541.555674 write 0x015df6 thru 0x02265c (0xc867 bytes) 000038 1778287541.558791 write 0x039aa6 thru 0x03e45e (0x49b9 bytes) 000041 1778287541.562399 write 0x00dd7f thru 0x010a60 (0x2ce2 bytes) 000042 1778287541.564645 trunc from 0x03e45f to 0x00e70f (0x2fd4f bytes) 000045 1778287541.589211 write 0x003aba thru 0x00edd0 (0xb317 bytes) 000046 1778287541.590896 write 0x005878 thru 0x006195 (0x91e bytes) 000048 1778287541.592698 write 0x0071b8 thru 0x012810 (0xb659 bytes) 000050 1778287541.605818 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x12811): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12811): Mirror 2: 0xcbf50e09 (stale) CRC-32 checksum value for chunk [0, 0x12811): Mirror 1: 0x5b29c9f7 Mirror 3: 0x5b29c9f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12811: skipped 000051 1778287542.516790 mapread 0x00820f thru 0x0091ab (0xf9d bytes) 000052 1778287542.534307 read 0x00b1cf thru 0x012810 (0x7642 bytes) 000055 1778287542.557825 mapread 0x00ea85 thru 0x012810 (0x3d8c bytes) 000056 1778287542.567900 read 0x00b3cc thru 0x012810 (0x7445 bytes) 000057 1778287542.572784 mapwrite 0x0126e4 thru 0x0195ab (0x6ec8 bytes) 000059 1778287542.776007 write 0x01364c thru 0x019ac9 (0x647e bytes) 000060 1778287542.778689 read 0x00940c thru 0x00f303 (0x5ef8 bytes) 000061 1778287542.780060 mapread 0x003b7f thru 0x0114cd (0xd94f bytes) 000063 1778287542.790762 MIRROR_RESYNC 3 mirrors 000064 1778287543.868333 read 0x012464 thru 0x019a40 (0x75dd bytes) 000065 1778287543.903687 MIRROR_RESYNC 3 mirrors 000066 1778287544.485716 read 0x0049fa thru 0x010f49 (0xc550 bytes) 000067 1778287544.517243 trunc from 0x019aca to 0x00acbb (0xee0e bytes) 000068 1778287544.649532 mapread 0x000440 thru 0x000f3c (0xafd bytes) 000071 1778287544.697137 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xacbc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xacbc): Mirror 2: 0xfbc00d7a (stale) Mirror 3: 0xfbc00d7a (stale) CRC-32 checksum value for chunk [0, 0xacbc): Mirror 1: 0xfbc00d7a Mirror 4: 0xfbc00d7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xacbc: skipped 000072 1778287546.554883 MIRROR_RESYNC 4 mirrors 000073 1778287548.115185 MIRROR_EXTEND to 5 mirrors truncating to largest ever: 0x190e7 000074 1778287548.945821 trunc from 0x00acbc to 0x0190e6 (0xe42b bytes) 000076 1778287549.144479 write 0x00c12f thru 0x00fdf2 (0x3cc4 bytes) 000077 1778287549.193060 write 0x015201 thru 0x01a5b1 (0x53b1 bytes) 000080 1778287549.216484 read 0x0022bf thru 0x006763 (0x44a5 bytes) truncating to largest ever: 0x2d9c8 000082 1778287549.226440 trunc from 0x01a5b2 to 0x02d9c7 (0x13416 bytes) 000084 1778287549.288025 mapread 0x001255 thru 0x003829 (0x25d5 bytes) 000086 1778287549.312698 mapwrite 0x022aec thru 0x023c41 (0x1156 bytes) 000087 1778287549.502603 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d9c8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d9c8): Mirror 2: 0xf5bc32d6 (stale) Mirror 3: 0xf5bc32d6 (stale) Mirror 4: 0xf5bc32d6 (stale) Mirror 5: 0xf5bc32d6 (stale) CRC-32 checksum value for chunk [0, 0x2d9c8): Mirror 1: 0x50edb4ff Mirror 6: 0x50edb4ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d9c8: skipped 000089 1778287550.649064 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d9c8): Mirror 3: 0xf5bc32d6 (stale) Mirror 4: 0xf5bc32d6 (stale) Mirror 5: 0xf5bc32d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d9c8): Mirror 3: 0xf5bc32d6 (stale) Mirror 4: 0xf5bc32d6 (stale) Mirror 5: 0xf5bc32d6 (stale) CRC-32 checksum value for chunk [0, 0x2d9c8): Mirror 1: 0x50edb4ff Mirror 6: 0x50edb4ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d9c8: skipped 000090 1778287551.358952 trunc from 0x02d9c8 to 0x022244 (0xb783 bytes) 000091 1778287551.470331 read 0x01c6f8 thru 0x022244 (0x5b4d bytes) 000096 1778287551.497517 mapread 0x0193e6 thru 0x022244 (0x8e5f bytes) 000098 1778287551.528675 trunc from 0x022245 to 0x00f4ad (0x12d97 bytes) 000099 1778287551.574991 mapwrite 0x013ef2 thru 0x019753 (0x5862 bytes) 000101 1778287551.715902 write 0x019af9 thru 0x01f2d1 (0x57d9 bytes) 000102 1778287551.720916 write 0x021627 thru 0x02daf1 (0xc4cb bytes) 000103 1778287551.722735 mapwrite 0x01f73c thru 0x02ef4a (0xf80f bytes) 000104 1778287551.828308 write 0x00d4b1 thru 0x01a5c7 (0xd117 bytes) 000105 1778287551.829696 write 0x033f36 thru 0x034fbb (0x1086 bytes) 000107 1778287551.832950 mapwrite 0x020900 thru 0x028266 (0x7967 bytes) 000110 1778287551.973043 trunc from 0x034fbc to 0x00545a (0x2fb61 bytes) 000111 1778287552.017182 write 0x03f487 thru 0x03ffff (0xb79 bytes) 000112 1778287552.023200 read 0x003b36 thru 0x00e5da (0xaaa5 bytes) 000113 1778287552.045231 read 0x00863c thru 0x00cdc2 (0x4787 bytes) 000114 1778287552.047052 write 0x02e67f thru 0x03a204 (0xbb86 bytes) 000115 1778287552.050467 mapread 0x01d567 thru 0x01f50e (0x1fa8 bytes) 000117 1778287552.059567 mapwrite 0x028e7b thru 0x035051 (0xc1d7 bytes) 000118 1778287552.127192 mapwrite 0x010253 thru 0x01f055 (0xee03 bytes) 000120 1778287552.215789 trunc from 0x040000 to 0x0132f8 (0x2cd07 bytes) truncating to largest ever: 0x3eb90 000121 1778287552.240870 trunc from 0x0132f9 to 0x03eb8f (0x2b897 bytes) 000122 1778287552.278559 mapwrite 0x0381a6 thru 0x03ffff (0x7e5a bytes) 000123 1778287552.455141 mapwrite 0x000a83 thru 0x0104b8 (0xfa36 bytes) 000125 1778287552.544120 read 0x033155 thru 0x03a734 (0x75e0 bytes) 000126 1778287552.546259 mapread 0x0020ff thru 0x01197a (0xf87c bytes) 000130 1778287552.556023 trunc from 0x040000 to 0x0362b7 (0x9d48 bytes) 000131 1778287552.590754 write 0x01cde0 thru 0x021c55 (0x4e76 bytes) 000133 1778287552.607475 mapread 0x021d10 thru 0x022d12 (0x1003 bytes) 000134 1778287552.608681 mapwrite 0x03f774 thru 0x03ffff (0x88c bytes) 000136 1778287552.741576 read 0x029b35 thru 0x032cc4 (0x9190 bytes) 000137 1778287552.742709 write 0x00a247 thru 0x011bbf (0x7979 bytes) 000138 1778287552.753564 mapread 0x016c3e thru 0x01cef0 (0x62b3 bytes) 000139 1778287552.756561 read 0x00db31 thru 0x00f942 (0x1e12 bytes) 000140 1778287552.758525 read 0x009c2b thru 0x00f679 (0x5a4f bytes) 000141 1778287552.760512 write 0x0328fb thru 0x03cbaf (0xa2b5 bytes) 000142 1778287552.765587 mapread 0x000464 thru 0x000f55 (0xaf2 bytes) 000144 1778287552.771563 write 0x02b7f9 thru 0x035feb (0xa7f3 bytes) 000145 1778287552.778599 mapread 0x033bd3 thru 0x03687d (0x2cab bytes) 000146 1778287552.792475 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfbeca587 (stale) Mirror 4: 0xfbeca587 (stale) Mirror 5: 0xfbeca587 (stale) Mirror 6: 0x4af06f11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd197530a Mirror 7: 0xd197530a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000147 1778287554.164254 write 0x02bbb1 thru 0x03728b (0xb6db bytes) 000148 1778287554.312314 MIRROR_RESYNC 6 mirrors 000149 1778287555.828144 MIRROR_RESYNC 6 mirrors 000150 1778287556.487841 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000152 1778287557.029796 MIRROR_EXTEND to 6 mirrors 000153 1778287558.094590 write 0x019369 thru 0x01a152 (0xdea bytes) 000155 1778287558.193946 mapread 0x01b115 thru 0x024f73 (0x9e5f bytes) 000156 1778287558.213403 mapwrite 0x015e44 thru 0x01e0af (0x826c bytes) 000157 1778287558.269707 mapread 0x03be4a thru 0x03cee2 (0x1099 bytes) 000158 1778287558.284709 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd5b3f2d (stale) Mirror 4: 0xd5b3f2d (stale) Mirror 5: 0xd5b3f2d (stale) Mirror 7: 0xd5b3f2d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd5b3f2d (stale) Mirror 4: 0xd5b3f2d (stale) Mirror 5: 0xd5b3f2d (stale) Mirror 7: 0xd5b3f2d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000159 1778287558.787641 trunc from 0x040000 to 0x0307b5 (0xf84a bytes) 000160 1778287558.820650 read 0x019ca2 thru 0x0202ce (0x662d bytes) 000162 1778287558.833265 mapwrite 0x01331e thru 0x016aa0 (0x3783 bytes) 000163 1778287558.913533 read 0x026b9b thru 0x030536 (0x999c bytes) 000166 1778287558.944954 write 0x012b16 thru 0x01ac75 (0x8160 bytes) 000167 1778287558.947032 write 0x008352 thru 0x009feb (0x1c9a bytes) 000168 1778287558.949051 mapread 0x02b910 thru 0x0307b5 (0x4ea6 bytes) 000169 1778287558.952108 mapwrite 0x013fbd thru 0x01eb2a (0xab6e bytes) 000170 1778287559.015481 trunc from 0x0307b6 to 0x02229a (0xe51b bytes) 000171 1778287559.039805 mapread 0x000a2b thru 0x001178 (0x74e bytes) 000172 1778287559.041289 mapwrite 0x033151 thru 0x03f1ae (0xc05e bytes) 000173 1778287559.160956 mapread 0x01a849 thru 0x022b74 (0x832c bytes) 000175 1778287559.162831 mapwrite 0x032aea thru 0x03d52d (0xaa44 bytes) 000176 1778287559.339450 MIRROR_RESYNC 5 mirrors 000177 1778287560.472073 write 0x03c67a thru 0x03ffff (0x3986 bytes) 000178 1778287560.586865 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb4961a0 (stale) Mirror 4: 0xdb4961a0 (stale) Mirror 5: 0xdb4961a0 (stale) Mirror 7: 0xdb4961a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x15d9824d Mirror 8: 0x15d9824d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000180 1778287561.714273 mapwrite 0x00774b thru 0x012ac3 (0xb379 bytes) 000183 1778287562.054077 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb4961a0 (stale) Mirror 5: 0xdb4961a0 (stale) Mirror 7: 0xdb4961a0 (stale) Mirror 8: 0x15d9824d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xdb4961a0 (stale) Mirror 5: 0xdb4961a0 (stale) Mirror 7: 0xdb4961a0 (stale) Mirror 8: 0x15d9824d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000184 1778287562.765573 trunc from 0x040000 to 0x017ebe (0x28141 bytes) 000185 1778287562.791814 mapread 0x004779 thru 0x00bad4 (0x735c bytes) 000187 1778287562.819626 trunc from 0x017ebf to 0x014556 (0x3968 bytes) 000188 1778287562.852929 mapread 0x005cf0 thru 0x007c8b (0x1f9c bytes) 000189 1778287562.856370 trunc from 0x014557 to 0x006f71 (0xd5e5 bytes) 000190 1778287562.912763 read 0x004947 thru 0x006f71 (0x262b bytes) 000193 1778287562.927171 write 0x001c65 thru 0x004c92 (0x302e bytes) 000194 1778287562.929409 read 0x0057da thru 0x006f71 (0x1798 bytes) 000195 1778287562.930919 read 0x000e39 thru 0x006f71 (0x6139 bytes) 000196 1778287562.940308 read 0x002406 thru 0x006f71 (0x4b6c bytes) 000197 1778287562.941732 trunc from 0x006f72 to 0x02cd2d (0x25dbc bytes) 000198 1778287562.977906 write 0x00ed1b thru 0x00f3fc (0x6e2 bytes) 000199 1778287562.988051 read 0x01daba thru 0x02546f (0x79b6 bytes) 000200 1778287562.989085 write 0x023691 thru 0x02caf9 (0x9469 bytes) 000201 1778287562.990979 read 0x01a1f5 thru 0x02518e (0xaf9a bytes) 000202 1778287562.996606 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd2e): Mirror 3: 0x1b42b800 (stale) Mirror 5: 0x1b42b800 (stale) Mirror 8: 0x1b42b800 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd2e): Mirror 3: 0x1b42b800 (stale) Mirror 5: 0x1b42b800 (stale) Mirror 8: 0x1b42b800 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd2e: skipped 000203 1778287563.483122 mapread 0x027ebd thru 0x02cd2d (0x4e71 bytes) 000206 1778287563.516253 read 0x00e996 thru 0x00f90f (0xf7a bytes) 000207 1778287563.526920 mapread 0x008b0d thru 0x00c67a (0x3b6e bytes) 000208 1778287563.540888 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2cd2e): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd2e): Mirror 3: 0x1b42b800 (stale) Mirror 5: 0x1b42b800 (stale) Mirror 8: 0x1b42b800 (stale) CRC-32 checksum value for chunk [0, 0x2cd2e): Mirror 1: 0x5343c507 Mirror 9: 0x5343c507 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd2e: skipped 000212 1778287564.745959 mapread 0x027762 thru 0x02cd2d (0x55cc bytes) 000213 1778287564.853691 MIRROR_RESYNC 5 mirrors 000214 1778287566.021570 write 0x026ac3 thru 0x034574 (0xdab2 bytes) 000215 1778287566.248886 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34575): Mirror 3: 0x10481fe1 (stale) Mirror 8: 0x10481fe1 (stale) Mirror 9: 0x10481fe1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34575): Mirror 3: 0x10481fe1 (stale) Mirror 8: 0x10481fe1 (stale) Mirror 9: 0x10481fe1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34575: skipped 000216 1778287566.819207 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34575): Mirror 8: 0x10481fe1 (stale) Mirror 9: 0x10481fe1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34575): Mirror 8: 0x10481fe1 (stale) Mirror 9: 0x10481fe1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34575: skipped 000219 1778287567.291620 write 0x00d03f thru 0x0128aa (0x586c bytes) 000220 1778287567.310423 trunc from 0x034575 to 0x011aff (0x22a75 bytes) 000222 1778287567.355701 read 0x00ee14 thru 0x011aff (0x2cec bytes) 000223 1778287567.357874 write 0x01fd07 thru 0x026d76 (0x7070 bytes) 000224 1778287567.381951 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26d77): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26d77): Mirror 8: 0xcd5389b1 (stale) Mirror 9: 0xcd5389b1 (stale) CRC-32 checksum value for chunk [0, 0x26d77): Mirror 1: 0xe91658c9 Mirror 10: 0xe91658c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26d77: skipped 000226 1778287568.460401 trunc from 0x026d77 to 0x00f181 (0x17bf5 bytes) 000227 1778287568.636206 read 0x0019ec thru 0x00f181 (0xd796 bytes) 000228 1778287568.653935 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xf182): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf182): Mirror 8: 0xf06c1fb1 (stale) Mirror 9: 0xf06c1fb1 (stale) Mirror 10: 0xe6b6ba4c (stale) CRC-32 checksum value for chunk [0, 0xf182): Mirror 1: 0xe6b6ba4c Mirror 11: 0xe6b6ba4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf182: skipped 000231 1778287569.958077 mapwrite 0x021b7a thru 0x029657 (0x7ade bytes) 000233 1778287570.183250 trunc from 0x029658 to 0x00dca6 (0x1b9b1 bytes) 000234 1778287570.211506 mapread 0x003f9f thru 0x00a7c2 (0x6824 bytes) 000235 1778287570.230515 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xdca7): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdca7): Mirror 8: 0x4a97cb44 (stale) Mirror 9: 0x4a97cb44 (stale) Mirror 10: 0x1d320bd6 (stale) Mirror 11: 0x1d320bd6 (stale) CRC-32 checksum value for chunk [0, 0xdca7): Mirror 1: 0x1d320bd6 Mirror 12: 0x1d320bd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdca7: skipped 000239 1778287571.196814 write 0x02910b thru 0x038c20 (0xfb16 bytes) 000241 1778287571.291247 trunc from 0x038c21 to 0x00b56c (0x2d6b4 bytes) 000242 1778287571.344771 trunc from 0x00b56d to 0x00803a (0x3532 bytes) 000243 1778287571.373019 read 0x00797f thru 0x00803a (0x6bc bytes) 000247 1778287571.397304 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x803b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x803b): Mirror 8: 0x2ebe4c12 (stale) Mirror 9: 0x2ebe4c12 (stale) Mirror 10: 0x2ebe4c12 (stale) Mirror 11: 0x2ebe4c12 (stale) Mirror 12: 0x2ebe4c12 (stale) CRC-32 checksum value for chunk [0, 0x803b): Mirror 1: 0x2ebe4c12 Mirror 13: 0x2ebe4c12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x803b: skipped 000252 1778287572.410235 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x803b): Mirror 8: 0x2ebe4c12 (stale) Mirror 10: 0x2ebe4c12 (stale) Mirror 11: 0x2ebe4c12 (stale) Mirror 12: 0x2ebe4c12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x803b): Mirror 8: 0x2ebe4c12 (stale) Mirror 10: 0x2ebe4c12 (stale) Mirror 11: 0x2ebe4c12 (stale) Mirror 12: 0x2ebe4c12 (stale) CRC-32 checksum value for chunk [0, 0x803b): Mirror 1: 0x2ebe4c12 Mirror 13: 0x2ebe4c12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x803b: skipped 000256 1778287573.662848 trunc from 0x00803b to 0x0069f9 (0x1641 bytes) 000257 1778287573.919139 mapwrite 0x0114cc thru 0x01586c (0x43a1 bytes) 000259 1778287574.259138 read 0x00e013 thru 0x01586c (0x785a bytes) 000261 1778287574.265061 trunc from 0x01586d to 0x026deb (0x1157f bytes) 000262 1778287574.315808 write 0x02e51a thru 0x03b668 (0xd14f bytes) 000263 1778287574.326148 trunc from 0x03b669 to 0x02b9b4 (0xfcb4 bytes) 000264 1778287574.425105 MIRROR_RESYNC 6 mirrors 000266 1778287576.483163 write 0x014716 thru 0x021723 (0xd00e bytes) 000267 1778287576.585742 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b9b5): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9b5): Mirror 8: 0x907b5538 (stale) Mirror 10: 0x907b5538 (stale) Mirror 11: 0x907b5538 (stale) Mirror 12: 0x907b5538 (stale) Mirror 13: 0x907b5538 (stale) CRC-32 checksum value for chunk [0, 0x2b9b5): Mirror 1: 0xea70a22 Mirror 14: 0xea70a22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9b5: skipped 000268 1778287577.916013 read 0x021399 thru 0x02b9b4 (0xa61c bytes) 000271 1778287577.965084 read 0x023c32 thru 0x02b9b4 (0x7d83 bytes) 000272 1778287577.975400 write 0x024e72 thru 0x034478 (0xf607 bytes) 000273 1778287578.090883 mapread 0x02bfad thru 0x03240a (0x645e bytes) 000274 1778287578.094313 write 0x033e15 thru 0x036867 (0x2a53 bytes) 000275 1778287578.096905 trunc from 0x036868 to 0x0092f5 (0x2d572 bytes) 000278 1778287578.128005 trunc from 0x0092f6 to 0x030a6b (0x27776 bytes) 000281 1778287578.153057 write 0x0056b9 thru 0x01510d (0xfa55 bytes) 000283 1778287578.166673 write 0x035f58 thru 0x03c68e (0x6737 bytes) 000285 1778287578.169513 mapwrite 0x0131a2 thru 0x01890d (0x576c bytes) 000286 1778287578.286989 write 0x00430f thru 0x012eb7 (0xeba9 bytes) 000287 1778287578.295150 write 0x01a857 thru 0x0208e3 (0x608d bytes) 000289 1778287578.334065 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c68f): Mirror 8: 0x3dc239f (stale) Mirror 10: 0x3dc239f (stale) Mirror 11: 0x3dc239f (stale) Mirror 12: 0x3dc239f (stale) Mirror 13: 0x3dc239f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c68f): Mirror 8: 0x3dc239f (stale) Mirror 10: 0x3dc239f (stale) Mirror 11: 0x3dc239f (stale) Mirror 12: 0x3dc239f (stale) Mirror 13: 0x3dc239f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c68f: skipped 000290 1778287578.866157 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c68f): Mirror 8: 0x3dc239f (stale) Mirror 10: 0x3dc239f (stale) Mirror 11: 0x3dc239f (stale) Mirror 12: 0x3dc239f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c68f): Mirror 8: 0x3dc239f (stale) Mirror 10: 0x3dc239f (stale) Mirror 11: 0x3dc239f (stale) Mirror 12: 0x3dc239f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c68f: skipped 000291 1778287579.407519 MIRROR_RESYNC 5 mirrors 000292 1778287580.448086 MIRROR_EXTEND to 6 mirrors 000295 1778287581.456279 mapwrite 0x00fb01 thru 0x019301 (0x9801 bytes) 000297 1778287581.733382 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c68f): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c68f): Mirror 8: 0x6d2e0984 (stale) Mirror 10: 0x6d2e0984 (stale) Mirror 11: 0x6d2e0984 (stale) Mirror 12: 0x6d2e0984 (stale) Mirror 13: 0x6d2e0984 (stale) CRC-32 checksum value for chunk [0, 0x3c68f): Mirror 1: 0xd3a2f9b9 Mirror 14: 0xd3a2f9b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c68f: skipped 000298 1778287583.291888 mapwrite 0x03df13 thru 0x03ffff (0x20ed bytes) 000299 1778287583.521600 read 0x000acd thru 0x004ada (0x400e bytes) 000300 1778287583.536436 mapread 0x0105eb thru 0x01c723 (0xc139 bytes) 000301 1778287583.569850 trunc from 0x040000 to 0x005bc2 (0x3a43d bytes) 000303 1778287583.608616 MIRROR_RESYNC 7 mirrors 000304 1778287585.091650 mapwrite 0x00eb00 thru 0x018fa8 (0xa4a9 bytes) 000305 1778287585.258600 mapwrite 0x00a394 thru 0x0162ad (0xbf1a bytes) 000306 1778287585.310162 mapread 0x001a23 thru 0x004308 (0x28e6 bytes) 000308 1778287585.327890 read 0x016eef thru 0x018fa8 (0x20ba bytes) 000309 1778287585.331824 mapread 0x00ce9a thru 0x00d1c0 (0x327 bytes) 000314 1778287585.337752 write 0x013afe thru 0x015856 (0x1d59 bytes) 000317 1778287585.344078 write 0x038828 thru 0x03ffff (0x77d8 bytes) 000318 1778287585.354578 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x242f3d20 (stale) Mirror 10: 0x242f3d20 (stale) Mirror 11: 0x242f3d20 (stale) Mirror 13: 0x242f3d20 (stale) Mirror 14: 0x242f3d20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x242f3d20 (stale) Mirror 10: 0x242f3d20 (stale) Mirror 11: 0x242f3d20 (stale) Mirror 13: 0x242f3d20 (stale) Mirror 14: 0x242f3d20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000320 1778287585.876760 read 0x02919b thru 0x031491 (0x82f7 bytes) 000321 1778287585.890132 read 0x030a43 thru 0x0318b9 (0xe77 bytes) 000323 1778287585.891321 write 0x001b10 thru 0x003abe (0x1faf bytes) 000324 1778287585.914262 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x242f3d20 (stale) Mirror 10: 0x242f3d20 (stale) Mirror 11: 0x242f3d20 (stale) Mirror 14: 0x242f3d20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x242f3d20 (stale) Mirror 10: 0x242f3d20 (stale) Mirror 11: 0x242f3d20 (stale) Mirror 14: 0x242f3d20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000325 1778287586.314905 write 0x0278a7 thru 0x0348bf (0xd019 bytes) 000326 1778287586.341002 MIRROR_RESYNC 5 mirrors 000329 1778287587.616032 read 0x029e28 thru 0x035fdb (0xc1b4 bytes) 000330 1778287587.631118 mapwrite 0x037f14 thru 0x03ffff (0x80ec bytes) 000331 1778287587.905901 write 0x02a5ff thru 0x039927 (0xf329 bytes) 000332 1778287587.936852 trunc from 0x040000 to 0x03b12d (0x4ed2 bytes) 000333 1778287588.000554 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b12e): Mirror 10: 0xeeeea759 (stale) Mirror 11: 0xeeeea759 (stale) Mirror 14: 0xeeeea759 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b12e): Mirror 10: 0xeeeea759 (stale) Mirror 11: 0xeeeea759 (stale) Mirror 14: 0xeeeea759 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b12e: skipped 000336 1778287588.600220 write 0x00ad57 thru 0x00fc9a (0x4f44 bytes) 000337 1778287588.622233 mapwrite 0x02a816 thru 0x031a9e (0x7289 bytes) 000339 1778287588.762749 mapread 0x0033e0 thru 0x00cc96 (0x98b7 bytes) 000340 1778287588.791682 mapread 0x01f33d thru 0x02c998 (0xd65c bytes) 000342 1778287588.809149 mapread 0x001868 thru 0x00b260 (0x99f9 bytes) 000343 1778287588.814293 write 0x018262 thru 0x021a48 (0x97e7 bytes) 000344 1778287588.824394 write 0x01725d thru 0x01c224 (0x4fc8 bytes) 000346 1778287588.836943 write 0x0099cd thru 0x009fe8 (0x61c bytes) 000347 1778287588.837940 read 0x021794 thru 0x023084 (0x18f1 bytes) 000349 1778287588.841367 read 0x01b637 thru 0x020d88 (0x5752 bytes) 000350 1778287588.842809 read 0x00f976 thru 0x013e4a (0x44d5 bytes) 000351 1778287588.844490 mapread 0x025f04 thru 0x0276a4 (0x17a1 bytes) 000352 1778287588.846800 trunc from 0x03b12e to 0x032953 (0x87da bytes) 000354 1778287588.878758 trunc from 0x032954 to 0x013e76 (0x1eadd bytes) 000356 1778287588.908401 MIRROR_RESYNC 4 mirrors 000357 1778287590.129510 mapread 0x00edf5 thru 0x013e76 (0x5082 bytes) 000358 1778287590.181976 write 0x03141f thru 0x03e39d (0xcf7f bytes) 000362 1778287590.272410 mapwrite 0x02d5b2 thru 0x033c7d (0x66cc bytes) 000364 1778287590.401759 MIRROR_RESYNC 4 mirrors 000365 1778287591.628778 mapwrite 0x033a45 thru 0x03ffff (0xc5bb bytes) 000367 1778287591.816812 mapread 0x01990f thru 0x01f522 (0x5c14 bytes) 000368 1778287591.841212 mapwrite 0x02e2de thru 0x0349b5 (0x66d8 bytes) 000370 1778287591.916210 write 0x016fbc thru 0x02376e (0xc7b3 bytes) 000371 1778287591.925692 read 0x01dad2 thru 0x0286b3 (0xabe2 bytes) 000372 1778287591.935302 write 0x0196d8 thru 0x024e0b (0xb734 bytes) 000373 1778287591.936951 read 0x024fc2 thru 0x025d39 (0xd78 bytes) 000377 1778287591.940829 mapwrite 0x00f176 thru 0x018244 (0x90cf bytes) 000379 1778287592.058721 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x4e4d2272 (stale) Mirror 14: 0x4e4d2272 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x4e4d2272 (stale) Mirror 14: 0x4e4d2272 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000380 1778287592.589376 trunc from 0x040000 to 0x009d1b (0x362e4 bytes) truncating to largest ever: 0x3fb77 000381 1778287592.612978 trunc from 0x009d1c to 0x03fb76 (0x35e5b bytes) 000383 1778287592.647384 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fb77): Mirror 10: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb77): Mirror 10: 0x288fe944 (stale) Mirror 14: 0x288fe944 (stale) CRC-32 checksum value for chunk [0, 0x3fb77): Mirror 1: 0x1775ace4 Mirror 15: 0x1775ace4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb77: skipped 000384 1778287594.328771 read 0x00b037 thru 0x00c8ad (0x1877 bytes) 000386 1778287594.369938 read 0x015970 thru 0x01c9fe (0x708f bytes) 000387 1778287594.446637 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fb77): Mirror 10: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb77): Mirror 10: 0x288fe944 (stale) Mirror 14: 0x288fe944 (stale) CRC-32 checksum value for chunk [0, 0x3fb77): Mirror 1: 0x1775ace4 Mirror 15: 0x1775ace4 Mirror 16: 0x1775ace4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb77: skipped 000388 1778287595.653293 write 0x03bdbd thru 0x03ffff (0x4243 bytes) 000389 1778287595.806552 mapwrite 0x001603 thru 0x0041e7 (0x2be5 bytes) 000392 1778287595.878997 trunc from 0x040000 to 0x0030c2 (0x3cf3d bytes) 000393 1778287595.914528 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30c3): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c3): Mirror 10: 0x91514ae5 (stale) Mirror 14: 0x91514ae5 (stale) Mirror 15: 0x91514ae5 (stale) Mirror 16: 0x91514ae5 (stale) CRC-32 checksum value for chunk [0, 0x30c3): Mirror 1: 0x6c286baa Mirror 17: 0x6c286baa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c3: skipped 000394 1778287596.770657 trunc from 0x0030c3 to 0x025663 (0x225a1 bytes) 000396 1778287596.917851 write 0x01e965 thru 0x0279f2 (0x908e bytes) 000397 1778287596.931801 trunc from 0x0279f3 to 0x022433 (0x55bf bytes) 000399 1778287596.988196 trunc from 0x022434 to 0x01a2dd (0x8156 bytes) 000401 1778287597.021739 read 0x0047e1 thru 0x0096f7 (0x4f17 bytes) 000402 1778287597.032521 mapwrite 0x02e4f8 thru 0x035dc4 (0x78cd bytes) 000405 1778287597.139151 mapread 0x02b40c thru 0x035dc4 (0xa9b9 bytes) 000406 1778287597.142058 trunc from 0x035dc5 to 0x032456 (0x396e bytes) 000408 1778287597.162294 mapwrite 0x005892 thru 0x006aac (0x121b bytes) 000409 1778287597.206667 mapwrite 0x039423 thru 0x03a1e7 (0xdc5 bytes) 000412 1778287597.295681 write 0x02c284 thru 0x037e3e (0xbbbb bytes) 000413 1778287597.299827 read 0x01bb31 thru 0x02571d (0x9bed bytes) 000414 1778287597.318181 read 0x00a00f thru 0x00b07b (0x106d bytes) 000415 1778287597.319377 write 0x02cbee thru 0x0313b8 (0x47cb bytes) 000416 1778287597.323317 mapread 0x039640 thru 0x03a1e7 (0xba8 bytes) 000417 1778287597.326648 trunc from 0x03a1e8 to 0x007aa5 (0x32742 bytes) 000419 1778287597.352161 trunc from 0x007aa6 to 0x01a4f1 (0x12a4c bytes) 000420 1778287597.372315 read 0x015a26 thru 0x01a4f1 (0x4acc bytes) 000421 1778287597.393844 write 0x017dc3 thru 0x01956e (0x17ac bytes) 000422 1778287597.401902 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a4f2): Mirror 14: 0xd4bb2da8 (stale) Mirror 15: 0x6094fab7 (stale) Mirror 16: 0x6094fab7 (stale) Mirror 17: 0x3bf195c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a4f2): Mirror 14: 0xd4bb2da8 (stale) Mirror 15: 0x6094fab7 (stale) Mirror 16: 0x6094fab7 (stale) Mirror 17: 0x3bf195c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a4f2: skipped 000423 1778287597.863637 MIRROR_RESYNC 5 mirrors 000426 1778287599.269742 trunc from 0x01a4f2 to 0x007b78 (0x12979 bytes) 000427 1778287599.376234 trunc from 0x007b79 to 0x01bf50 (0x143d8 bytes) 000428 1778287599.428671 mapread 0x0193df thru 0x01bf50 (0x2b72 bytes) 000430 1778287599.507064 trunc from 0x01bf51 to 0x0168f1 (0x565f bytes) 000431 1778287599.558161 mapwrite 0x01b10f thru 0x01bf68 (0xe5a bytes) 000432 1778287599.760499 write 0x026814 thru 0x027700 (0xeed bytes) 000433 1778287599.764003 trunc from 0x027701 to 0x02226e (0x5492 bytes) 000434 1778287599.806672 trunc from 0x02226f to 0x008d28 (0x19546 bytes) 000436 1778287599.845988 mapread 0x00535d thru 0x008d28 (0x39cc bytes) 000437 1778287599.904050 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x8d29): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8d29): Mirror 14: 0xcde32c4d (stale) Mirror 15: 0xcde32c4d (stale) Mirror 16: 0xcde32c4d (stale) Mirror 17: 0xcde32c4d (stale) CRC-32 checksum value for chunk [0, 0x8d29): Mirror 1: 0xcde32c4d Mirror 18: 0xcde32c4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8d29: skipped 000439 1778287601.144888 read 0x00025d thru 0x006aa9 (0x684d bytes) 000440 1778287601.167770 MIRROR_RESYNC 6 mirrors 000441 1778287602.637565 mapwrite 0x018cc6 thru 0x0284aa (0xf7e5 bytes) 000442 1778287602.928233 mapread 0x025ae2 thru 0x0284aa (0x29c9 bytes) 000443 1778287602.931648 mapread 0x0223fc thru 0x02431a (0x1f1f bytes) 000445 1778287602.934837 mapwrite 0x00bd34 thru 0x00e90f (0x2bdc bytes) 000446 1778287603.067663 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x284ab): Mirror 15: 0x5ebd22af (stale) Mirror 16: 0x5ebd22af (stale) Mirror 17: 0x5ebd22af (stale) Mirror 18: 0x5ebd22af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x284ab): Mirror 15: 0x5ebd22af (stale) Mirror 16: 0x5ebd22af (stale) Mirror 17: 0x5ebd22af (stale) Mirror 18: 0x5ebd22af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x284ab: skipped 000447 1778287603.626087 mapread 0x00b5a0 thru 0x01945f (0xdec0 bytes) 000448 1778287603.657550 mapwrite 0x034940 thru 0x038b23 (0x41e4 bytes) 000449 1778287603.742465 write 0x005c16 thru 0x00c6e8 (0x6ad3 bytes) 000450 1778287603.751693 write 0x01c73c thru 0x026138 (0x99fd bytes) 000452 1778287603.762495 MIRROR_RESYNC 5 mirrors 000453 1778287605.125464 mapwrite 0x019273 thru 0x01ec07 (0x5995 bytes) 000454 1778287605.360670 read 0x001218 thru 0x00d6bb (0xc4a4 bytes) 000455 1778287605.418935 MIRROR_RESYNC 5 mirrors 000456 1778287607.153978 trunc from 0x038b24 to 0x01bd13 (0x1ce10 bytes) 000457 1778287607.306580 write 0x009d8f thru 0x018afe (0xed70 bytes) 000459 1778287607.351705 mapread 0x003073 thru 0x003786 (0x714 bytes) 000460 1778287607.368989 read 0x009817 thru 0x00a4fd (0xce7 bytes) 000461 1778287607.369841 write 0x037b47 thru 0x039a84 (0x1f3e bytes) 000462 1778287607.371406 read 0x023d88 thru 0x02f009 (0xb282 bytes) 000463 1778287607.386780 read 0x003a10 thru 0x00ca2d (0x901e bytes) 000465 1778287607.390119 mapwrite 0x0053ca thru 0x0084dc (0x3113 bytes) 000470 1778287607.473454 write 0x039e7a thru 0x03bf98 (0x211f bytes) 000471 1778287607.475946 read 0x004178 thru 0x00c6ff (0x8588 bytes) 000473 1778287607.478240 mapwrite 0x00fa54 thru 0x01848c (0x8a39 bytes) 000475 1778287607.605047 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bf99): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf99): Mirror 15: 0x5e59bd2 (stale) Mirror 16: 0x5e59bd2 (stale) Mirror 17: 0x5e59bd2 (stale) Mirror 18: 0x5e59bd2 (stale) CRC-32 checksum value for chunk [0, 0x3bf99): Mirror 1: 0x9dbb660f Mirror 19: 0x9dbb660f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf99: skipped 000479 1778287608.804455 mapwrite 0x011a03 thru 0x015788 (0x3d86 bytes) 000481 1778287609.089586 write 0x017fbe thru 0x021709 (0x974c bytes) 000482 1778287609.096319 write 0x00c006 thru 0x019922 (0xd91d bytes) 000483 1778287609.097824 mapread 0x005dc3 thru 0x012ae9 (0xcd27 bytes) 000485 1778287609.118274 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf99): Mirror 15: 0x5e59bd2 (stale) Mirror 17: 0x5e59bd2 (stale) Mirror 18: 0x5e59bd2 (stale) Mirror 19: 0x9dbb660f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf99): Mirror 15: 0x5e59bd2 (stale) Mirror 17: 0x5e59bd2 (stale) Mirror 18: 0x5e59bd2 (stale) Mirror 19: 0x9dbb660f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf99: skipped 000486 1778287609.597181 MIRROR_RESYNC 5 mirrors 000487 1778287610.842807 write 0x03a611 thru 0x03f7f8 (0x51e8 bytes) 000490 1778287610.965281 read 0x016275 thru 0x019a0e (0x379a bytes) 000491 1778287610.973729 read 0x017b95 thru 0x01e31f (0x678b bytes) 000492 1778287610.990100 mapwrite 0x003fdd thru 0x010731 (0xc755 bytes) 000497 1778287611.099210 write 0x00fddb thru 0x01e815 (0xea3b bytes) truncating to largest ever: 0x3fd8e 000498 1778287611.105887 trunc from 0x03f7f9 to 0x03fd8d (0x595 bytes) 000499 1778287611.157975 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd8e): Mirror 15: 0x122cdc6b (stale) Mirror 17: 0x122cdc6b (stale) Mirror 18: 0x122cdc6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd8e): Mirror 15: 0x122cdc6b (stale) Mirror 17: 0x122cdc6b (stale) Mirror 18: 0x122cdc6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd8e: skipped 000501 1778287611.706281 MIRROR_RESYNC 4 mirrors 000503 1778287612.963054 read 0x02c708 thru 0x02cfa1 (0x89a bytes) 000506 1778287612.982578 mapwrite 0x02645f thru 0x026dc3 (0x965 bytes) 000507 1778287613.181462 mapwrite 0x02660c thru 0x031420 (0xae15 bytes) 000508 1778287613.304328 write 0x01f0e3 thru 0x02e839 (0xf757 bytes) 000511 1778287613.327527 mapread 0x03f1a0 thru 0x03fd8d (0xbee bytes) 000512 1778287613.352675 read 0x03c1ae thru 0x03fd8d (0x3be0 bytes) 000513 1778287613.357109 write 0x029054 thru 0x032ac9 (0x9a76 bytes) 000515 1778287613.368438 MIRROR_RESYNC 4 mirrors 000517 1778287614.674121 trunc from 0x03fd8e to 0x027326 (0x18a67 bytes) 000518 1778287614.748476 write 0x038847 thru 0x03ffff (0x77b9 bytes) 000520 1778287614.752009 mapwrite 0x02d4c6 thru 0x035321 (0x7e5c bytes) 000521 1778287614.826658 trunc from 0x040000 to 0x00bebc (0x34143 bytes) 000522 1778287614.857881 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xbebd): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbebd): Mirror 15: 0x9c38efae (stale) Mirror 17: 0x9c38efae (stale) Mirror 18: 0x9c38efae (stale) CRC-32 checksum value for chunk [0, 0xbebd): Mirror 1: 0x9c38efae Mirror 19: 0x9c38efae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbebd: skipped 000523 1778287616.020090 read 0x00969b thru 0x00bebc (0x2822 bytes) 000525 1778287616.042641 trunc from 0x00bebd to 0x01bac3 (0xfc07 bytes) 000526 1778287616.191474 mapread 0x007de6 thru 0x00a6e1 (0x28fc bytes) 000527 1778287616.228327 mapwrite 0x0104a1 thru 0x019aee (0x964e bytes) 000529 1778287616.369403 read 0x017f1b thru 0x01bac3 (0x3ba9 bytes) 000532 1778287616.393933 mapread 0x000c25 thru 0x00e174 (0xd550 bytes) 000537 1778287616.417447 MIRROR_RESYNC 5 mirrors 000538 1778287617.784079 mapread 0x004246 thru 0x00d336 (0x90f1 bytes) 000539 1778287617.812566 trunc from 0x01bac4 to 0x013986 (0x813d bytes) 000540 1778287617.897402 read 0x001fcf thru 0x005e57 (0x3e89 bytes) 000542 1778287617.906699 trunc from 0x013987 to 0x0356bc (0x21d36 bytes) 000543 1778287617.939622 write 0x00162a thru 0x00a841 (0x9218 bytes) 000545 1778287617.951191 mapwrite 0x030b3e thru 0x03a6dc (0x9b9f bytes) 000549 1778287618.060710 mapread 0x036748 thru 0x03a6dc (0x3f95 bytes) 000552 1778287618.065865 write 0x01043c thru 0x01fab3 (0xf678 bytes) 000553 1778287618.080692 write 0x01ed8b thru 0x02c276 (0xd4ec bytes) 000554 1778287618.083111 mapread 0x0098d2 thru 0x010792 (0x6ec1 bytes) 000555 1778287618.085307 trunc from 0x03a6dd to 0x00ca39 (0x2dca3 bytes) 000559 1778287618.105944 mapread 0x00c489 thru 0x00ca39 (0x5b1 bytes) 000560 1778287618.107676 trunc from 0x00ca3a to 0x036a5b (0x2a022 bytes) 000562 1778287618.161827 MIRROR_RESYNC 5 mirrors 000563 1778287619.073277 write 0x03e7e9 thru 0x03ffff (0x1817 bytes) 000564 1778287619.142192 trunc from 0x040000 to 0x00c0a0 (0x33f5f bytes) 000565 1778287619.173107 trunc from 0x00c0a1 to 0x03d657 (0x315b7 bytes) 000566 1778287619.190121 read 0x015e6b thru 0x02191c (0xbab2 bytes) 000567 1778287619.205435 mapread 0x02daf1 thru 0x03d657 (0xfb67 bytes) 000568 1778287619.269999 mapread 0x000ecb thru 0x002bd0 (0x1d06 bytes) 000569 1778287619.284671 write 0x039c65 thru 0x03ffff (0x639b bytes) 000573 1778287619.289406 write 0x02f36a thru 0x03a581 (0xb218 bytes) 000575 1778287619.300510 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x35d09b9e (stale) Mirror 17: 0x35d09b9e (stale) Mirror 18: 0x35d09b9e (stale) Mirror 19: 0x35d09b9e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfd589fb3 Mirror 20: 0xfd589fb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000577 1778287620.331137 MIRROR_RESYNC 6 mirrors 000578 1778287621.321702 read 0x026567 thru 0x0305a5 (0xa03f bytes) 000580 1778287621.338627 mapread 0x0059da thru 0x014ec1 (0xf4e8 bytes) 000583 1778287621.397211 mapread 0x03e8f2 thru 0x03ffff (0x170e bytes) 000584 1778287621.413843 trunc from 0x040000 to 0x01e567 (0x21a98 bytes) 000586 1778287621.527041 mapwrite 0x02ded1 thru 0x0321e0 (0x4310 bytes) 000589 1778287621.617208 mapread 0x008e0b thru 0x010b4b (0x7d41 bytes) 000590 1778287621.630309 write 0x0054ac thru 0x011dd0 (0xc925 bytes) 000591 1778287621.637694 trunc from 0x0321e1 to 0x00e227 (0x23fb9 bytes) 000592 1778287621.659160 write 0x02ffd8 thru 0x032f39 (0x2f62 bytes) 000593 1778287621.660908 read 0x003c58 thru 0x0082f4 (0x469d bytes) 000594 1778287621.669799 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 15: 0x45ef822e (stale) Mirror 18: 0x45ef822e (stale) Mirror 19: 0x45ef822e (stale) Mirror 20: 0x45ef822e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 15: 0x45ef822e (stale) Mirror 18: 0x45ef822e (stale) Mirror 19: 0x45ef822e (stale) Mirror 20: 0x45ef822e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f3a: skipped 000596 1778287622.193390 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 15: 0x45ef822e (stale) Mirror 18: 0x45ef822e (stale) Mirror 19: 0x45ef822e (stale) Mirror 20: 0x45ef822e (stale) CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 1: 0xd30b2cd6 Mirror 21: 0xd30b2cd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f3a: skipped 000599 1778287623.391837 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 15: 0x45ef822e (stale) Mirror 18: 0x45ef822e (stale) Mirror 20: 0x45ef822e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 15: 0x45ef822e (stale) Mirror 18: 0x45ef822e (stale) Mirror 20: 0x45ef822e (stale) CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 1: 0xd30b2cd6 Mirror 21: 0xd30b2cd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f3a: skipped 000600 1778287624.194589 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 18: 0x45ef822e (stale) Mirror 20: 0x45ef822e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 18: 0x45ef822e (stale) Mirror 20: 0x45ef822e (stale) CRC-32 checksum value for chunk [0, 0x32f3a): Mirror 1: 0xd30b2cd6 Mirror 21: 0xd30b2cd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f3a: skipped 000601 1778287624.743964 write 0x001e03 thru 0x00e2d2 (0xc4d0 bytes) 000604 1778287624.815408 mapread 0x004fa1 thru 0x00a4bc (0x551c bytes) 000605 1778287624.816411 write 0x03877d thru 0x03ffff (0x7883 bytes) 000606 1778287624.817843 read 0x00c322 thru 0x015ae3 (0x97c2 bytes) 000607 1778287624.836606 read 0x029f74 thru 0x0363ae (0xc43b bytes) 000611 1778287624.848785 trunc from 0x040000 to 0x003696 (0x3c969 bytes) 000612 1778287624.870326 read 0x001898 thru 0x001ad2 (0x23b bytes) 000613 1778287624.876041 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3697): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3697): Mirror 18: 0x79ae28cd (stale) Mirror 20: 0x79ae28cd (stale) Mirror 21: 0x79ae28cd (stale) CRC-32 checksum value for chunk [0, 0x3697): Mirror 1: 0x7e4daf97 Mirror 22: 0x7e4daf97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3697: skipped 000615 1778287625.879494 mapwrite 0x00da5a thru 0x01809e (0xa645 bytes) 000616 1778287626.182043 trunc from 0x01809f to 0x0052d1 (0x12dcd bytes) 000618 1778287626.250108 trunc from 0x0052d2 to 0x012e90 (0xdbbf bytes) 000619 1778287626.279064 mapread 0x003a85 thru 0x012e90 (0xf40c bytes) 000620 1778287626.304092 trunc from 0x012e91 to 0x02843c (0x155ac bytes) 000624 1778287626.346546 mapwrite 0x01983b thru 0x028f96 (0xf75c bytes) 000625 1778287626.509204 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28f97): Mirror 18: 0x6a5a0ded (stale) Mirror 21: 0xc4e66647 (stale) Mirror 22: 0xbc913d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28f97): Mirror 18: 0x6a5a0ded (stale) Mirror 21: 0xc4e66647 (stale) Mirror 22: 0xbc913d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28f97: skipped 000627 1778287627.015238 read 0x0023d5 thru 0x007365 (0x4f91 bytes) 000628 1778287627.036531 MIRROR_RESYNC 4 mirrors 000630 1778287628.443385 mapwrite 0x017cb8 thru 0x01cc21 (0x4f6a bytes) 000631 1778287628.733479 read 0x014e3a thru 0x01bbf3 (0x6dba bytes) 000633 1778287628.748014 MIRROR_RESYNC 4 mirrors 000634 1778287629.854195 trunc from 0x028f97 to 0x02e617 (0x5681 bytes) 000635 1778287630.053552 mapwrite 0x01d6eb thru 0x02ad2f (0xd645 bytes) 000637 1778287630.288852 mapwrite 0x014cc9 thru 0x014f0c (0x244 bytes) 000638 1778287630.505234 read 0x014461 thru 0x020fb6 (0xcb56 bytes) 000640 1778287630.511846 read 0x019e97 thru 0x022af9 (0x8c63 bytes) 000642 1778287630.519038 read 0x0199c7 thru 0x019f18 (0x552 bytes) 000643 1778287630.521056 read 0x022edb thru 0x025f41 (0x3067 bytes) 000645 1778287630.528010 mapread 0x007640 thru 0x0137a2 (0xc163 bytes) 000647 1778287630.561003 trunc from 0x02e618 to 0x03385a (0x5243 bytes) 000649 1778287630.611151 mapwrite 0x030f3d thru 0x03d885 (0xc949 bytes) 000650 1778287630.774979 trunc from 0x03d886 to 0x02a77c (0x13109 bytes) 000652 1778287630.837010 MIRROR_RESYNC 4 mirrors 000653 1778287631.759961 trunc from 0x02a77d to 0x03b357 (0x10bdb bytes) 000654 1778287631.828059 write 0x00ed17 thru 0x010e05 (0x20ef bytes) 000655 1778287631.838387 mapread 0x03a2f7 thru 0x03b357 (0x1061 bytes) 000656 1778287631.848909 mapwrite 0x0353f6 thru 0x03ffff (0xac0a bytes) 000657 1778287631.936593 trunc from 0x040000 to 0x00b638 (0x349c7 bytes) 000658 1778287631.955318 write 0x029088 thru 0x02a05b (0xfd4 bytes) 000659 1778287631.958610 mapwrite 0x0258bd thru 0x02ba1a (0x615e bytes) 000660 1778287632.078518 MIRROR_RESYNC 4 mirrors 000662 1778287632.898591 mapwrite 0x019dce thru 0x028382 (0xe5b5 bytes) 000664 1778287633.215587 write 0x02ee34 thru 0x03dd27 (0xeef4 bytes) 000665 1778287633.220223 write 0x03691c thru 0x03b6ee (0x4dd3 bytes) 000667 1778287633.222800 write 0x0246d7 thru 0x033204 (0xeb2e bytes) 000670 1778287633.225823 mapread 0x008b11 thru 0x012867 (0x9d57 bytes) 000672 1778287633.254634 read 0x01be86 thru 0x0202b6 (0x4431 bytes) 000674 1778287633.268667 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd28): Mirror 21: 0x9d2436cd (stale) Mirror 22: 0x9d2436cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd28): Mirror 21: 0x9d2436cd (stale) Mirror 22: 0x9d2436cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd28: skipped 000676 1778287633.701912 mapwrite 0x039886 thru 0x03c690 (0x2e0b bytes) 000677 1778287633.782314 MIRROR_RESYNC 3 mirrors 000678 1778287634.801299 mapwrite 0x039468 thru 0x03ffff (0x6b98 bytes) 000679 1778287635.044969 mapwrite 0x0309bf thru 0x033c4f (0x3291 bytes) 000682 1778287635.099546 mapread 0x00aa3e thru 0x00dfff (0x35c2 bytes) 000683 1778287635.118969 MIRROR_RESYNC 3 mirrors 000684 1778287635.764602 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000686 1778287636.114456 MIRROR_RESYNC 2 mirrors 000687 1778287636.477287 MIRROR_RESYNC 2 mirrors 000688 1778287636.981454 MIRROR_EXTEND to 3 mirrors 000689 1778287637.741508 trunc from 0x040000 to 0x0310f6 (0xef09 bytes) 000691 1778287637.909488 read 0x01284a thru 0x016c7c (0x4433 bytes) 000692 1778287637.932155 mapwrite 0x019c39 thru 0x026fee (0xd3b6 bytes) 000693 1778287638.121315 trunc from 0x0310f7 to 0x011315 (0x1fde1 bytes) 000695 1778287638.157943 trunc from 0x011316 to 0x02d18c (0x1be77 bytes) 000697 1778287638.179865 mapwrite 0x025a2c thru 0x02a9d2 (0x4fa7 bytes) 000698 1778287638.337557 MIRROR_RESYNC 3 mirrors 000699 1778287639.280209 write 0x03c6b8 thru 0x03ffff (0x3948 bytes) 000701 1778287639.366178 read 0x01d88a thru 0x026b75 (0x92ec bytes) 000702 1778287639.379175 write 0x021731 thru 0x0232c0 (0x1b90 bytes) 000703 1778287639.393229 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xa58502b8 (stale) Mirror 22: 0xa58502b8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ced9964 Mirror 23: 0x3ced9964 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000704 1778287640.123718 read 0x004209 thru 0x013558 (0xf350 bytes) 000705 1778287640.143225 mapread 0x006237 thru 0x00e35a (0x8124 bytes) 000706 1778287640.190717 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0xa58502b8 (stale) Mirror 22: 0xa58502b8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ced9964 Mirror 23: 0x3ced9964 Mirror 24: 0x3ced9964 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000707 1778287641.256552 trunc from 0x040000 to 0x007494 (0x38b6b bytes) 000709 1778287641.447868 mapread 0x004eae thru 0x007494 (0x25e7 bytes) 000712 1778287641.469093 mapwrite 0x027b71 thru 0x02e32d (0x67bd bytes) 000716 1778287641.554683 mapwrite 0x018699 thru 0x0207a2 (0x810a bytes) 000717 1778287641.642161 mapread 0x0095be thru 0x009ca3 (0x6e6 bytes) 000718 1778287641.657871 mapwrite 0x0156f6 thru 0x01654b (0xe56 bytes) 000719 1778287641.775275 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e32e): Mirror 21: 0xcd4a462e (stale) Mirror 22: 0xcd4a462e (stale) Mirror 24: 0x159203ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e32e): Mirror 21: 0xcd4a462e (stale) Mirror 22: 0xcd4a462e (stale) Mirror 24: 0x159203ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e32e: skipped 000721 1778287642.149535 write 0x02d81e thru 0x037ea8 (0xa68b bytes) 000723 1778287642.160763 trunc from 0x037ea9 to 0x036946 (0x1562 bytes) 000724 1778287642.185262 mapwrite 0x02c821 thru 0x037d5a (0xb53a bytes) 000725 1778287642.331154 read 0x000f8c thru 0x00e713 (0xd788 bytes) 000727 1778287642.342167 mapwrite 0x03d010 thru 0x03ffff (0x2ff0 bytes) 000733 1778287642.481273 MIRROR_RESYNC 4 mirrors 000734 1778287643.724277 trunc from 0x040000 to 0x00d992 (0x3266d bytes) 000735 1778287643.803290 read 0x00030b thru 0x00d992 (0xd688 bytes) 000737 1778287643.825557 write 0x021fb2 thru 0x0318fc (0xf94b bytes) 000739 1778287643.829253 write 0x00fe58 thru 0x01afc3 (0xb16c bytes) 000742 1778287643.843711 trunc from 0x0318fd to 0x0384b5 (0x6bb9 bytes) 000744 1778287643.882996 read 0x025a61 thru 0x035117 (0xf6b7 bytes) 000745 1778287643.905245 read 0x0069e0 thru 0x00bf1a (0x553b bytes) 000746 1778287643.905943 trunc from 0x0384b6 to 0x02894f (0xfb66 bytes) 000748 1778287643.924168 trunc from 0x028950 to 0x027604 (0x134b bytes) 000749 1778287643.948518 MIRROR_RESYNC 4 mirrors 000750 1778287644.939644 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000753 1778287645.290931 mapwrite 0x03131a thru 0x03d727 (0xc40e bytes) 000757 1778287645.594900 write 0x0069a0 thru 0x007604 (0xc65 bytes) 000759 1778287645.606400 trunc from 0x03d728 to 0x00ee1c (0x2e90b bytes) 000760 1778287645.631269 write 0x022ae0 thru 0x025484 (0x29a5 bytes) 000761 1778287645.634596 trunc from 0x025485 to 0x0037a1 (0x21ce3 bytes) 000762 1778287645.659832 mapwrite 0x02093c thru 0x02237b (0x1a40 bytes) 000763 1778287645.731816 read 0x0129c8 thru 0x02237b (0xf9b4 bytes) 000764 1778287645.741126 mapread 0x00e024 thru 0x011cd8 (0x3cb5 bytes) 000765 1778287645.755826 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2237c): Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2237c): Mirror 22: 0xe5d26d49 (stale) Mirror 24: 0xe5d26d49 (stale) CRC-32 checksum value for chunk [0, 0x2237c): Mirror 1: 0x86216897 Mirror 25: 0x86216897 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2237c: skipped 000766 1778287646.794699 write 0x02be2f thru 0x032f50 (0x7122 bytes) 000767 1778287646.972103 trunc from 0x032f51 to 0x030406 (0x2b4a bytes) 000768 1778287647.025226 mapwrite 0x0355b0 thru 0x03ffff (0xaa50 bytes) 000769 1778287647.240832 mapread 0x01b1b0 thru 0x02243c (0x728d bytes) 000777 1778287647.262164 trunc from 0x040000 to 0x005af7 (0x3a508 bytes) 000780 1778287647.291643 mapread 0x003f39 thru 0x005af7 (0x1bbf bytes) 000781 1778287647.333032 mapwrite 0x0291be thru 0x0309d3 (0x7816 bytes) 000784 1778287647.431105 read 0x018220 thru 0x02812b (0xff0c bytes) 000785 1778287647.452508 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x309d4): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x309d4): Mirror 22: 0x96b1960 (stale) Mirror 24: 0x96b1960 (stale) Mirror 25: 0x57a42d53 (stale) CRC-32 checksum value for chunk [0, 0x309d4): Mirror 1: 0x74ed04dd Mirror 26: 0x74ed04dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x309d4: skipped 000786 1778287648.421290 trunc from 0x0309d4 to 0x0365a2 (0x5bcf bytes) 000789 1778287648.574167 trunc from 0x0365a3 to 0x029590 (0xd012 bytes) 000791 1778287648.603055 mapread 0x028c13 thru 0x029590 (0x97e bytes) 000792 1778287648.621779 mapread 0x01c26f thru 0x023454 (0x71e6 bytes) 000793 1778287648.649054 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29591): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29591): Mirror 22: 0x373193b3 (stale) Mirror 24: 0x373193b3 (stale) Mirror 25: 0x6f9e666 (stale) Mirror 26: 0xa4cd56ae (stale) CRC-32 checksum value for chunk [0, 0x29591): Mirror 1: 0xa4cd56ae Mirror 27: 0xa4cd56ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29591: skipped 000794 1778287649.531525 trunc from 0x029591 to 0x0219d9 (0x7bb7 bytes) 000795 1778287649.632455 mapread 0x01e716 thru 0x0219d9 (0x32c4 bytes) 000796 1778287649.647612 trunc from 0x0219da to 0x03ca4e (0x1b075 bytes) 000797 1778287649.677702 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca4f): Mirror 24: 0x8b144b22 (stale) Mirror 25: 0x68392cd1 (stale) Mirror 26: 0x366c4300 (stale) Mirror 27: 0xfc64f011 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca4f): Mirror 24: 0x8b144b22 (stale) Mirror 25: 0x68392cd1 (stale) Mirror 26: 0x366c4300 (stale) Mirror 27: 0xfc64f011 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca4f: skipped 000798 1778287650.043011 mapread 0x002fb5 thru 0x00a21b (0x7267 bytes) 000800 1778287650.072482 MIRROR_RESYNC 5 mirrors 000802 1778287651.120304 read 0x02a6fa thru 0x02d7cc (0x30d3 bytes) 000803 1778287651.126146 write 0x022eb5 thru 0x023c85 (0xdd1 bytes) 000805 1778287651.190002 trunc from 0x03ca4f to 0x030b9b (0xbeb3 bytes) 000806 1778287651.214479 write 0x01b8b0 thru 0x02817e (0xc8cf bytes) 000807 1778287651.235997 read 0x00359d thru 0x00f228 (0xbc8c bytes) 000809 1778287651.263024 mapread 0x010f1e thru 0x01ceaf (0xbf92 bytes) 000810 1778287651.266448 mapread 0x0081bf thru 0x00b2b2 (0x30f4 bytes) 000816 1778287651.272836 trunc from 0x030b9c to 0x0366c6 (0x5b2b bytes) 000817 1778287651.299740 MIRROR_RESYNC 5 mirrors 000818 1778287652.104278 read 0x00f702 thru 0x01cbea (0xd4e9 bytes) 000819 1778287652.114904 trunc from 0x0366c7 to 0x01d759 (0x18f6d bytes) 000821 1778287652.211967 trunc from 0x01d75a to 0x024949 (0x71f0 bytes) 000822 1778287652.253713 read 0x01e4e8 thru 0x024949 (0x6462 bytes) 000823 1778287652.273858 read 0x017f8f thru 0x01da49 (0x5abb bytes) 000824 1778287652.275895 read 0x00e382 thru 0x012c12 (0x4891 bytes) 000826 1778287652.278049 trunc from 0x02494a to 0x012b01 (0x11e48 bytes) 000827 1778287652.319666 write 0x0292a9 thru 0x02ec2c (0x5984 bytes) 000828 1778287652.323745 read 0x0101d3 thru 0x01aff6 (0xae24 bytes) 000829 1778287652.346932 mapread 0x01108e thru 0x014c77 (0x3bea bytes) 000830 1778287652.348248 mapread 0x0157d3 thru 0x022191 (0xc9bf bytes) 000831 1778287652.350742 mapread 0x005f3c thru 0x00731b (0x13e0 bytes) 000833 1778287652.352506 trunc from 0x02ec2d to 0x003e18 (0x2ae14 bytes) 000834 1778287652.401217 MIRROR_RESYNC 5 mirrors 000836 1778287653.423148 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000837 1778287653.830520 mapwrite 0x0025d0 thru 0x002e74 (0x8a5 bytes) 000838 1778287654.023920 write 0x031de6 thru 0x03ffff (0xe21a bytes) 000839 1778287654.025387 read 0x03d4d9 thru 0x03ffff (0x2b27 bytes) 000842 1778287654.035565 MIRROR_RESYNC 4 mirrors 000844 1778287654.861919 mapwrite 0x0149d0 thru 0x021b94 (0xd1c5 bytes) 000846 1778287655.058599 trunc from 0x040000 to 0x01b871 (0x2478e bytes) 000849 1778287655.109993 mapwrite 0x03d3bd thru 0x03ffff (0x2c43 bytes) 000850 1778287655.207979 mapwrite 0x002d01 thru 0x003bae (0xeae bytes) 000852 1778287655.257171 read 0x038b33 thru 0x03fefa (0x73c8 bytes) 000853 1778287655.261438 mapwrite 0x022af4 thru 0x0312c1 (0xe7ce bytes) 000854 1778287655.366858 mapread 0x007b88 thru 0x00e98d (0x6e06 bytes) 000855 1778287655.368521 mapread 0x005f12 thru 0x00db36 (0x7c25 bytes) 000856 1778287655.369876 trunc from 0x040000 to 0x018962 (0x2769d bytes) 000857 1778287655.388898 mapread 0x00cedb thru 0x015ef6 (0x901c bytes) 000858 1778287655.390237 mapwrite 0x01769b thru 0x02745c (0xfdc2 bytes) 000859 1778287655.503581 MIRROR_RESYNC 4 mirrors 000863 1778287656.400858 write 0x017a12 thru 0x01f9a9 (0x7f98 bytes) 000867 1778287656.468088 read 0x004c5d thru 0x012293 (0xd637 bytes) 000868 1778287656.479439 write 0x02de58 thru 0x031ca2 (0x3e4b bytes) 000871 1778287656.481743 trunc from 0x031ca3 to 0x0291b9 (0x8ae9 bytes) 000872 1778287656.542584 read 0x021088 thru 0x0291b9 (0x8132 bytes) 000873 1778287656.556152 read 0x002179 thru 0x005d86 (0x3c0e bytes) 000876 1778287656.560592 read 0x015483 thru 0x02520a (0xfd88 bytes) 000877 1778287656.562218 mapread 0x007818 thru 0x00ad4d (0x3536 bytes) 000878 1778287656.569546 write 0x03a474 thru 0x03ffff (0x5b8c bytes) 000879 1778287656.575468 read 0x005197 thru 0x00c99f (0x7809 bytes) 000880 1778287656.582598 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0x97de1a24 (stale) Mirror 27: 0x97de1a24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0x97de1a24 (stale) Mirror 27: 0x97de1a24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000882 1778287656.990493 trunc from 0x040000 to 0x033bf5 (0xc40a bytes) 000883 1778287657.009991 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33bf6): Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33bf6): Mirror 26: 0x4bb5dee5 (stale) Mirror 27: 0x4bb5dee5 (stale) CRC-32 checksum value for chunk [0, 0x33bf6): Mirror 1: 0xc7cf5495 Mirror 28: 0xc7cf5495 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33bf6: skipped 000884 1778287657.876665 trunc from 0x033bf6 to 0x02f878 (0x437d bytes) 000885 1778287657.993099 MIRROR_RESYNC 4 mirrors 000889 1778287658.887551 mapread 0x021c18 thru 0x02af74 (0x935d bytes) 000891 1778287658.911533 mapread 0x015669 thru 0x01635f (0xcf7 bytes) 000894 1778287658.928607 trunc from 0x02f879 to 0x00669e (0x291da bytes) 000895 1778287659.097097 write 0x03f75b thru 0x03ffff (0x8a5 bytes) 000897 1778287659.104121 read 0x0135ef thru 0x01e1cb (0xabdd bytes) 000898 1778287659.119818 mapread 0x01ff8d thru 0x02293a (0x29ae bytes) 000899 1778287659.159845 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0x2417eab3 (stale) Mirror 27: 0x2417eab3 (stale) Mirror 28: 0x2417eab3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x531a9c20 Mirror 29: 0x531a9c20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000900 1778287660.051546 read 0x018704 thru 0x01ac02 (0x24ff bytes) 000902 1778287660.065301 read 0x038f5b thru 0x03ffff (0x70a5 bytes) 000903 1778287660.079558 mapwrite 0x013171 thru 0x0145b0 (0x1440 bytes) 000905 1778287660.237471 trunc from 0x040000 to 0x003c6a (0x3c395 bytes) 000906 1778287660.309294 mapwrite 0x03e5d0 thru 0x03ffff (0x1a30 bytes) 000908 1778287660.408126 MIRROR_RESYNC 5 mirrors 000909 1778287661.535458 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) 000911 1778287662.040503 mapread 0x011159 thru 0x013490 (0x2338 bytes) 000912 1778287662.097651 MIRROR_EXTEND to 5 mirrors 000914 1778287662.982671 read 0x010a82 thru 0x018f0f (0x848e bytes) 000915 1778287663.013380 MIRROR_RESYNC 5 mirrors 000916 1778287663.438005 mapwrite 0x00ef55 thru 0x0182b0 (0x935c bytes) 000917 1778287663.656051 MIRROR_RESYNC 5 mirrors 000918 1778287664.936781 trunc from 0x040000 to 0x0063f5 (0x39c0a bytes) 000921 1778287665.023491 mapwrite 0x009a08 thru 0x0128e0 (0x8ed9 bytes) 000922 1778287665.139943 MIRROR_RESYNC 5 mirrors 000926 1778287666.183414 MIRROR_RESYNC 5 mirrors 000927 1778287666.629921 read 0x006b40 thru 0x007bd2 (0x1093 bytes) 000930 1778287666.639685 mapwrite 0x00e031 thru 0x00ed04 (0xcd4 bytes) 000932 1778287666.836176 write 0x008470 thru 0x0091e5 (0xd76 bytes) 000933 1778287666.856611 trunc from 0x0128e1 to 0x024669 (0x11d89 bytes) 000935 1778287666.933556 trunc from 0x02466a to 0x00f007 (0x15662 bytes) 000936 1778287666.950395 write 0x0382fe thru 0x03ffff (0x7d02 bytes) 000937 1778287666.951927 mapwrite 0x00959c thru 0x00ae22 (0x1887 bytes) 000941 1778287667.002898 write 0x01af83 thru 0x01d809 (0x2887 bytes) 000944 1778287667.014718 trunc from 0x040000 to 0x009f7a (0x36085 bytes) 000950 1778287667.066314 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x9f7b): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f7b): Mirror 26: 0x1d7eaa2f (stale) Mirror 27: 0x1d7eaa2f (stale) Mirror 28: 0x1d7eaa2f (stale) Mirror 29: 0x1d7eaa2f (stale) CRC-32 checksum value for chunk [0, 0x9f7b): Mirror 1: 0xe92593a Mirror 30: 0xe92593a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f7b: skipped 000952 1778287668.080242 MIRROR_RESYNC 6 mirrors 000954 1778287669.168392 mapread 0x001bd3 thru 0x005bc9 (0x3ff7 bytes) 000956 1778287669.197513 write 0x03bf5e thru 0x03ffff (0x40a2 bytes) 000957 1778287669.370138 write 0x028a81 thru 0x02fc06 (0x7186 bytes) 000958 1778287669.396688 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x173:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0xfc224506 (stale) Mirror 27: 0xfc224506 (stale) Mirror 28: 0xfc224506 (stale) Mirror 30: 0xfc224506 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 26: 0xfc224506 (stale) Mirror 27: 0xfc224506 (stale) Mirror 28: 0xfc224506 (stale) Mirror 30: 0xfc224506 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000960 1778287669.887949 mapread 0x034acd thru 0x03ffff (0xb533 bytes) 000962 1778287669.916192 write 0x00fdd2 thru 0x0197ee (0x9a1d bytes) 000964 1778287669.934197 trunc from 0x040000 to 0x02eb09 (0x114f6 bytes) 000965 1778287669.961393 mapwrite 0x014d65 thru 0x015ce9 (0xf85 bytes) 000969 1778287670.082584 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2eb0a): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb0a): Mirror 26: 0x75317aa1 (stale) Mirror 27: 0x75317aa1 (stale) Mirror 28: 0x75317aa1 (stale) Mirror 30: 0x75317aa1 (stale) CRC-32 checksum value for chunk [0, 0x2eb0a): Mirror 1: 0x9cc98484 Mirror 31: 0x9cc98484 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb0a: skipped 000970 1778287671.207935 write 0x020a17 thru 0x0294c6 (0x8ab0 bytes) 000971 1778287671.347182 read 0x01c517 thru 0x01fcb5 (0x379f bytes) 000972 1778287671.354277 trunc from 0x02eb0a to 0x03eefe (0x103f5 bytes) 000975 1778287671.377291 mapread 0x01689b thru 0x01e85c (0x7fc2 bytes) 000976 1778287671.388154 write 0x014bfe thru 0x02038d (0xb790 bytes) 000978 1778287671.390190 write 0x001cfe thru 0x0056fa (0x39fd bytes) 000979 1778287671.399839 write 0x008312 thru 0x00fafa (0x77e9 bytes) 000980 1778287671.412377 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3eeff): Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eeff): Mirror 26: 0x6db79b3a (stale) Mirror 27: 0x6db79b3a (stale) Mirror 28: 0x6db79b3a (stale) Mirror 30: 0x6db79b3a (stale) Mirror 31: 0x5d4e323b (stale) CRC-32 checksum value for chunk [0, 0x3eeff): Mirror 1: 0xa3654164 Mirror 32: 0xa3654164 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eeff: skipped 000981 1778287672.260986 mapread 0x03034c thru 0x034380 (0x4035 bytes) 000986 1778287672.313079 mapwrite 0x00d58f thru 0x014073 (0x6ae5 bytes) 000990 1778287672.563006 mapwrite 0x036285 thru 0x03be7f (0x5bfb bytes) 000991 1778287672.633493 trunc from 0x03eeff to 0x010ad4 (0x2e42a bytes) 000993 1778287672.705943 read 0x0039c1 thru 0x00a68a (0x6cca bytes) 000995 1778287672.712943 mapread 0x00e8c4 thru 0x00f202 (0x93f bytes) All operations completed A-OK! PASS 70a (142s) == sanity-flr test 71: check mirror extend parent fid ==== 20:47:58 (1778287678) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:48:02 (1778287682) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255473 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x249:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225166 s, 4.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x249:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x249:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x22c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x24a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 20:48:11 (1778287691) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1872384 .. done Extending file size to 9210496 .. Extending file size to 6099264 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6753440 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7414304 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7548256 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8393504 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9311264 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56073 56074 56076 56077 56078 done 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) 192.168.204.128@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:49:34 (1778287774) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1307712 .. Extending file size to 5165216 .. Extending file size to 5513184 .. Extending file size to 7769120 .. Extending file size to 7877632 .. Extending file size to 8133440 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9014752 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8760704 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57615 57616 57617 57620 57621 done 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) 192.168.204.128@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:50:51 (1778287851) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166203 s, 12.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266803 s, 7.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe9811cc9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:51:09 (1778287869) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12486 s, 16.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 20:51:16 (1778287876) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x243:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x243:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x244:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:51:23 (1778287883) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299979 s, 7.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x245:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:51:31 (1778287891) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.288657 s, 7.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x246:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x246:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:51:39 (1778287899) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.298533 s, 7.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x264:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x265:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:52:12 (1778287932) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124541 s, 8.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x267:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x249:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x267:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24a:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:53:08 (1778287988) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x268:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x269:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211204 s, 9.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x268:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x269:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x268:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x269:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:54:07 (1778288047) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg428-server mds-ost sync done. Creating new pool test_204f oleg428-server: Pool lustre.test_204f created Adding targets to pool oleg428-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg428-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22501 s, 9.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x24c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg428-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg428-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg428-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:55:26 (1778288126) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:55:34 (1778288134) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:55:41 (1778288141) oleg428-server: Pool lustre.test_206 created Adding targets to pool oleg428-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg428-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg428-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg428-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg428-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:56:07 (1778288167) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11497 s, 9.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295186 s, 35.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7e6860fb Mirror 3: 0x7e6860fb PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:56:20 (1778288180) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:56:24 (1778288184) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:56:27 (1778288187) Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.124966 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2090000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1390000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1390000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1390000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x790000 Read this (4 bytes): BBBB 192.168.204.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg428-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:57:02 (1778288222) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0758238 s, 108 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:57:08 (1778288228) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304653 s, 34.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ed lmm_fid: [0x200000404:0x1ed:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ed lmm_fid: [0x200000404:0x1ed:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x256:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834413 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 20:57:09.000000000 -0400 Modify: 2026-05-08 20:57:09.000000000 -0400 Change: 2026-05-08 20:57:10.000000000 -0400 Birth: 2026-05-08 20:57:09.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:57:15 (1778288235) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:57:18 (1778288238) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.681949 s, 60.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0507574 s, 80.7 kB/s size after second write -rw-r--r-- 1 root root 4096 May 8 20:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ef:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 8 20:57 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:57:25 (1778288245) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:57:32 (1778288252) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1712 sec =========== 20:57:41 (1778288261) === sanity-flr: start cleanup 20:57:43 (1778288263) === === sanity-flr: finish cleanup 20:57:50 (1778288270) ===