-----============= acceptance-small: sanity-flr ============----- Tue Nov 18 18:07:41 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg240-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:08:01 (1763507281) === oleg240-client.virtnet: executing check_config_client /mnt/lustre oleg240-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg240-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8daa133dd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8daa133dd000.idle_timeout=debug disable quota as required oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:08:34 (1763507314) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7555] == sanity-flr test 0a: lfs mirror create with -N option == 18:08:37 (1763507317) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:08:56 (1763507336) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:09:01 (1763507341) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:09:04 (1763507344) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:09:32 (1763507372) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:09:36 (1763507376) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 18:09:40 (1763507380) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8daa133dd000.stats=clear osc.lustre-OST0001-osc-ffff8daa133dd000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:09:49 (1763507389) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:10:00 (1763507400) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:10:08 (1763507408) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:10:11 (1763507411) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 18:10:20 (1763507420) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:10:26 (1763507426) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:10:30 (1763507430) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:10:37 (1763507437) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:10:47 (1763507447) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:10:54 (1763507454) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.46723 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11084 s, 9.5 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:11:03 (1763507463) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.174534 s, 60.1 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:11:10 (1763507470) Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (64s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:12:15 (1763507535) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.494931 s, 23.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:12:50 (1763507570) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (67s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:13:57 (1763507637) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.33671 s, 12.1 MB/s -rw-r--r-- 1 root root 52428800 Nov 18 18:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.929s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8daa133dd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.980s user 0m0.001s sys 0m0.173s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.106s user 0m0.002s sys 0m0.193s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8daa133dd000.ost_server_uuid in FULL state after 0 sec PASS 33b (65s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:15:02 (1763507702) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:15:05 (1763507705) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:15:08 (1763507708) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 18:15:11 (1763507711) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 18:15:18 (1763507718) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.547243 s, 23.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:15:32 (1763507732) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295326 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377673 s, 27.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:15:39 (1763507739) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:15:47 (1763507747) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.003043 s, 337 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00730601 s, 140 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000903556 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00126322 s, 811 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 18:15:57 (1763507757) create test files with size 15732306 .. c7313a40fcea936d867445eae003f772 - bbc45186263ef714aed7f34121336a3d - e0d79f6bda35c8bd2e7cf4516dd27e08 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre PASS 37 (19s) == sanity-flr test 38: resync ============================ 18:16:16 (1763507776) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:16:27 (1763507787) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:16:34 (1763507794) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104663 s, 10.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480755 s, 21.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 18:16:41 (1763507801) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126385 s, 16.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.301539 s, 13.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182636 s, 11.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (11s) == sanity-flr test 42: lfs mirror verify ================= 18:16:53 (1763507813) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 18:16:55 (1763507815) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:16:57 (1763507817) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259733 s, 40.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 18:17:02 (1763507822) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:17:05 (1763507825) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:17:12 (1763507832) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.226766 s, 46.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:17:22 (1763507842) llite.lustre-ffff8daa130f3800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.26332 s, 39.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8daa109dd000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:17:34 (1763507854) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (26s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:18:00 (1763507880) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:18:30 (1763507910) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 18:18:33 (1763507913) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.212338 s, 14.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.171421 s, 18.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa4c1fefe (stale) Mirror 2: 0xa4c1fefe (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 2d091eaa0c4ea6ab7f3d97f93666c826 - mirror 1 checksum: 960223c396de55ce1eb26797f6651106 - mirror 2 checksum: 2d091eaa0c4ea6ab7f3d97f93666c826 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 18 18:18:35 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Nov 18 18:18:35 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2d091eaa0c4ea6ab7f3d97f93666c826 - mirror 1 checksum: 960223c396de55ce1eb26797f6651106 - mirror 2 checksum: 960223c396de55ce1eb26797f6651106 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 18 18:18:35 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Nov 18 18:18:35 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:18:43 (1763507923) Failing mds1 on oleg240-server Stopping /mnt/lustre-mds1 (opts:) on oleg240-server 18:18:49 (1763507929) shut down facet: mds1 facet_host: oleg240-server facet_failover_host: oleg240-server Failover mds1 to oleg240-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 18:19:09 (1763507949) targets are mounted 18:19:09 (1763507949) facet_failover done oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209474 s, 50.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (38s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:19:21 (1763507961) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:19:24 (1763507964) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:19:26 (1763507966) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00540182 s, 3.0 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:19:34 (1763507974) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:19:36 (1763507976) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:19:39 (1763507979) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:20:01 (1763508001) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:20:27 (1763508027) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:20:50 (1763508050) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:21:11 (1763508071) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 18:21:17 (1763508077) mirror create pid 52267 mirror split pid 52268 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52267 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52268 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 18:22:23 (1763508143) Seed set to 8144 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x157eb 000001 1763508145.153714 trunc from 00000000 to 0x0157ea (0x157eb bytes) 000002 1763508145.172844 mapwrite 0x03fc6b thru 0x03ffff (0x395 bytes) 000005 1763508145.239385 trunc from 0x040000 to 0x013af6 (0x2c509 bytes) 000008 1763508145.262011 trunc from 0x013af7 to 0x011624 (0x24d2 bytes) 000011 1763508145.284559 mapwrite 0x036279 thru 0x03dd24 (0x7aac bytes) truncating to largest ever: 0x3fb97 000013 1763508145.360442 trunc from 0x03dd25 to 0x03fb96 (0x1e72 bytes) 000018 1763508145.384487 trunc from 0x03fb97 to 0x02229b (0x1d8fb bytes) 000025 1763508145.406442 mapwrite 0x01999a thru 0x023ce8 (0xa34f bytes) 000026 1763508145.510587 trunc from 0x023ce9 to 0x019957 (0xa391 bytes) 000030 1763508145.535387 mapwrite 0x021681 thru 0x02acdd (0x965d bytes) 000031 1763508145.639207 mapread 0x010dba thru 0x01f90b (0xeb52 bytes) 000032 1763508145.651350 trunc from 0x02acde to 0x020311 (0xa9cc bytes) 000034 1763508145.674674 trunc from 0x020312 to 0x01595c (0xa9b5 bytes) 000036 1763508145.694918 mapread 0x0141ce thru 0x0158fc (0x172f bytes) 000037 1763508145.695923 read 0x014f67 thru 0x01595c (0x9f6 bytes) 000038 1763508145.696529 trunc from 0x01595d to 0x00c5fa (0x9362 bytes) 000041 1763508145.725074 trunc from 0x00c5fb to 0x0053a8 (0x7252 bytes) 000044 1763508145.751956 read 0x00436b thru 0x0053a8 (0x103e bytes) 000045 1763508145.783640 MIRROR_EXTEND to 2 mirrors 000048 1763508146.426737 mapread 0x003d10 thru 0x0053a8 (0x1699 bytes) 000051 1763508146.483669 mapwrite 0x00baef thru 0x014ff0 (0x9502 bytes) 000052 1763508146.652616 trunc from 0x014ff1 to 0x021f17 (0xcf27 bytes) 000054 1763508146.702342 mapwrite 0x00c1c0 thru 0x01ba25 (0xf866 bytes) 000055 1763508146.760121 write 0x00b00e thru 0x01800a (0xcffd bytes) 000056 1763508146.761599 write 0x018b0f thru 0x0200c5 (0x75b7 bytes) 000060 1763508146.778680 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000061 1763508146.943177 mapwrite 0x000e1f thru 0x003b84 (0x2d66 bytes) 000062 1763508147.059507 write 0x03bbd6 thru 0x03ff32 (0x435d bytes) 000063 1763508147.061602 mapread 0x00e335 thru 0x01138d (0x3059 bytes) 000065 1763508147.077646 write 0x006ad5 thru 0x00a7c5 (0x3cf1 bytes) 000070 1763508147.082672 write 0x01e8a1 thru 0x02973a (0xae9a bytes) 000071 1763508147.103587 MIRROR_EXTEND to 2 mirrors 000074 1763508147.791637 trunc from 0x03ff33 to 0x003be8 (0x3c34a bytes) 000076 1763508147.849561 read 0x001b5b thru 0x003be8 (0x208e bytes) 000077 1763508147.857845 trunc from 0x003be9 to 0x01b1f1 (0x17609 bytes) 000078 1763508147.893223 trunc from 0x01b1f2 to 0x0259eb (0xa7fa bytes) 000081 1763508147.915059 write 0x02dfe8 thru 0x032624 (0x463d bytes) 000082 1763508147.924111 MIRROR_RESYNC 2 mirrors 000084 1763508148.482030 mapwrite 0x00f45e thru 0x0141b0 (0x4d53 bytes) 000085 1763508148.702968 mapread 0x00b11d thru 0x018293 (0xd177 bytes) 000088 1763508148.708650 read 0x02c9a0 thru 0x032624 (0x5c85 bytes) 000089 1763508148.727090 mapwrite 0x005c80 thru 0x008bd6 (0x2f57 bytes) 000092 1763508148.794976 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000094 1763508148.949762 write 0x004db8 thru 0x00aa9c (0x5ce5 bytes) 000097 1763508148.972485 read 0x00f453 thru 0x012e81 (0x3a2f bytes) 000098 1763508148.991886 mapread 0x006a5e thru 0x0082da (0x187d bytes) 000100 1763508148.993504 mapwrite 0x036871 thru 0x039f6b (0x36fb bytes) 000102 1763508149.116971 write 0x035a14 thru 0x03f525 (0x9b12 bytes) 000103 1763508149.118318 mapread 0x00c0a1 thru 0x00e939 (0x2899 bytes) 000104 1763508149.135652 trunc from 0x03f526 to 0x02615d (0x193c8 bytes) 000106 1763508149.162630 mapwrite 0x00716d thru 0x00dfbe (0x6e52 bytes) 000108 1763508149.247290 trunc from 0x02615e to 0x01dd8f (0x83ce bytes) 000109 1763508149.279569 read 0x00ea3d thru 0x01a4c7 (0xba8b bytes) 000111 1763508149.299491 write 0x01f6ac thru 0x026e76 (0x77cb bytes) 000116 1763508149.303887 read 0x022d8e thru 0x026e76 (0x40e9 bytes) 000118 1763508149.313700 mapwrite 0x009b8b thru 0x00dc0c (0x4082 bytes) 000119 1763508149.371792 mapwrite 0x0263b0 thru 0x0317ff (0xb450 bytes) 000120 1763508149.456597 write 0x03b2be thru 0x03f9b4 (0x46f7 bytes) 000121 1763508149.457854 read 0x03e569 thru 0x03f9b4 (0x144c bytes) 000122 1763508149.458444 write 0x00f48d thru 0x013c45 (0x47b9 bytes) 000125 1763508149.460981 write 0x016b14 thru 0x01a974 (0x3e61 bytes) 000126 1763508149.462255 trunc from 0x03f9b5 to 0x028bd6 (0x16dde bytes) 000127 1763508149.482950 read 0x00e10d thru 0x010660 (0x2554 bytes) 000130 1763508149.489537 MIRROR_EXTEND to 2 mirrors 000132 1763508150.000457 write 0x03ff96 thru 0x03ffff (0x6a bytes) 000133 1763508150.085863 mapwrite 0x02ef45 thru 0x0314f2 (0x25ae bytes) 000135 1763508150.185449 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000136 1763508150.398212 mapread 0x0347f4 thru 0x037181 (0x298e bytes) 000137 1763508150.420957 trunc from 0x040000 to 0x010edf (0x2f120 bytes) 000139 1763508150.467537 MIRROR_EXTEND to 2 mirrors 000140 1763508151.190039 trunc from 0x010ee0 to 0x0327e2 (0x21903 bytes) 000142 1763508151.262925 write 0x0378fc thru 0x03ffff (0x8704 bytes) 000143 1763508151.264261 read 0x01c118 thru 0x02735d (0xb246 bytes) 000144 1763508151.278901 mapread 0x010520 thru 0x013dd2 (0x38b3 bytes) 000145 1763508151.299476 mapwrite 0x012f6a thru 0x0213ff (0xe496 bytes) 000148 1763508151.381456 trunc from 0x040000 to 0x003ca3 (0x3c35c bytes) 000149 1763508151.425100 trunc from 0x003ca4 to 0x003f47 (0x2a4 bytes) 000154 1763508151.460059 MIRROR_RESYNC 2 mirrors 000155 1763508152.090222 mapwrite 0x01fccb thru 0x02ee82 (0xf1b8 bytes) 000156 1763508152.369698 mapread 0x012cbc thru 0x01d87c (0xabc1 bytes) 000158 1763508152.379767 mapread 0x01cfdb thru 0x02862d (0xb653 bytes) 000159 1763508152.380729 trunc from 0x02ee83 to 0x01d825 (0x1165d bytes) 000160 1763508152.414624 read 0x00dd55 thru 0x01566e (0x791a bytes) 000163 1763508152.416354 read 0x012e78 thru 0x01c153 (0x92dc bytes) 000168 1763508152.419406 mapwrite 0x032f4e thru 0x0345b5 (0x1668 bytes) 000170 1763508152.513658 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x345b6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x345b6): Mirror 2: 0x6b819d12 (stale) CRC-32 checksum value for chunk [0, 0x345b6): Mirror 1: 0xdfe8a712 Mirror 3: 0xdfe8a712 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x345b6: skipped 000171 1763508153.433277 trunc from 0x0345b6 to 0x002a34 (0x31b81 bytes) 000172 1763508153.560846 mapwrite 0x017f89 thru 0x01eb3e (0x6bb6 bytes) 000173 1763508153.648922 write 0x01c9c2 thru 0x02ba58 (0xf097 bytes) 000174 1763508153.652717 mapread 0x024bfa thru 0x02ba58 (0x6e5f bytes) 000176 1763508153.654037 write 0x03bb3b thru 0x03ffff (0x44c5 bytes) 000178 1763508153.655603 mapwrite 0x03fd83 thru 0x03ffff (0x27d bytes) 000179 1763508153.712684 trunc from 0x040000 to 0x0055b9 (0x3aa46 bytes) 000180 1763508153.736588 read 0x002578 thru 0x004ae3 (0x256c bytes) 000181 1763508153.747001 mapread 0x004af1 thru 0x0055b9 (0xac9 bytes) 000182 1763508153.755655 write 0x017725 thru 0x018806 (0x10e2 bytes) 000184 1763508153.756970 write 0x01b192 thru 0x022cf9 (0x7b68 bytes) 000186 1763508153.759341 mapwrite 0x01f057 thru 0x02691a (0x78c4 bytes) 000188 1763508153.893738 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2691b): Mirror 3: 0x7e66fffc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2691b): Mirror 3: 0x7e66fffc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2691b: skipped 000189 1763508154.165609 mapwrite 0x03225d thru 0x037caa (0x5a4e bytes) 000190 1763508154.279184 write 0x011950 thru 0x016dd3 (0x5484 bytes) 000191 1763508154.293812 mapread 0x00c67a thru 0x012f12 (0x6899 bytes) 000192 1763508154.299618 write 0x00f509 thru 0x01029a (0xd92 bytes) 000193 1763508154.302909 read 0x033a39 thru 0x037caa (0x4272 bytes) 000194 1763508154.303617 mapwrite 0x036044 thru 0x03c459 (0x6416 bytes) 000196 1763508154.443105 mapwrite 0x01fe94 thru 0x0245d2 (0x473f bytes) 000198 1763508154.530577 mapwrite 0x03f7be thru 0x03ffff (0x842 bytes) 000199 1763508154.647595 trunc from 0x040000 to 0x024ebd (0x1b142 bytes) 000201 1763508154.679946 mapwrite 0x018e9b thru 0x018fdb (0x141 bytes) 000202 1763508154.742218 mapread 0x00cd1f thru 0x01961c (0xc8fe bytes) 000203 1763508154.743289 write 0x002f72 thru 0x00d22b (0xa2ba bytes) 000207 1763508154.777254 write 0x03d15c thru 0x03ffff (0x2ea4 bytes) 000208 1763508154.778495 mapwrite 0x015f8f thru 0x022d39 (0xcdab bytes) 000209 1763508154.884602 mapwrite 0x0158c3 thru 0x01ff39 (0xa677 bytes) 000212 1763508154.975102 read 0x0376c1 thru 0x03ffff (0x893f bytes) 000214 1763508155.007757 trunc from 0x040000 to 0x026969 (0x19696 bytes) 000215 1763508155.065875 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000218 1763508155.379377 mapwrite 0x001192 thru 0x00842f (0x729e bytes) 000219 1763508155.576531 write 0x03d4c4 thru 0x03ffff (0x2b3c bytes) 000222 1763508155.581830 mapwrite 0x0046b5 thru 0x00bf70 (0x78bc bytes) 000224 1763508155.676558 read 0x01d817 thru 0x01e84b (0x1035 bytes) 000226 1763508155.687376 mapwrite 0x0385d5 thru 0x039029 (0xa55 bytes) 000227 1763508155.772921 read 0x03ab8c thru 0x03ffff (0x5474 bytes) 000228 1763508155.776048 mapwrite 0x03362e thru 0x03ffff (0xc9d2 bytes) 000229 1763508155.861984 trunc from 0x040000 to 0x035139 (0xaec6 bytes) 000230 1763508155.889209 read 0x0284f5 thru 0x02db13 (0x561f bytes) 000232 1763508155.911402 mapread 0x008ddc thru 0x00b598 (0x27bd bytes) 000233 1763508155.912376 mapwrite 0x01e2eb thru 0x02baa9 (0xd7bf bytes) 000236 1763508156.010442 mapwrite 0x018ef5 thru 0x02362e (0xa73a bytes) 000237 1763508156.144000 MIRROR_EXTEND to 2 mirrors 000240 1763508156.821424 read 0x019582 thru 0x025317 (0xbd96 bytes) 000242 1763508156.839551 mapwrite 0x0226a0 thru 0x031cf3 (0xf654 bytes) 000244 1763508157.044414 mapwrite 0x034f94 thru 0x0364b1 (0x151e bytes) 000245 1763508157.174692 read 0x016f83 thru 0x01753a (0x5b8 bytes) 000246 1763508157.185056 mapwrite 0x0137b6 thru 0x018a6d (0x52b8 bytes) 000250 1763508157.245444 trunc from 0x0364b2 to 0x02b8a5 (0xac0c bytes) 000251 1763508157.284271 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000252 1763508157.444519 MIRROR_EXTEND to 2 mirrors 000255 1763508158.103774 trunc from 0x02b8a6 to 0x02ebd9 (0x3334 bytes) 000257 1763508158.238333 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000258 1763508158.400965 mapwrite 0x0308fa thru 0x0339c4 (0x30cb bytes) 000261 1763508158.500534 mapread 0x00565c thru 0x012b3e (0xd4e3 bytes) 000263 1763508158.524284 MIRROR_EXTEND to 2 mirrors 000268 1763508159.163019 write 0x014381 thru 0x01e471 (0xa0f1 bytes) 000269 1763508159.254431 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x339c5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x339c5): Mirror 2: 0x321de267 (stale) CRC-32 checksum value for chunk [0, 0x339c5): Mirror 1: 0x49e3b13e Mirror 3: 0x49e3b13e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x339c5: skipped 000272 1763508160.232276 write 0x00655e thru 0x0153f3 (0xee96 bytes) 000273 1763508160.369387 mapread 0x00d2b0 thru 0x0165f4 (0x9345 bytes) 000275 1763508160.380682 mapwrite 0x02ba9f thru 0x035e52 (0xa3b4 bytes) 000277 1763508160.469819 read 0x0267d2 thru 0x02cea2 (0x66d1 bytes) 000279 1763508160.476749 MIRROR_RESYNC 3 mirrors 000282 1763508161.247629 trunc from 0x035e53 to 0x029e4e (0xc004 bytes) 000283 1763508161.408552 read 0x01ff0f thru 0x029e4e (0x9f40 bytes) 000284 1763508161.428242 mapwrite 0x038d0b thru 0x03f239 (0x652f bytes) 000285 1763508161.592558 MIRROR_RESYNC 3 mirrors 000286 1763508162.369868 read 0x0290a0 thru 0x0341f6 (0xb157 bytes) 000287 1763508162.383387 mapwrite 0x007145 thru 0x00ed30 (0x7bec bytes) 000289 1763508162.569700 mapread 0x01ed99 thru 0x024b09 (0x5d71 bytes) 000290 1763508162.597853 MIRROR_RESYNC 3 mirrors 000291 1763508163.508083 mapwrite 0x026b35 thru 0x0355ea (0xeab6 bytes) 000293 1763508163.832034 trunc from 0x03f23a to 0x023974 (0x1b8c5 bytes) 000294 1763508163.869667 mapwrite 0x02f6fb thru 0x0356f3 (0x5ff9 bytes) 000296 1763508163.990804 write 0x0002db thru 0x0010f0 (0xe16 bytes) 000297 1763508164.021461 mapread 0x014632 thru 0x017029 (0x29f8 bytes) 000298 1763508164.035556 read 0x0245a1 thru 0x02c346 (0x7da6 bytes) 000300 1763508164.057743 mapwrite 0x004e84 thru 0x01220a (0xd387 bytes) 000301 1763508164.117360 read 0x025601 thru 0x03099c (0xb39c bytes) 000305 1763508164.123214 write 0x0044be thru 0x01230c (0xde4f bytes) 000306 1763508164.139631 MIRROR_RESYNC 3 mirrors 000307 1763508165.020835 write 0x024dcf thru 0x02badf (0x6d11 bytes) 000308 1763508165.104774 write 0x004cda thru 0x00604d (0x1374 bytes) 000309 1763508165.114918 read 0x032829 thru 0x0356f3 (0x2ecb bytes) 000311 1763508165.126995 MIRROR_RESYNC 3 mirrors 000314 1763508166.035240 read 0x029b3f thru 0x02f03b (0x54fd bytes) 000316 1763508166.045484 trunc from 0x0356f4 to 0x00d328 (0x283cb bytes) 000317 1763508166.189216 write 0x01beda thru 0x01e151 (0x2278 bytes) 000318 1763508166.191509 trunc from 0x01e152 to 0x00b610 (0x12b41 bytes) 000319 1763508166.238541 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb611): Mirror 3: 0x53cc6f79 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb611): Mirror 3: 0x53cc6f79 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb611: skipped 000323 1763508166.586701 write 0x003042 thru 0x009d45 (0x6d04 bytes) 000324 1763508166.613692 mapwrite 0x0388bb thru 0x039c29 (0x136f bytes) 000326 1763508166.765776 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000327 1763508166.974902 read 0x029f66 thru 0x02d748 (0x37e3 bytes) 000329 1763508166.993001 write 0x0102d8 thru 0x016ad7 (0x6800 bytes) 000330 1763508167.021029 trunc from 0x039c2a to 0x02e082 (0xbba7 bytes) 000331 1763508167.068869 write 0x0259de thru 0x02c666 (0x6c89 bytes) 000334 1763508167.104315 MIRROR_EXTEND to 2 mirrors 000336 1763508167.875584 read 0x01d9d2 thru 0x02054d (0x2b7c bytes) 000340 1763508167.974008 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000343 1763508168.212799 read 0x0216ae thru 0x02c51f (0xae72 bytes) 000344 1763508168.239343 write 0x0261f8 thru 0x027cac (0x1ab5 bytes) 000345 1763508168.293840 mapwrite 0x014a64 thru 0x018e33 (0x43d0 bytes) 000346 1763508168.453180 mapread 0x01f0be thru 0x02220a (0x314d bytes) 000350 1763508168.480777 mapread 0x00cedb thru 0x0188a9 (0xb9cf bytes) 000352 1763508168.504124 write 0x01f2fb thru 0x022985 (0x368b bytes) 000357 1763508168.507359 mapwrite 0x023727 thru 0x0281c1 (0x4a9b bytes) 000358 1763508168.593653 mapwrite 0x02bdda thru 0x02e091 (0x22b8 bytes) 000363 1763508168.682409 mapread 0x02c8f8 thru 0x02e091 (0x179a bytes) 000364 1763508168.683304 write 0x0001c1 thru 0x00149f (0x12df bytes) 000365 1763508168.707053 write 0x012123 thru 0x014513 (0x23f1 bytes) 000369 1763508168.709274 mapread 0x023819 thru 0x023824 (0xc bytes) 000371 1763508168.711990 write 0x0309e8 thru 0x0313ae (0x9c7 bytes) 000372 1763508168.714570 write 0x014d14 thru 0x017cdf (0x2fcc bytes) 000373 1763508168.716591 read 0x00508a thru 0x00ce8f (0x7e06 bytes) 000375 1763508168.718311 mapwrite 0x02ad45 thru 0x02cbb9 (0x1e75 bytes) 000376 1763508168.786216 trunc from 0x0313af to 0x01e550 (0x12e5e bytes) 000380 1763508168.821957 mapwrite 0x01fc71 thru 0x02e779 (0xeb09 bytes) 000382 1763508168.895948 trunc from 0x02e77a to 0x019348 (0x15431 bytes) 000383 1763508168.916078 read 0x014406 thru 0x019348 (0x4f43 bytes) 000385 1763508168.917228 mapread 0x00d3e9 thru 0x00e439 (0x1051 bytes) 000387 1763508168.918841 trunc from 0x019349 to 0x00f534 (0x9e14 bytes) 000392 1763508168.948032 read 0x00c452 thru 0x00d8bc (0x146b bytes) 000395 1763508168.950496 mapread 0x005c5d thru 0x00f534 (0x98d8 bytes) 000396 1763508168.953989 mapwrite 0x012bd8 thru 0x0150a2 (0x24cb bytes) 000398 1763508169.067105 mapwrite 0x00386e thru 0x00b403 (0x7b96 bytes) 000403 1763508169.171517 read 0x005d4e thru 0x00df9b (0x824e bytes) 000406 1763508169.172934 write 0x013a5f thru 0x020744 (0xcce6 bytes) 000407 1763508169.174237 mapwrite 0x00aaf6 thru 0x0158b8 (0xadc3 bytes) 000409 1763508169.282489 read 0x0127bc thru 0x017b16 (0x535b bytes) 000410 1763508169.283085 mapread 0x00da40 thru 0x010f21 (0x34e2 bytes) 000411 1763508169.286533 read 0x01d631 thru 0x020744 (0x3114 bytes) 000412 1763508169.288473 mapread 0x003969 thru 0x00403f (0x6d7 bytes) 000413 1763508169.289454 write 0x03c5c5 thru 0x03ffff (0x3a3b bytes) 000416 1763508169.291577 mapwrite 0x033398 thru 0x0371cc (0x3e35 bytes) 000417 1763508169.377620 trunc from 0x040000 to 0x0284a0 (0x17b5f bytes) 000418 1763508169.404803 trunc from 0x0284a1 to 0x03e219 (0x15d79 bytes) 000419 1763508169.420043 mapwrite 0x018402 thru 0x01dc91 (0x5890 bytes) 000421 1763508169.490715 mapread 0x035130 thru 0x03999e (0x486f bytes) 000425 1763508169.520463 write 0x01254e thru 0x015924 (0x33d7 bytes) 000426 1763508169.526394 MIRROR_EXTEND to 2 mirrors 000432 1763508170.257884 write 0x03195a thru 0x038333 (0x69da bytes) 000434 1763508170.348500 mapwrite 0x00a276 thru 0x00cc12 (0x299d bytes) 000435 1763508170.437418 mapread 0x017749 thru 0x01c45d (0x4d15 bytes) 000439 1763508170.446197 write 0x023132 thru 0x02cbde (0x9aad bytes) 000443 1763508170.462867 read 0x010b13 thru 0x0200ad (0xf59b bytes) 000444 1763508170.463568 mapread 0x0021cc thru 0x009656 (0x748b bytes) 000445 1763508170.473202 trunc from 0x03e21a to 0x003116 (0x3b103 bytes) 000447 1763508170.521344 trunc from 0x003117 to 0x0167fa (0x136e4 bytes) 000451 1763508170.546421 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x167fb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x167fb): Mirror 2: 0x83a506ce (stale) CRC-32 checksum value for chunk [0, 0x167fb): Mirror 1: 0xe6f7e9c Mirror 3: 0xe6f7e9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x167fb: skipped 000452 1763508171.389688 trunc from 0x0167fb to 0x00143c (0x153be bytes) 000453 1763508171.505810 trunc from 0x00143d to 0x03ebc2 (0x3d786 bytes) 000454 1763508171.534027 trunc from 0x03ebc3 to 0x03a703 (0x44bf bytes) 000456 1763508171.566717 read 0x0387f5 thru 0x03a703 (0x1f0f bytes) 000457 1763508171.576934 mapwrite 0x00786b thru 0x00ad16 (0x34ac bytes) 000458 1763508171.663369 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a704): Mirror 3: 0xc7a3cf56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a704): Mirror 3: 0xc7a3cf56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a704: skipped 000459 1763508172.032936 mapwrite 0x006d77 thru 0x0115cf (0xa859 bytes) 000460 1763508172.115920 mapwrite 0x026aaf thru 0x02dbd6 (0x7128 bytes) 000461 1763508172.188681 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000462 1763508172.332992 mapread 0x032f77 thru 0x036e39 (0x3ec3 bytes) 000466 1763508172.363802 trunc from 0x03a704 to 0x0269d7 (0x13d2c bytes) 000472 1763508172.398376 write 0x028df6 thru 0x02e18d (0x5398 bytes) 000475 1763508172.401882 write 0x009c5b thru 0x00fde3 (0x6189 bytes) 000477 1763508172.413720 mapread 0x006a62 thru 0x00e6ea (0x7c89 bytes) 000478 1763508172.414774 mapread 0x025de2 thru 0x027c7c (0x1e9b bytes) 000480 1763508172.426617 mapwrite 0x02c406 thru 0x0361d7 (0x9dd2 bytes) 000481 1763508172.509983 trunc from 0x0361d8 to 0x00fb9d (0x2663a bytes) 000483 1763508172.541692 write 0x00dd91 thru 0x0112a0 (0x3510 bytes) 000484 1763508172.542914 write 0x0025d8 thru 0x007318 (0x4d41 bytes) 000485 1763508172.545241 read 0x00ec3f thru 0x0112a0 (0x2662 bytes) 000486 1763508172.546427 mapwrite 0x03d97e thru 0x03ffff (0x2682 bytes) 000488 1763508172.652873 trunc from 0x040000 to 0x0125a5 (0x2da5a bytes) 000489 1763508172.669775 mapread 0x00db68 thru 0x0125a5 (0x4a3e bytes) 000490 1763508172.677526 mapwrite 0x03fcb9 thru 0x03ffff (0x347 bytes) 000491 1763508172.737041 read 0x024017 thru 0x02defb (0x9ee5 bytes) 000492 1763508172.746834 write 0x03cdad thru 0x03f661 (0x28b5 bytes) 000496 1763508172.750057 read 0x03629d thru 0x03ffff (0x9d63 bytes) 000497 1763508172.750969 trunc from 0x040000 to 0x03c3bf (0x3c40 bytes) 000498 1763508172.778266 trunc from 0x03c3c0 to 0x036896 (0x5b29 bytes) 000500 1763508172.796958 trunc from 0x036897 to 0x032d6e (0x3b28 bytes) 000502 1763508172.816720 mapwrite 0x01ba00 thru 0x024d4b (0x934c bytes) 000504 1763508172.860149 trunc from 0x032d6f to 0x00dc0d (0x25161 bytes) 000505 1763508172.877480 read 0x004258 thru 0x00dc0d (0x99b6 bytes) 000506 1763508172.878153 read 0x0029a7 thru 0x00b244 (0x889e bytes) 000508 1763508172.879067 write 0x0031ad thru 0x0070c1 (0x3f15 bytes) 000509 1763508172.880001 mapread 0x00b64d thru 0x00dc0d (0x25c1 bytes) 000510 1763508172.882509 mapwrite 0x007fa8 thru 0x00c3b7 (0x4410 bytes) 000512 1763508172.922095 trunc from 0x00dc0e to 0x00bd5c (0x1eb1 bytes) 000516 1763508172.949491 read 0x001c0c thru 0x006be7 (0x4fdc bytes) 000520 1763508172.953841 mapwrite 0x03b65e thru 0x03ffff (0x49a2 bytes) 000522 1763508173.038116 read 0x017912 thru 0x020b62 (0x9251 bytes) 000523 1763508173.038781 write 0x01b508 thru 0x01ee78 (0x3971 bytes) 000525 1763508173.040086 write 0x03c169 thru 0x03faa1 (0x3939 bytes) 000527 1763508173.043557 trunc from 0x040000 to 0x00a323 (0x35cdc bytes) 000530 1763508173.073939 write 0x0303c6 thru 0x037c70 (0x78ab bytes) 000531 1763508173.075245 read 0x001ba9 thru 0x0050d1 (0x3529 bytes) 000533 1763508173.076225 read 0x035492 thru 0x037c70 (0x27df bytes) 000537 1763508173.086941 MIRROR_EXTEND to 2 mirrors 000538 1763508173.695123 read 0x013f38 thru 0x01dac6 (0x9b8f bytes) 000539 1763508173.708676 mapread 0x00672d thru 0x015932 (0xf206 bytes) 000540 1763508173.739880 mapread 0x001821 thru 0x010a01 (0xf1e1 bytes) 000541 1763508173.749678 mapread 0x00be79 thru 0x01a0e1 (0xe269 bytes) 000542 1763508173.758475 write 0x007163 thru 0x009c42 (0x2ae0 bytes) 000543 1763508173.851033 mapwrite 0x0246a3 thru 0x02f16d (0xaacb bytes) 000544 1763508173.923220 mapread 0x026f44 thru 0x032a0d (0xbaca bytes) 000545 1763508173.924947 write 0x01b6c0 thru 0x02a277 (0xebb8 bytes) 000547 1763508173.941482 mapwrite 0x0273ee thru 0x02c622 (0x5235 bytes) 000549 1763508173.995071 mapread 0x017164 thru 0x020bfc (0x9a99 bytes) 000550 1763508173.996189 trunc from 0x037c71 to 0x004fa6 (0x32cca bytes) 000551 1763508174.050881 mapread 0x002977 thru 0x004fa6 (0x2630 bytes) 000553 1763508174.059007 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x4fa7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4fa7): Mirror 2: 0x20844de6 (stale) CRC-32 checksum value for chunk [0, 0x4fa7): Mirror 1: 0x20844de6 Mirror 3: 0x20844de6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4fa7: skipped 000554 1763508174.780918 write 0x026b7c thru 0x02befe (0x5383 bytes) 000555 1763508174.837179 mapwrite 0x02e4dd thru 0x037aca (0x95ee bytes) 000556 1763508174.924574 mapread 0x0113b5 thru 0x0127ab (0x13f7 bytes) 000557 1763508174.944543 mapread 0x01be1b thru 0x01f159 (0x333f bytes) 000558 1763508174.953567 mapread 0x02f817 thru 0x037aca (0x82b4 bytes) 000561 1763508174.955479 trunc from 0x037acb to 0x02b1c3 (0xc907 bytes) 000562 1763508174.980190 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b1c4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1c4): Mirror 2: 0x32b8f990 (stale) Mirror 3: 0x1ee427a5 (stale) CRC-32 checksum value for chunk [0, 0x2b1c4): Mirror 1: 0x951774d Mirror 4: 0x951774d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1c4: skipped 000563 1763508175.765784 write 0x033cb7 thru 0x03ffff (0xc349 bytes) 000565 1763508175.830432 trunc from 0x040000 to 0x01f103 (0x20efc bytes) 000567 1763508175.859166 write 0x03dae7 thru 0x03ffff (0x2519 bytes) 000571 1763508175.862429 mapwrite 0x009ad4 thru 0x0145c3 (0xaaf0 bytes) 000572 1763508175.942790 write 0x00513b thru 0x010dd3 (0xbc99 bytes) 000574 1763508175.944666 trunc from 0x040000 to 0x032441 (0xdbbe bytes) 000575 1763508175.964566 mapwrite 0x031361 thru 0x03d3cd (0xc06d bytes) 000576 1763508176.048251 mapwrite 0x024439 thru 0x031067 (0xcc2f bytes) 000577 1763508176.143561 read 0x00f78c thru 0x012799 (0x300e bytes) 000578 1763508176.144323 read 0x017ea2 thru 0x019805 (0x1964 bytes) 000580 1763508176.146168 write 0x002068 thru 0x008099 (0x6032 bytes) 000582 1763508176.154376 write 0x038013 thru 0x03ffff (0x7fed bytes) 000583 1763508176.155875 trunc from 0x040000 to 0x00f8f7 (0x30708 bytes) 000584 1763508176.176439 read 0x005d4b thru 0x00f8f7 (0x9bad bytes) 000587 1763508176.178019 mapread 0x0030df thru 0x00f8f7 (0xc819 bytes) 000588 1763508176.179633 write 0x017417 thru 0x0175fc (0x1e6 bytes) 000590 1763508176.186619 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x175fd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x175fd): Mirror 2: 0x4aa8ad68 (stale) Mirror 3: 0xc0fb506b (stale) Mirror 4: 0xc0fb506b (stale) CRC-32 checksum value for chunk [0, 0x175fd): Mirror 1: 0xdf7a1a0f Mirror 5: 0xdf7a1a0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x175fd: skipped 000593 1763508177.041437 mapwrite 0x0391b0 thru 0x03a7f7 (0x1648 bytes) 000594 1763508177.197109 mapwrite 0x005c5e thru 0x00cfe4 (0x7387 bytes) 000597 1763508177.257941 trunc from 0x03a7f8 to 0x0200e0 (0x1a717 bytes) 000601 1763508177.289385 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x200e1): Mirror 3: 0xaca84a25 (stale) Mirror 4: 0xaca84a25 (stale) Mirror 5: 0x79811b2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200e1): Mirror 3: 0xaca84a25 (stale) Mirror 4: 0xaca84a25 (stale) Mirror 5: 0x79811b2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200e1: skipped 000603 1763508177.630102 mapread 0x007f59 thru 0x013116 (0xb1be bytes) 000605 1763508177.665395 mapwrite 0x028edd thru 0x034a3e (0xbb62 bytes) 000606 1763508177.823917 MIRROR_RESYNC 4 mirrors 000610 1763508178.743089 trunc from 0x034a3f to 0x018894 (0x1c1aa bytes) 000611 1763508178.845815 MIRROR_RESYNC 4 mirrors 000612 1763508179.630881 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000615 1763508180.013428 read 0x00325f thru 0x0076fe (0x44a0 bytes) 000616 1763508180.021717 read 0x00b816 thru 0x017d6f (0xc55a bytes) 000620 1763508180.038151 mapread 0x0063a6 thru 0x010a97 (0xa6f2 bytes) 000624 1763508180.041611 read 0x0047b1 thru 0x00b899 (0x70e9 bytes) 000625 1763508180.042192 mapread 0x00536d thru 0x00ef28 (0x9bbc bytes) 000626 1763508180.044048 mapread 0x003947 thru 0x004a97 (0x1151 bytes) 000628 1763508180.046316 write 0x01b9c3 thru 0x023f13 (0x8551 bytes) 000630 1763508180.105830 read 0x006c47 thru 0x0146b4 (0xda6e bytes) 000631 1763508180.121225 trunc from 0x023f14 to 0x02f7f4 (0xb8e1 bytes) 000633 1763508180.148211 mapwrite 0x028a8f thru 0x030b37 (0x80a9 bytes) 000635 1763508180.284398 write 0x002da5 thru 0x00b050 (0x82ac bytes) 000637 1763508180.286037 read 0x02f818 thru 0x030b37 (0x1320 bytes) 000643 1763508180.291708 mapread 0x00d823 thru 0x012a32 (0x5210 bytes) 000645 1763508180.293893 trunc from 0x030b38 to 0x008884 (0x282b3 bytes) 000646 1763508180.311342 trunc from 0x008885 to 0x031df3 (0x2956f bytes) 000658 1763508180.332961 mapwrite 0x012101 thru 0x01931d (0x721d bytes) 000659 1763508180.436856 read 0x002753 thru 0x00f65f (0xcf0d bytes) 000660 1763508180.437606 mapwrite 0x026c0c thru 0x0306fb (0x9af0 bytes) 000662 1763508180.488659 trunc from 0x031df4 to 0x0255c1 (0xc832 bytes) 000663 1763508180.517118 mapwrite 0x03bd2c thru 0x03ffff (0x42d4 bytes) 000665 1763508180.618363 read 0x0103fe thru 0x01f35e (0xef61 bytes) 000667 1763508180.619568 mapwrite 0x017bad thru 0x01d8c6 (0x5d1a bytes) 000669 1763508180.677598 mapwrite 0x0054d4 thru 0x00d00d (0x7b3a bytes) 000670 1763508180.737992 mapread 0x00f30a thru 0x01e700 (0xf3f7 bytes) 000671 1763508180.739850 read 0x00cefb thru 0x014507 (0x760d bytes) 000672 1763508180.740465 write 0x0121ff thru 0x0163e5 (0x41e7 bytes) 000673 1763508180.741451 trunc from 0x040000 to 0x00e2f2 (0x31d0d bytes) 000675 1763508180.779956 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe2f3): Mirror 5: 0x5260da36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe2f3): Mirror 5: 0x5260da36 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe2f3: skipped 000676 1763508181.107116 mapread 0x004d47 thru 0x00c1e5 (0x749f bytes) 000677 1763508181.144536 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe2f3): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe2f3): Mirror 5: 0x5260da36 (stale) CRC-32 checksum value for chunk [0, 0xe2f3): Mirror 1: 0xd419baf1 Mirror 6: 0xd419baf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe2f3: skipped 000678 1763508182.143066 mapwrite 0x01fbbb thru 0x02d36a (0xd7b0 bytes) 000679 1763508182.322093 mapread 0x00b8f8 thru 0x00bd71 (0x47a bytes) 000681 1763508182.341166 trunc from 0x02d36b to 0x0179d2 (0x15998 bytes) 000682 1763508182.379919 mapread 0x0083ac thru 0x014fa6 (0xcbfb bytes) 000683 1763508182.395036 mapwrite 0x004a03 thru 0x0135d6 (0xebd4 bytes) 000684 1763508182.462832 trunc from 0x0179d3 to 0x015cfc (0x1cd6 bytes) 000685 1763508182.481713 mapread 0x00a2fd thru 0x015cfc (0xba00 bytes) 000686 1763508182.485638 write 0x03d0a4 thru 0x03ffff (0x2f5c bytes) 000687 1763508182.486730 mapread 0x012b8e thru 0x016af8 (0x3f6b bytes) 000688 1763508182.499689 MIRROR_RESYNC 3 mirrors 000689 1763508183.270840 mapwrite 0x03a567 thru 0x03ffff (0x5a99 bytes) 000690 1763508183.504836 mapwrite 0x024ea6 thru 0x02fbd6 (0xad31 bytes) 000691 1763508183.615702 write 0x00ac9a thru 0x011ede (0x7245 bytes) 000692 1763508183.627532 mapread 0x026095 thru 0x0298e9 (0x3855 bytes) 000694 1763508183.629523 write 0x03ffca thru 0x03ffff (0x36 bytes) 000697 1763508183.636265 read 0x0031ec thru 0x00a775 (0x758a bytes) 000699 1763508183.644513 write 0x02d5a2 thru 0x036567 (0x8fc6 bytes) 000702 1763508183.647070 mapread 0x0307bc thru 0x03ffff (0xf844 bytes) 000704 1763508183.650449 mapread 0x02d63d thru 0x03922e (0xbbf2 bytes) 000707 1763508183.653740 read 0x00db61 thru 0x01c6ba (0xeb5a bytes) 000708 1763508183.672481 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x30f9f5d9 (stale) Mirror 6: 0x30f9f5d9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7c64bc4c Mirror 7: 0x7c64bc4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000709 1763508184.654636 mapread 0x03a96f thru 0x03ffff (0x5691 bytes) 000711 1763508184.704538 write 0x003adf thru 0x008bcc (0x50ee bytes) 000713 1763508184.816512 mapwrite 0x039e28 thru 0x03ffff (0x61d8 bytes) 000714 1763508184.928347 read 0x00447e thru 0x013274 (0xedf7 bytes) 000715 1763508184.946715 mapwrite 0x014971 thru 0x01d698 (0x8d28 bytes) 000717 1763508185.079107 trunc from 0x040000 to 0x017be0 (0x2841f bytes) 000720 1763508185.119804 read 0x000d55 thru 0x0093d5 (0x8681 bytes) 000721 1763508185.134691 mapwrite 0x00d17a thru 0x015e6e (0x8cf5 bytes) 000722 1763508185.225993 read 0x0076c9 thru 0x00a4ff (0x2e37 bytes) 000723 1763508185.226694 write 0x020b46 thru 0x02738e (0x6849 bytes) 000724 1763508185.227983 read 0x00cbaa thru 0x017c6d (0xb0c4 bytes) 000726 1763508185.229031 mapwrite 0x02fa72 thru 0x02ff21 (0x4b0 bytes) 000729 1763508185.338918 trunc from 0x02ff22 to 0x005df1 (0x2a130 bytes) 000730 1763508185.371416 read 0x002a60 thru 0x003bf6 (0x1197 bytes) 000732 1763508185.372596 mapread 0x00389c thru 0x005df1 (0x2556 bytes) 000736 1763508185.384631 read 0x0039b1 thru 0x005df1 (0x2441 bytes) 000737 1763508185.387274 trunc from 0x005df2 to 0x039610 (0x3381f bytes) 000739 1763508185.416289 trunc from 0x039611 to 0x03b16f (0x1b5f bytes) 000740 1763508185.437565 read 0x022510 thru 0x02e44a (0xbf3b bytes) 000742 1763508185.455033 MIRROR_RESYNC 4 mirrors 000743 1763508186.658676 mapwrite 0x027d8d thru 0x02fad9 (0x7d4d bytes) 000744 1763508186.887497 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b170): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b170): Mirror 5: 0xadbc184d (stale) Mirror 6: 0xadbc184d (stale) Mirror 7: 0xadbc184d (stale) CRC-32 checksum value for chunk [0, 0x3b170): Mirror 1: 0x196a4aad Mirror 8: 0x196a4aad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b170: skipped 000745 1763508187.788016 mapwrite 0x017da2 thru 0x0184ea (0x749 bytes) 000746 1763508188.046823 mapread 0x001694 thru 0x00344a (0x1db7 bytes) 000747 1763508188.064110 mapread 0x02d72d thru 0x033fc5 (0x6899 bytes) 000748 1763508188.080664 write 0x01337b thru 0x018287 (0x4f0d bytes) 000749 1763508188.082857 read 0x01d1ee thru 0x02942d (0xc240 bytes) 000750 1763508188.102271 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b170): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b170): Mirror 5: 0xadbc184d (stale) Mirror 6: 0xadbc184d (stale) Mirror 7: 0xadbc184d (stale) Mirror 8: 0x196a4aad (stale) CRC-32 checksum value for chunk [0, 0x3b170): Mirror 1: 0x8edc8dab Mirror 9: 0x8edc8dab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b170: skipped 000752 1763508188.988247 read 0x01c2a5 thru 0x024d2d (0x8a89 bytes) 000753 1763508189.016575 read 0x01b64b thru 0x028e9a (0xd850 bytes) 000754 1763508189.059945 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b170): Mirror 5: 0xadbc184d (stale) Mirror 6: 0xadbc184d (stale) Mirror 8: 0x196a4aad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b170): Mirror 5: 0xadbc184d (stale) Mirror 6: 0xadbc184d (stale) Mirror 8: 0x196a4aad (stale) CRC-32 checksum value for chunk [0, 0x3b170): Mirror 1: 0x8edc8dab Mirror 9: 0x8edc8dab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b170: skipped 000755 1763508189.654802 read 0x02b42e thru 0x02de0f (0x29e2 bytes) 000757 1763508189.676100 trunc from 0x03b170 to 0x023ff4 (0x1717b bytes) 000758 1763508189.812916 mapwrite 0x004ba7 thru 0x010cbe (0xc118 bytes) 000760 1763508189.885528 mapread 0x010b83 thru 0x020abe (0xff3c bytes) 000761 1763508189.902403 trunc from 0x023ff5 to 0x01f886 (0x476e bytes) 000762 1763508189.935062 mapread 0x00d935 thru 0x00e633 (0xcff bytes) 000763 1763508189.936026 write 0x002504 thru 0x0058ff (0x33fc bytes) 000765 1763508189.941649 MIRROR_RESYNC 5 mirrors 000768 1763508190.927952 write 0x03de3f thru 0x03ffff (0x21c1 bytes) 000769 1763508190.987201 mapread 0x003f8e thru 0x00e0f0 (0xa163 bytes) 000772 1763508191.017396 write 0x01f23d thru 0x01f9b7 (0x77b bytes) 000776 1763508191.026671 mapread 0x00dd1b thru 0x01d72d (0xfa13 bytes) 000777 1763508191.035865 read 0x032ceb thru 0x033f0c (0x1222 bytes) 000779 1763508191.043122 mapwrite 0x007737 thru 0x016155 (0xea1f bytes) 000783 1763508191.119384 read 0x029764 thru 0x0343dd (0xac7a bytes) 000784 1763508191.134757 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6ac458e8 (stale) Mirror 6: 0x6ac458e8 (stale) Mirror 8: 0x6ac458e8 (stale) Mirror 9: 0x6ac458e8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x816f29b8 Mirror 10: 0x816f29b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000785 1763508191.990814 mapread 0x015a2f thru 0x01621e (0x7f0 bytes) 000786 1763508192.012183 mapwrite 0x016d70 thru 0x024850 (0xdae1 bytes) 000787 1763508192.230790 read 0x01a73b thru 0x01e8d5 (0x419b bytes) 000789 1763508192.234622 mapwrite 0x00c71f thru 0x013bf3 (0x74d5 bytes) 000790 1763508192.329857 write 0x03cda8 thru 0x03ffff (0x3258 bytes) 000792 1763508192.342845 read 0x009f23 thru 0x014d7e (0xae5c bytes) 000793 1763508192.343517 write 0x025554 thru 0x0346ab (0xf158 bytes) 000794 1763508192.355783 mapread 0x022394 thru 0x02b56d (0x91da bytes) 000795 1763508192.356624 mapwrite 0x013b7d thru 0x01b68e (0x7b12 bytes) 000797 1763508192.408722 trunc from 0x040000 to 0x00a615 (0x359ea bytes) 000799 1763508192.489120 MIRROR_RESYNC 6 mirrors 000802 1763508193.445869 mapwrite 0x017de9 thru 0x01a061 (0x2279 bytes) 000805 1763508193.595936 mapwrite 0x0344bf thru 0x037ce5 (0x3827 bytes) 000808 1763508193.678270 read 0x006ad9 thru 0x00c614 (0x5b3c bytes) 000809 1763508193.689969 mapwrite 0x0081bc thru 0x0081bd (0x2 bytes) 000810 1763508193.732040 MIRROR_RESYNC 6 mirrors 000811 1763508194.909009 write 0x03fcd8 thru 0x03ffff (0x328 bytes) 000812 1763508194.971909 MIRROR_RESYNC 6 mirrors 000813 1763508196.029226 write 0x0070cc thru 0x014aed (0xda22 bytes) 000814 1763508196.127971 read 0x00e1d6 thru 0x0163e5 (0x8210 bytes) 000815 1763508196.137615 mapread 0x00afbd thru 0x00f8aa (0x48ee bytes) 000816 1763508196.138443 mapwrite 0x00681e thru 0x015599 (0xed7c bytes) 000818 1763508196.238011 read 0x025359 thru 0x025b72 (0x81a bytes) 000820 1763508196.253634 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) Mirror 10: 0xdf759ddd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) Mirror 10: 0xdf759ddd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000822 1763508196.835950 mapread 0x019f08 thru 0x01cced (0x2de6 bytes) 000824 1763508196.861802 mapread 0x038dd4 thru 0x03ffff (0x722c bytes) 000825 1763508196.882360 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000827 1763508197.292573 write 0x012d9c thru 0x0179d3 (0x4c38 bytes) 000829 1763508197.307507 mapwrite 0x01107b thru 0x01c21f (0xb1a5 bytes) 000831 1763508197.403269 write 0x02245a thru 0x027d86 (0x592d bytes) 000832 1763508197.411021 read 0x01029c thru 0x010832 (0x597 bytes) 000834 1763508197.412816 read 0x039127 thru 0x03d53e (0x4418 bytes) 000835 1763508197.426624 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x41d7ff7c Mirror 10: 0x41d7ff7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000836 1763508198.190316 read 0x00f9e4 thru 0x016525 (0x6b42 bytes) 000837 1763508198.210297 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x41d7ff7c Mirror 10: 0x41d7ff7c Mirror 11: 0x41d7ff7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000839 1763508199.080029 mapread 0x0365a0 thru 0x0386f8 (0x2159 bytes) 000840 1763508199.112400 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdf759ddd (stale) Mirror 6: 0xdf759ddd (stale) Mirror 9: 0xdf759ddd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x41d7ff7c Mirror 11: 0x41d7ff7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000842 1763508199.872628 mapread 0x025e98 thru 0x026501 (0x66a bytes) 000846 1763508200.020589 mapread 0x001eb7 thru 0x0044ee (0x2638 bytes) 000847 1763508200.092103 write 0x017b58 thru 0x02061a (0x8ac3 bytes) 000848 1763508200.208650 mapread 0x030bb2 thru 0x03f53e (0xe98d bytes) 000853 1763508200.234369 mapread 0x02d4ce thru 0x037415 (0x9f48 bytes) 000855 1763508200.241289 read 0x01cb14 thru 0x01ddf8 (0x12e5 bytes) 000856 1763508200.244236 trunc from 0x040000 to 0x00b060 (0x34f9f bytes) 000857 1763508200.323101 MIRROR_RESYNC 5 mirrors 000858 1763508201.154871 mapread 0x002cac thru 0x00b060 (0x83b5 bytes) 000860 1763508201.176510 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000861 1763508201.482474 MIRROR_RESYNC 4 mirrors 000862 1763508201.891648 trunc from 0x00b061 to 0x03ef89 (0x33f29 bytes) 000865 1763508201.986635 read 0x02fa8f thru 0x037235 (0x77a7 bytes) 000867 1763508201.996712 write 0x02bd4a thru 0x03a351 (0xe608 bytes) 000868 1763508202.016257 trunc from 0x03ef8a to 0x017ff2 (0x26f97 bytes) 000871 1763508202.051604 trunc from 0x017ff3 to 0x020405 (0x8413 bytes) 000873 1763508202.075210 read 0x00e2b6 thru 0x01980c (0xb557 bytes) 000874 1763508202.087929 mapread 0x01dda2 thru 0x020405 (0x2664 bytes) 000875 1763508202.088916 write 0x03aeb0 thru 0x03ffff (0x5150 bytes) 000876 1763508202.097533 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6bbcf01e (stale) Mirror 11: 0x6bbcf01e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6bbcf01e (stale) Mirror 11: 0x6bbcf01e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000877 1763508202.420760 mapwrite 0x030231 thru 0x03b7dd (0xb5ad bytes) 000879 1763508202.523254 mapread 0x00122c thru 0x009f73 (0x8d48 bytes) 000880 1763508202.537102 write 0x0072c5 thru 0x00d368 (0x60a4 bytes) 000881 1763508202.538299 mapwrite 0x01a017 thru 0x020c45 (0x6c2f bytes) 000882 1763508202.609058 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6bbcf01e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6bbcf01e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000884 1763508202.938579 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) 000885 1763508203.089552 read 0x02e993 thru 0x031552 (0x2bc0 bytes) 000886 1763508203.102981 write 0x03dc01 thru 0x03dd8b (0x18b bytes) 000888 1763508203.110859 mapwrite 0x028231 thru 0x02dcdf (0x5aaf bytes) 000889 1763508203.174697 mapwrite 0x02f6c6 thru 0x034253 (0x4b8e bytes) 000892 1763508203.234277 trunc from 0x040000 to 0x03868d (0x7972 bytes) 000894 1763508203.260345 mapwrite 0x00d75b thru 0x014588 (0x6e2e bytes) 000896 1763508203.338168 write 0x02dbe0 thru 0x0379be (0x9ddf bytes) 000898 1763508203.339929 mapwrite 0x01093f thru 0x01c81c (0xbede bytes) 000900 1763508203.452472 mapread 0x01d6a1 thru 0x01ed32 (0x1692 bytes) 000901 1763508203.453250 read 0x00569d thru 0x00da91 (0x83f5 bytes) 000902 1763508203.465977 read 0x013f62 thru 0x019e17 (0x5eb6 bytes) 000904 1763508203.467007 mapwrite 0x0040fc thru 0x007d12 (0x3c17 bytes) 000906 1763508203.501445 mapread 0x007495 thru 0x009bf3 (0x275f bytes) 000907 1763508203.502193 trunc from 0x03868e to 0x01a283 (0x1e40a bytes) 000908 1763508203.521306 mapwrite 0x030c3e thru 0x0380f3 (0x74b6 bytes) 000909 1763508203.648857 mapwrite 0x03b33f thru 0x03cc28 (0x18ea bytes) 000910 1763508203.722768 trunc from 0x03cc29 to 0x0393ef (0x3839 bytes) 000911 1763508203.748761 read 0x00878e thru 0x017b58 (0xf3cb bytes) 000913 1763508203.749735 mapwrite 0x03af41 thru 0x03fae2 (0x4ba2 bytes) 000915 1763508203.846449 trunc from 0x03fae3 to 0x027104 (0x189de bytes) 000918 1763508203.864936 write 0x01cbcb thru 0x021a0e (0x4e44 bytes) 000920 1763508203.868115 mapread 0x00e2b7 thru 0x01d34f (0xf099 bytes) 000924 1763508203.874949 trunc from 0x027105 to 0x03dd70 (0x16c6c bytes) 000926 1763508203.893347 trunc from 0x03dd71 to 0x00fcb6 (0x2e0ba bytes) 000927 1763508203.917291 mapwrite 0x009303 thru 0x01593c (0xc63a bytes) 000930 1763508204.022776 read 0x00270f thru 0x003f23 (0x1815 bytes) 000932 1763508204.023926 mapread 0x0097be thru 0x00c817 (0x305a bytes) 000936 1763508204.030156 write 0x02ec1a thru 0x03d064 (0xe44b bytes) 000937 1763508204.033282 read 0x01249f thru 0x016bd6 (0x4738 bytes) 000939 1763508204.043330 read 0x00e45b thru 0x01515e (0x6d04 bytes) 000940 1763508204.043987 mapread 0x02cf41 thru 0x039e4c (0xcf0c bytes) 000943 1763508204.045826 mapwrite 0x00a214 thru 0x00d294 (0x3081 bytes) 000944 1763508204.095543 MIRROR_EXTEND to 2 mirrors 000945 1763508204.679715 write 0x00bfdf thru 0x01a8fd (0xe91f bytes) 000948 1763508204.783559 read 0x030ffd thru 0x03d064 (0xc068 bytes) 000949 1763508204.793653 write 0x01bc40 thru 0x01e0be (0x247f bytes) 000954 1763508204.814319 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d065): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d065): Mirror 2: 0x938dfc3 (stale) CRC-32 checksum value for chunk [0, 0x3d065): Mirror 1: 0x1b217ae9 Mirror 3: 0x1b217ae9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d065: skipped 000957 1763508205.688631 mapwrite 0x03b615 thru 0x03f9b0 (0x439c bytes) 000958 1763508205.852944 read 0x010d00 thru 0x01840d (0x770e bytes) 000959 1763508205.865899 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f9b1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9b1): Mirror 2: 0x71d9d947 (stale) Mirror 3: 0xd936ba50 (stale) CRC-32 checksum value for chunk [0, 0x3f9b1): Mirror 1: 0xc7a56439 Mirror 4: 0xc7a56439 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9b1: skipped 000964 1763508206.717188 read 0x029aaf thru 0x032cb5 (0x9207 bytes) 000966 1763508206.741860 mapwrite 0x016041 thru 0x024211 (0xe1d1 bytes) 000967 1763508206.962228 mapwrite 0x004e91 thru 0x00f9c4 (0xab34 bytes) 000968 1763508207.025671 mapwrite 0x03848f thru 0x03ffff (0x7b71 bytes) 000969 1763508207.114882 mapwrite 0x0305a1 thru 0x03b3d9 (0xae39 bytes) 000970 1763508207.173436 write 0x0153f6 thru 0x0216c9 (0xc2d4 bytes) 000971 1763508207.174851 write 0x0304a1 thru 0x03a30f (0x9e6f bytes) 000973 1763508207.181868 trunc from 0x040000 to 0x01acea (0x25315 bytes) 000974 1763508207.201895 read 0x012e3b thru 0x017025 (0x41eb bytes) 000975 1763508207.202667 mapread 0x0156c1 thru 0x01acea (0x562a bytes) 000977 1763508207.205456 mapwrite 0x02f4a2 thru 0x032976 (0x34d5 bytes) 000979 1763508207.282682 trunc from 0x032977 to 0x00c762 (0x26214 bytes) 000980 1763508207.304819 mapread 0x008ecc thru 0x00c762 (0x3897 bytes) 000981 1763508207.306671 mapwrite 0x01aeb1 thru 0x029de2 (0xef32 bytes) 000983 1763508207.459889 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29de3): Mirror 2: 0x12f2a7e3 (stale) Mirror 3: 0xc75e1dea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29de3): Mirror 2: 0x12f2a7e3 (stale) Mirror 3: 0xc75e1dea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29de3: skipped 000984 1763508207.810827 mapread 0x011fa8 thru 0x01a87b (0x88d4 bytes) 000986 1763508207.846758 mapwrite 0x0106f4 thru 0x020501 (0xfe0e bytes) 000990 1763508207.917099 mapwrite 0x032dda thru 0x03ba08 (0x8c2f bytes) 000991 1763508208.047811 read 0x001c2d thru 0x009730 (0x7b04 bytes) 000995 1763508208.059319 mapwrite 0x03b56c thru 0x03ffff (0x4a94 bytes) 000999 1763508208.158017 read 0x009289 thru 0x009a2f (0x7a7 bytes) 001000 1763508208.158612 mapwrite 0x016ac1 thru 0x0248c7 (0xde07 bytes) All operations completed A-OK! PASS 70a (69s) == sanity-flr test 71: check mirror extend parent fid ==== 18:23:32 (1763508212) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 18:23:35 (1763508215) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6129056 .. Extending file size to 6475104 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7898528 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7220320 .. Extending file size to 8150464 .. Extending file size to 7450944 .. Extending file size to 8189088 .. done Extending file size to 7456928 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8177472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8389312 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8620224 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8765312 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 54856 54857 54858 54861 54863 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:24:50 (1763508290) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3031776 .. Extending file size to 6960864 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 3238432 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 3641216 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7428064 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7486336 .. failed Extending file size to 8958176 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7481280 .. Extending file size to 8504928 .. Extending file size to 8647680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8830080 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9292960 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8889856 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56940 56941 56942 56943 56944 failed 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:26:06 (1763508366) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0932347 s, 22.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.339563 s, 6.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7daadcbb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:26:25 (1763508385) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0756548 s, 27.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 18:26:33 (1763508393) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2b9:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:26:40 (1763508400) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206193 s, 10.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:26:49 (1763508409) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229439 s, 9.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:26:57 (1763508417) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220091 s, 9.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:27:31 (1763508451) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108277 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:28:30 (1763508510) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.378409 s, 5.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:29:26 (1763508566) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. Creating new pool test_204f oleg240-server: Pool lustre.test_204f created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334153 s, 6.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg240-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:30:43 (1763508643) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:30:52 (1763508652) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:30:59 (1763508659) oleg240-server: Pool lustre.test_206 created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg240-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:31:22 (1763508682) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820243 s, 12.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360724 s, 29.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1377e4d7 Mirror 3: 0x1377e4d7 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:31:33 (1763508693) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:31:37 (1763508697) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:31:40 (1763508700) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.111112 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1250000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x8a0000 Read this (4 bytes): BBBB 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:32:14 (1763508734) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.02142 s, 382 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:32:21 (1763508741) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537199 s, 19.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x250 lmm_fid: [0x200000404:0x250:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x250 lmm_fid: [0x200000404:0x250:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834512 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-18 18:32:23.000000000 -0500 Modify: 2025-11-18 18:32:23.000000000 -0500 Change: 2025-11-18 18:32:25.000000000 -0500 Birth: 2025-11-18 18:32:23.000000000 -0500 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:32:30 (1763508750) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:32:33 (1763508753) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.555856 s, 73.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0604076 s, 67.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 18 18:32 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x252:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 18 18:32 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:32:41 (1763508761) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:32:49 (1763508769) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1515 sec =========== 18:32:58 (1763508778) === sanity-flr: start cleanup 18:32:59 (1763508779) === === sanity-flr: finish cleanup 18:33:06 (1763508786) ===