-----============= acceptance-small: sanity-flr ============----- Sat Feb 7 01:47:52 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg404-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:48:00 (1770446880) === oleg404-client.virtnet: executing check_config_client /mnt/lustre oleg404-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg404-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a454ad04000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a454ad04000.idle_timeout=debug disable quota as required oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:48:17 (1770446897) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7554] == sanity-flr test 0a: lfs mirror create with -N option == 01:48:18 (1770446898) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:48:28 (1770446908) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:48:30 (1770446910) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:48:32 (1770446912) PASS 0d (15s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:48:47 (1770446927) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:48:49 (1770446929) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 01:48:51 (1770446931) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a454ad04000.stats=clear osc.lustre-OST0001-osc-ffff8a454ad04000.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:48:55 (1770446935) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:49:02 (1770446942) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:49:08 (1770446948) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:49:10 (1770446950) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 01:49:15 (1770446955) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:49:19 (1770446959) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:49:21 (1770446961) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:49:26 (1770446966) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:49:32 (1770446972) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:49:37 (1770446977) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.239535 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987943 s, 10.6 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:49:43 (1770446983) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0964114 s, 87.0 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:49:47 (1770446987) Stopping /mnt/lustre-ost1 (opts:) on oleg404-server oleg404-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg404-server: oleg404-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg404-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg404-server oleg404-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg404-server: oleg404-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg404-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (42s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:50:29 (1770447029) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.140813 s, 37.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg404-server oleg404-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg404-server: oleg404-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg404-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:50:49 (1770447049) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg404-server oleg404-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg404-server: oleg404-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg404-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg404-server oleg404-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg404-server: oleg404-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg404-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (36s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:51:25 (1770447085) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.52981 s, 20.7 MB/s -rw-r--r-- 1 root root 52428800 Feb 7 01:51 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.618s user 0m0.000s sys 0m0.168s Stopping /mnt/lustre-ost1 (opts:) on oleg404-server oleg404-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a454ad04000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.719s user 0m0.000s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg404-server: oleg404-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg404-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg404-server oleg404-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.696s user 0m0.001s sys 0m0.183s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg404-server: oleg404-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg404-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg404-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a454ad04000.ost_server_uuid in FULL state after 0 sec PASS 33b (45s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:52:11 (1770447131) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:52:13 (1770447133) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:52:14 (1770447134) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 01:52:16 (1770447136) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 01:52:20 (1770447140) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.321141 s, 39.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (8s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:52:28 (1770447148) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144871 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302123 s, 34.7 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:52:32 (1770447152) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:52:37 (1770447157) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00130941 s, 782 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000903577 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000663235 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110965 s, 923 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 01:52:42 (1770447162) create test files with size 15753428 .. 66a3aa1de50dc7af943b2a2e9e9a7c26 - ae406ef38276722bce80cd5394655af3 - 2e5425f6ca7666f02091d3426b3c8efe - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg404-client.virtnet /mnt/lustre (opts:) Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 01:52:53 (1770447173) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (8s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:53:01 (1770447181) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:53:04 (1770447184) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954378 s, 11.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179364 s, 58.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 01:53:09 (1770447189) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0574529 s, 36.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.242413 s, 17.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0825924 s, 25.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (10s) == sanity-flr test 42: lfs mirror verify ================= 01:53:20 (1770447200) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 01:53:24 (1770447204) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:53:29 (1770447209) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350068 s, 30.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 01:53:38 (1770447218) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:53:42 (1770447222) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4d:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:53:55 (1770447235) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.325292 s, 32.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg404-client.virtnet /mnt/lustre (opts:) Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:54:06 (1770447246) llite.lustre-ffff8a4550380800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.292018 s, 35.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg404-client.virtnet /mnt/lustre (opts:) Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8a4542c84000.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:54:17 (1770447257) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (26s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:54:42 (1770447282) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (22s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:55:05 (1770447305) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 01:55:07 (1770447307) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.250654 s, 12.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0964462 s, 32.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x598f3bb6 (stale) Mirror 2: 0x598f3bb6 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: d765b6ce4531b882429fd53970c935ac - mirror 1 checksum: d25b9f2efae8c34fc0650e84852aaad6 - mirror 2 checksum: d765b6ce4531b882429fd53970c935ac - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Feb 7 01:55:08 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat Feb 7 01:55:08 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d765b6ce4531b882429fd53970c935ac - mirror 1 checksum: d25b9f2efae8c34fc0650e84852aaad6 - mirror 2 checksum: d25b9f2efae8c34fc0650e84852aaad6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Feb 7 01:55:08 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat Feb 7 01:55:08 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (8s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:55:16 (1770447316) Failing mds1 on oleg404-server Stopping /mnt/lustre-mds1 (opts:) on oleg404-server 01:55:21 (1770447321) shut down facet: mds1 facet_host: oleg404-server facet_failover_host: oleg404-server Failover mds1 to oleg404-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg404-server: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg404-client: oleg404-server: ssh exited with exit code 1 Started lustre-MDT0000 01:55:39 (1770447339) targets are mounted 01:55:39 (1770447339) facet_failover done oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213623 s, 49.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:55:49 (1770447349) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:55:51 (1770447351) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:55:53 (1770447353) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0063029 s, 2.6 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:55:58 (1770447358) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:55:59 (1770447359) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:56:01 (1770447361) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (14s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:56:15 (1770447375) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (23s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:56:39 (1770447399) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:57:00 (1770447420) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:57:21 (1770447441) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 01:57:26 (1770447446) mirror create pid 52264 mirror split pid 52265 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52264 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52265 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 01:58:31 (1770447511) Seed set to 7512 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x2c503 000004 1770447512.836308 trunc from 00000000 to 0x02c502 (0x2c503 bytes) 000005 1770447512.852337 mapwrite 0x01b572 thru 0x0255e5 (0xa074 bytes) 000006 1770447512.910920 read 0x006a5d thru 0x011b35 (0xb0d9 bytes) 000007 1770447512.920439 write 0x016179 thru 0x02268b (0xc513 bytes) 000008 1770447512.923558 read 0x0196dd thru 0x02082e (0x7152 bytes) 000009 1770447512.924096 write 0x02c20a thru 0x02ccc1 (0xab8 bytes) 000010 1770447512.929988 trunc from 0x02ccc2 to 0x010aa8 (0x1c219 bytes) 000012 1770447512.949005 read 0x004d31 thru 0x010aa8 (0xbd78 bytes) 000014 1770447512.953084 write 0x03cfc9 thru 0x03ffff (0x3037 bytes) 000016 1770447512.955948 read 0x00774b thru 0x012700 (0xafb6 bytes) 000018 1770447512.963368 trunc from 0x040000 to 0x02acb6 (0x15349 bytes) 000021 1770447512.979452 read 0x001d26 thru 0x00d710 (0xb9eb bytes) 000023 1770447512.980500 read 0x018f19 thru 0x01e9db (0x5ac3 bytes) 000029 1770447512.990568 MIRROR_EXTEND to 2 mirrors 000031 1770447513.458931 mapread 0x020f16 thru 0x02933b (0x8426 bytes) 000033 1770447513.484470 mapread 0x017e55 thru 0x026c54 (0xee00 bytes) 000034 1770447513.504978 write 0x0221a5 thru 0x031def (0xfc4b bytes) 000036 1770447513.567382 mapwrite 0x022e49 thru 0x030e39 (0xdff1 bytes) 000037 1770447513.618540 MIRROR_RESYNC 2 mirrors 000038 1770447514.030009 MIRROR_EXTEND to 3 mirrors 000039 1770447514.562749 read 0x02f6c5 thru 0x031def (0x272b bytes) 000041 1770447514.591363 write 0x031cad thru 0x03279c (0xaf0 bytes) 000043 1770447514.673962 trunc from 0x03279d to 0x000482 (0x3231a bytes) 000045 1770447514.709705 mapread 0x000243 thru 0x000482 (0x240 bytes) 000046 1770447514.715803 trunc from 0x000483 to 0x007662 (0x71e0 bytes) 000047 1770447514.728288 mapread 0x006510 thru 0x007662 (0x1153 bytes) 000050 1770447514.735812 write 0x03c465 thru 0x03ffff (0x3b9b bytes) 000051 1770447514.740783 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xea0f2e87 (stale) Mirror 3: 0xea0f2e87 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x424f6640 Mirror 4: 0x424f6640 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000052 1770447515.223577 write 0x0357e7 thru 0x038c75 (0x348f bytes) 000053 1770447515.276900 mapread 0x0016a9 thru 0x0114cd (0xfe25 bytes) 000054 1770447515.295246 write 0x008dd8 thru 0x00d906 (0x4b2f bytes) 000058 1770447515.297465 read 0x0268c1 thru 0x02bc23 (0x5363 bytes) 000059 1770447515.310061 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xea0f2e87 (stale) Mirror 3: 0xea0f2e87 (stale) Mirror 4: 0x424f6640 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad1810a6 Mirror 5: 0xad1810a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000060 1770447515.827350 mapread 0x016289 thru 0x017e46 (0x1bbe bytes) 000064 1770447515.837320 mapread 0x01b730 thru 0x01ce2b (0x16fc bytes) truncating to largest ever: 0x3f68d 000066 1770447515.844399 trunc from 0x040000 to 0x03f68c (0x973 bytes) 000067 1770447515.929754 mapwrite 0x02b96f thru 0x02cc53 (0x12e5 bytes) 000069 1770447516.012803 trunc from 0x03f68d to 0x003b4d (0x3bb3f bytes) 000072 1770447516.031862 write 0x004ff7 thru 0x012528 (0xd532 bytes) 000073 1770447516.033616 trunc from 0x012529 to 0x00981a (0x8d0e bytes) 000076 1770447516.047913 MIRROR_RESYNC 5 mirrors 000077 1770447516.893672 trunc from 0x00981b to 0x013945 (0xa12b bytes) 000079 1770447516.983484 write 0x01d737 thru 0x01fa73 (0x233d bytes) 000080 1770447516.984690 read 0x01e556 thru 0x01fa73 (0x151e bytes) 000081 1770447516.989769 mapwrite 0x03a126 thru 0x03ffff (0x5eda bytes) 000082 1770447517.123564 mapread 0x00f5f7 thru 0x013e90 (0x489a bytes) 000083 1770447517.156736 mapread 0x012034 thru 0x01aabe (0x8a8b bytes) 000084 1770447517.190669 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd54b256 (stale) Mirror 3: 0xd54b256 (stale) Mirror 4: 0xd54b256 (stale) Mirror 5: 0xd54b256 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e2f47c5 Mirror 6: 0x9e2f47c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000086 1770447517.966178 MIRROR_RESYNC 6 mirrors 000089 1770447518.866177 write 0x02bc07 thru 0x02d525 (0x191f bytes) 000090 1770447518.970423 trunc from 0x040000 to 0x0076cb (0x38934 bytes) 000091 1770447519.031152 MIRROR_RESYNC 6 mirrors 000093 1770447519.931843 read 0x00321c thru 0x0076cb (0x44b0 bytes) 000094 1770447519.940964 read 0x004549 thru 0x0076cb (0x3183 bytes) 000096 1770447519.941380 trunc from 0x0076cc to 0x012618 (0xaf4d bytes) 000097 1770447520.002862 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12619): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12619): Mirror 2: 0xf8b8761c (stale) Mirror 3: 0xf8b8761c (stale) Mirror 4: 0xf8b8761c (stale) Mirror 5: 0xf8b8761c (stale) Mirror 6: 0xf8b8761c (stale) CRC-32 checksum value for chunk [0, 0x12619): Mirror 1: 0xf8b8761c Mirror 7: 0xf8b8761c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12619: skipped 000098 1770447520.908020 mapread 0x0105ea thru 0x012618 (0x202f bytes) 000099 1770447520.968483 mapwrite 0x00285f thru 0x0100e5 (0xd887 bytes) 000100 1770447521.241166 write 0x032437 thru 0x03ffff (0xdbc9 bytes) 000103 1770447521.253833 write 0x0349af thru 0x03ffff (0xb651 bytes) 000108 1770447521.263220 write 0x03ff02 thru 0x03ffff (0xfe bytes) 000109 1770447521.273429 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x268b38ca (stale) Mirror 3: 0x268b38ca (stale) Mirror 4: 0x268b38ca (stale) Mirror 5: 0x268b38ca (stale) Mirror 6: 0x268b38ca (stale) Mirror 7: 0x268b38ca (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x76e8ed8b Mirror 8: 0x76e8ed8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000111 1770447522.119705 mapwrite 0x013cd9 thru 0x01acea (0x7012 bytes) 000115 1770447522.260181 trunc from 0x040000 to 0x01e53e (0x21ac1 bytes) 000116 1770447522.294473 read 0x01ca25 thru 0x01e53e (0x1b1a bytes) 000117 1770447522.308308 MIRROR_RESYNC 8 mirrors 000119 1770447523.544190 read 0x0037c3 thru 0x011ebc (0xe6fa bytes) 000120 1770447523.548887 mapread 0x00048d thru 0x00be24 (0xb998 bytes) 000121 1770447523.556894 read 0x0053bd thru 0x009a51 (0x4695 bytes) 000123 1770447523.557380 read 0x0046d9 thru 0x00a3b7 (0x5cdf bytes) 000124 1770447523.557563 write 0x00ccda thru 0x0112e0 (0x4607 bytes) 000127 1770447523.623576 read 0x00bd61 thru 0x00fe85 (0x4125 bytes) 000130 1770447523.626247 mapread 0x001505 thru 0x00c491 (0xaf8d bytes) 000132 1770447523.632273 MIRROR_RESYNC 8 mirrors 000133 1770447524.653499 mapread 0x018a61 thru 0x01b807 (0x2da7 bytes) 000134 1770447524.667438 mapread 0x0016d3 thru 0x004627 (0x2f55 bytes) 000135 1770447524.674654 mapread 0x00126f thru 0x007f7f (0x6d11 bytes) 000136 1770447524.675589 mapwrite 0x039e19 thru 0x03a6bf (0x8a7 bytes) 000138 1770447524.801548 mapwrite 0x03245e thru 0x0348a0 (0x2443 bytes) 000139 1770447524.838226 trunc from 0x03a6c0 to 0x01edfc (0x1b8c3 bytes) 000140 1770447524.862993 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1edfd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1edfd): Mirror 2: 0x870e0c74 (stale) Mirror 3: 0x870e0c74 (stale) Mirror 4: 0x870e0c74 (stale) Mirror 5: 0x870e0c74 (stale) Mirror 6: 0x870e0c74 (stale) Mirror 7: 0x870e0c74 (stale) Mirror 8: 0x870e0c74 (stale) CRC-32 checksum value for chunk [0, 0x1edfd): Mirror 1: 0x870e0c74 Mirror 9: 0x870e0c74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1edfd: skipped 000143 1770447525.610312 mapread 0x019fc6 thru 0x01edfc (0x4e37 bytes) 000144 1770447525.635861 mapread 0x01b3e5 thru 0x01d0c5 (0x1ce1 bytes) 000147 1770447525.662876 read 0x006687 thru 0x006c84 (0x5fe bytes) 000148 1770447525.673403 trunc from 0x01edfd to 0x007998 (0x17464 bytes) 000150 1770447525.746840 write 0x010359 thru 0x01981a (0x94c2 bytes) 000151 1770447525.749067 mapwrite 0x019c7a thru 0x028bc9 (0xef50 bytes) 000152 1770447525.885887 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x28bca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28bca): Mirror 2: 0xc4f0a6db (stale) Mirror 3: 0xc4f0a6db (stale) Mirror 4: 0xc4f0a6db (stale) Mirror 5: 0xc4f0a6db (stale) Mirror 6: 0xc4f0a6db (stale) Mirror 7: 0xc4f0a6db (stale) Mirror 8: 0xc4f0a6db (stale) Mirror 9: 0xc4f0a6db (stale) CRC-32 checksum value for chunk [0, 0x28bca): Mirror 1: 0xc2f0e4da Mirror 10: 0xc2f0e4da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28bca: skipped 000153 1770447526.583813 MIRROR_RESYNC 10 mirrors 000154 1770447527.669029 MIRROR_RESYNC 10 mirrors 000155 1770447528.025894 mapwrite 0x00d49f thru 0x01a638 (0xd19a bytes) 000156 1770447528.189562 read 0x018734 thru 0x021219 (0x8ae6 bytes) 000157 1770447528.195592 mapread 0x003ecc thru 0x009070 (0x51a5 bytes) 000158 1770447528.205792 mapwrite 0x00a2ee thru 0x014cbf (0xa9d2 bytes) 000160 1770447528.251530 mapread 0x009f8c thru 0x00b221 (0x1296 bytes) 000161 1770447528.252865 write 0x03b825 thru 0x03ffff (0x47db bytes) 000163 1770447528.254603 trunc from 0x040000 to 0x02297c (0x1d683 bytes) 000164 1770447528.276980 write 0x01ffa1 thru 0x0276f5 (0x7755 bytes) 000165 1770447528.282329 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x276f6): Mirror 2: 0x26d7ef93 (stale) Mirror 3: 0x26d7ef93 (stale) Mirror 4: 0x26d7ef93 (stale) Mirror 5: 0x26d7ef93 (stale) Mirror 6: 0x26d7ef93 (stale) Mirror 8: 0x26d7ef93 (stale) Mirror 9: 0x26d7ef93 (stale) Mirror 10: 0x26d7ef93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x276f6): Mirror 2: 0x26d7ef93 (stale) Mirror 3: 0x26d7ef93 (stale) Mirror 4: 0x26d7ef93 (stale) Mirror 5: 0x26d7ef93 (stale) Mirror 6: 0x26d7ef93 (stale) Mirror 8: 0x26d7ef93 (stale) Mirror 9: 0x26d7ef93 (stale) Mirror 10: 0x26d7ef93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x276f6: skipped 000167 1770447528.609566 mapwrite 0x023621 thru 0x02cbbd (0x959d bytes) 000168 1770447528.685258 read 0x014e8b thru 0x019d37 (0x4ead bytes) 000169 1770447528.690516 write 0x02a110 thru 0x02f3ba (0x52ab bytes) 000170 1770447528.691957 mapread 0x01a90b thru 0x026e98 (0xc58e bytes) 000172 1770447528.707015 mapread 0x01625b thru 0x025017 (0xedbd bytes) 000173 1770447528.708023 mapwrite 0x009e7b thru 0x013683 (0x9809 bytes) 000174 1770447528.764170 read 0x00ee79 thru 0x00ff11 (0x1099 bytes) 000177 1770447528.765577 mapread 0x0079f5 thru 0x016ff7 (0xf603 bytes) 000178 1770447528.779689 read 0x01ba83 thru 0x02ab53 (0xf0d1 bytes) 000180 1770447528.780997 write 0x003579 thru 0x00630c (0x2d94 bytes) 000181 1770447528.782840 trunc from 0x02f3bb to 0x022e96 (0xc524 bytes) 000184 1770447528.798969 read 0x002efe thru 0x009be6 (0x6ce9 bytes) 000190 1770447528.809245 mapread 0x00d68e thru 0x01467f (0x6ff2 bytes) 000191 1770447528.814119 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22e97): Mirror 3: 0x94f18c0f (stale) Mirror 4: 0x94f18c0f (stale) Mirror 5: 0x94f18c0f (stale) Mirror 6: 0x94f18c0f (stale) Mirror 8: 0x94f18c0f (stale) Mirror 9: 0x94f18c0f (stale) Mirror 10: 0x94f18c0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e97): Mirror 3: 0x94f18c0f (stale) Mirror 4: 0x94f18c0f (stale) Mirror 5: 0x94f18c0f (stale) Mirror 6: 0x94f18c0f (stale) Mirror 8: 0x94f18c0f (stale) Mirror 9: 0x94f18c0f (stale) Mirror 10: 0x94f18c0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e97: skipped 000194 1770447529.133877 write 0x01a699 thru 0x026ff9 (0xc961 bytes) 000195 1770447529.138984 mapwrite 0x0305c7 thru 0x033413 (0x2e4d bytes) 000196 1770447529.192967 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33414): Mirror 4: 0x5c3a8808 (stale) Mirror 5: 0x5c3a8808 (stale) Mirror 6: 0x5c3a8808 (stale) Mirror 8: 0x5c3a8808 (stale) Mirror 9: 0x5c3a8808 (stale) Mirror 10: 0x5c3a8808 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33414): Mirror 4: 0x5c3a8808 (stale) Mirror 5: 0x5c3a8808 (stale) Mirror 6: 0x5c3a8808 (stale) Mirror 8: 0x5c3a8808 (stale) Mirror 9: 0x5c3a8808 (stale) Mirror 10: 0x5c3a8808 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33414: skipped 000198 1770447529.552547 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33414): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33414): Mirror 4: 0x5c3a8808 (stale) Mirror 5: 0x5c3a8808 (stale) Mirror 6: 0x5c3a8808 (stale) Mirror 8: 0x5c3a8808 (stale) Mirror 9: 0x5c3a8808 (stale) Mirror 10: 0x5c3a8808 (stale) CRC-32 checksum value for chunk [0, 0x33414): Mirror 1: 0xa1f13d3e Mirror 11: 0xa1f13d3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33414: skipped 000201 1770447530.274843 write 0x037cd8 thru 0x03ffff (0x8328 bytes) 000204 1770447530.374494 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c3e6776 (stale) Mirror 5: 0x5c3e6776 (stale) Mirror 6: 0x5c3e6776 (stale) Mirror 8: 0x5c3e6776 (stale) Mirror 10: 0x5c3e6776 (stale) Mirror 11: 0x227fef2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5c3e6776 (stale) Mirror 5: 0x5c3e6776 (stale) Mirror 6: 0x5c3e6776 (stale) Mirror 8: 0x5c3e6776 (stale) Mirror 10: 0x5c3e6776 (stale) Mirror 11: 0x227fef2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000206 1770447530.743063 read 0x03660b thru 0x03ffff (0x99f5 bytes) 000207 1770447530.748861 trunc from 0x040000 to 0x00f132 (0x30ecd bytes) 000208 1770447530.794965 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf133): Mirror 4: 0x9ed056c2 (stale) Mirror 5: 0x9ed056c2 (stale) Mirror 6: 0x9ed056c2 (stale) Mirror 8: 0x9ed056c2 (stale) Mirror 10: 0x9ed056c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf133): Mirror 4: 0x9ed056c2 (stale) Mirror 5: 0x9ed056c2 (stale) Mirror 6: 0x9ed056c2 (stale) Mirror 8: 0x9ed056c2 (stale) Mirror 10: 0x9ed056c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf133: skipped 000211 1770447531.068325 read 0x00a8fa thru 0x00f132 (0x4839 bytes) 000212 1770447531.074058 read 0x00aa87 thru 0x00f132 (0x46ac bytes) 000214 1770447531.074845 mapwrite 0x0392f4 thru 0x03e428 (0x5135 bytes) 000215 1770447531.132592 trunc from 0x03e429 to 0x00cfa7 (0x31481 bytes) 000217 1770447531.146593 mapwrite 0x00565d thru 0x010ca3 (0xb647 bytes) 000218 1770447531.212960 mapwrite 0x009b86 thru 0x01609b (0xc516 bytes) 000219 1770447531.288074 mapread 0x0106be thru 0x013884 (0x31c7 bytes) 000220 1770447531.292216 MIRROR_RESYNC 6 mirrors 000222 1770447531.988353 mapread 0x013932 thru 0x01609b (0x276a bytes) 000223 1770447532.012016 trunc from 0x01609c to 0x03326a (0x1d1cf bytes) 000225 1770447532.101236 read 0x000290 thru 0x0082bb (0x802c bytes) 000227 1770447532.110881 mapread 0x026433 thru 0x03035b (0x9f29 bytes) 000230 1770447532.130279 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3326b): Mirror 4: 0x431609ad (stale) Mirror 5: 0x431609ad (stale) Mirror 8: 0x431609ad (stale) Mirror 10: 0x431609ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3326b): Mirror 4: 0x431609ad (stale) Mirror 5: 0x431609ad (stale) Mirror 8: 0x431609ad (stale) Mirror 10: 0x431609ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3326b: skipped 000232 1770447532.456361 read 0x00cb97 thru 0x00d33a (0x7a4 bytes) 000234 1770447532.464392 read 0x02fedf thru 0x03326a (0x338c bytes) 000235 1770447532.473696 mapread 0x01b4ef thru 0x01e118 (0x2c2a bytes) 000236 1770447532.474594 trunc from 0x03326b to 0x02c72a (0x6b40 bytes) 000237 1770447532.486493 read 0x0183e5 thru 0x02048e (0x80aa bytes) 000238 1770447532.487316 mapwrite 0x01c836 thru 0x0286a6 (0xbe71 bytes) 000239 1770447532.533892 write 0x02651c thru 0x0343a5 (0xde8a bytes) 000242 1770447532.539212 MIRROR_RESYNC 5 mirrors 000244 1770447533.132591 write 0x020dee thru 0x022e86 (0x2099 bytes) 000245 1770447533.190547 read 0x017171 thru 0x025a14 (0xe8a4 bytes) 000246 1770447533.198314 read 0x024976 thru 0x0274a4 (0x2b2f bytes) 000248 1770447533.205117 read 0x02388e thru 0x02ab7c (0x72ef bytes) 000249 1770447533.207993 MIRROR_RESYNC 5 mirrors 000250 1770447533.896207 read 0x020417 thru 0x0254b5 (0x509f bytes) 000251 1770447533.902485 mapread 0x02570c thru 0x02a8ef (0x51e4 bytes) 000252 1770447533.916987 mapwrite 0x020564 thru 0x021a91 (0x152e bytes) 000253 1770447534.031429 mapwrite 0x01de29 thru 0x024a6a (0x6c42 bytes) 000256 1770447534.069975 read 0x00f0d1 thru 0x01816d (0x909d bytes) 000257 1770447534.084063 mapread 0x00aff6 thru 0x00b899 (0x8a4 bytes) 000258 1770447534.092729 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x343a6): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x343a6): Mirror 4: 0xb1fecd80 (stale) Mirror 5: 0xb1fecd80 (stale) Mirror 8: 0xb1fecd80 (stale) Mirror 10: 0xb1fecd80 (stale) CRC-32 checksum value for chunk [0, 0x343a6): Mirror 1: 0x3cd2e3b7 Mirror 11: 0x3cd2e3b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x343a6: skipped 000262 1770447534.673417 read 0x01505b thru 0x0203b3 (0xb359 bytes) 000263 1770447534.682543 mapwrite 0x0062a6 thru 0x00a21f (0x3f7a bytes) 000264 1770447534.817493 mapread 0x0101c7 thru 0x01d62c (0xd466 bytes) 000265 1770447534.831103 trunc from 0x0343a6 to 0x01c342 (0x18063 bytes) 000266 1770447534.886490 mapwrite 0x03baf4 thru 0x03ffff (0x450c bytes) 000267 1770447534.978732 trunc from 0x040000 to 0x023f5d (0x1c0a2 bytes) 000269 1770447534.992422 read 0x0218f3 thru 0x021b0d (0x21b bytes) truncating to largest ever: 0x3fc01 000270 1770447535.004260 trunc from 0x023f5e to 0x03fc00 (0x1bca3 bytes) 000272 1770447535.041961 trunc from 0x03fc01 to 0x0293b0 (0x16850 bytes) 000274 1770447535.063349 mapread 0x027fce thru 0x0293b0 (0x13e3 bytes) 000278 1770447535.073468 trunc from 0x0293b1 to 0x02b3b4 (0x2004 bytes) 000279 1770447535.093443 read 0x0228d4 thru 0x02b3b4 (0x8ae1 bytes) 000280 1770447535.100028 read 0x002ac7 thru 0x009a8b (0x6fc5 bytes) 000281 1770447535.100473 read 0x0234d0 thru 0x02a9a7 (0x74d8 bytes) 000282 1770447535.100931 write 0x028f0a thru 0x031d37 (0x8e2e bytes) 000283 1770447535.110688 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31d38): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d38): Mirror 4: 0xb30c8ce2 (stale) Mirror 5: 0xb30c8ce2 (stale) Mirror 8: 0xb30c8ce2 (stale) Mirror 10: 0xb30c8ce2 (stale) Mirror 11: 0x42319b95 (stale) CRC-32 checksum value for chunk [0, 0x31d38): Mirror 1: 0x6a272208 Mirror 12: 0x6a272208 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d38: skipped 000284 1770447535.847106 mapread 0x00f543 thru 0x018127 (0x8be5 bytes) 000285 1770447535.858760 mapread 0x01e7c9 thru 0x02524d (0x6a85 bytes) 000286 1770447535.866372 mapread 0x023458 thru 0x023ddf (0x988 bytes) 000287 1770447535.869611 mapread 0x002ae1 thru 0x005478 (0x2998 bytes) 000288 1770447535.877808 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31d38): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d38): Mirror 4: 0xb30c8ce2 (stale) Mirror 5: 0xb30c8ce2 (stale) Mirror 8: 0xb30c8ce2 (stale) Mirror 10: 0xb30c8ce2 (stale) Mirror 11: 0x42319b95 (stale) CRC-32 checksum value for chunk [0, 0x31d38): Mirror 1: 0x6a272208 Mirror 12: 0x6a272208 Mirror 13: 0x6a272208 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d38: skipped 000291 1770447536.434883 trunc from 0x031d38 to 0x02984c (0x84eb bytes) 000292 1770447536.546229 write 0x01e187 thru 0x01e60f (0x489 bytes) 000293 1770447536.559553 write 0x01aaef thru 0x01f880 (0x4d92 bytes) 000294 1770447536.588833 write 0x01f6ac thru 0x02c868 (0xd1bd bytes) 000295 1770447536.592272 write 0x0186a8 thru 0x01a07c (0x19d5 bytes) 000299 1770447536.596459 write 0x010a92 thru 0x01bff9 (0xb568 bytes) 000300 1770447536.610061 trunc from 0x02c869 to 0x01adc0 (0x11aa8 bytes) 000303 1770447536.654390 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1adc1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1adc1): Mirror 4: 0x8f370cea (stale) Mirror 5: 0x8f370cea (stale) Mirror 8: 0x8f370cea (stale) Mirror 10: 0x8f370cea (stale) Mirror 11: 0x8f370cea (stale) Mirror 12: 0xc99441bd (stale) Mirror 13: 0xc99441bd (stale) CRC-32 checksum value for chunk [0, 0x1adc1): Mirror 1: 0x15ca2b57 Mirror 14: 0x15ca2b57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1adc1: skipped 000304 1770447537.494033 mapread 0x016f3c thru 0x01adc0 (0x3e85 bytes) 000305 1770447537.524356 trunc from 0x01adc1 to 0x00f657 (0xb769 bytes) 000306 1770447537.614036 write 0x027755 thru 0x033e1b (0xc6c7 bytes) 000307 1770447537.615462 trunc from 0x033e1c to 0x01a7d1 (0x1964a bytes) 000308 1770447537.628347 trunc from 0x01a7d2 to 0x000557 (0x1a27a bytes) 000309 1770447537.648034 read 0x0002d5 thru 0x000557 (0x283 bytes) 000310 1770447537.652930 read 0x000328 thru 0x000557 (0x230 bytes) 000312 1770447537.654711 write 0x010b8a thru 0x016ce8 (0x615f bytes) 000313 1770447537.655815 write 0x001844 thru 0x001fa5 (0x762 bytes) 000314 1770447537.667245 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x16ce9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16ce9): Mirror 4: 0x39890d9c (stale) Mirror 5: 0x39890d9c (stale) Mirror 8: 0x39890d9c (stale) Mirror 10: 0x39890d9c (stale) Mirror 11: 0x39890d9c (stale) Mirror 12: 0xd6667d30 (stale) Mirror 13: 0xd6667d30 (stale) Mirror 14: 0xdb678b67 (stale) CRC-32 checksum value for chunk [0, 0x16ce9): Mirror 1: 0xe981ffd0 Mirror 15: 0xe981ffd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16ce9: skipped 000315 1770447538.300854 trunc from 0x016ce9 to 0x0257c3 (0xeadb bytes) 000318 1770447538.381232 read 0x00f18b thru 0x01e402 (0xf278 bytes) 000320 1770447538.394732 mapwrite 0x00e7f0 thru 0x01bc78 (0xd489 bytes) 000321 1770447538.431153 trunc from 0x0257c4 to 0x026151 (0x98e bytes) 000322 1770447538.450836 MIRROR_RESYNC 10 mirrors 000323 1770447539.432260 write 0x0295ca thru 0x02fce2 (0x6719 bytes) 000324 1770447539.488045 trunc from 0x02fce3 to 0x03e373 (0xe691 bytes) 000326 1770447539.509865 mapwrite 0x01ea71 thru 0x025eb5 (0x7445 bytes) 000327 1770447539.566439 mapread 0x01239a thru 0x0173bf (0x5026 bytes) 000328 1770447539.573318 write 0x01c699 thru 0x01d7c6 (0x112e bytes) 000330 1770447539.575012 trunc from 0x03e374 to 0x028316 (0x1605d bytes) 000332 1770447539.587809 read 0x017d61 thru 0x0226c3 (0xa963 bytes) 000333 1770447539.588369 trunc from 0x028317 to 0x008ce9 (0x1f62d bytes) 000334 1770447539.613321 mapwrite 0x02b07c thru 0x031805 (0x678a bytes) 000335 1770447539.698830 mapread 0x00f031 thru 0x01cae2 (0xdab2 bytes) 000339 1770447539.717111 mapread 0x00d379 thru 0x00d972 (0x5fa bytes) 000340 1770447539.718034 read 0x022e87 thru 0x031805 (0xe97f bytes) 000343 1770447539.725711 mapwrite 0x0108d7 thru 0x01b3bd (0xaae7 bytes) 000344 1770447539.767058 read 0x02e24a thru 0x031805 (0x35bc bytes) 000345 1770447539.767591 mapwrite 0x0376fd thru 0x03ffff (0x8903 bytes) 000346 1770447539.831883 write 0x0163dc thru 0x0182f5 (0x1f1a bytes) 000347 1770447539.832841 write 0x01a165 thru 0x0209ed (0x6889 bytes) 000349 1770447539.835303 trunc from 0x040000 to 0x005a33 (0x3a5cc bytes) 000350 1770447539.853648 read 0x001ddd thru 0x005a33 (0x3c57 bytes) 000352 1770447539.855673 mapwrite 0x029e01 thru 0x030370 (0x6570 bytes) 000354 1770447539.945400 trunc from 0x030371 to 0x005951 (0x2aa1f bytes) 000356 1770447539.982150 MIRROR_RESYNC 10 mirrors 000357 1770447540.863829 mapread 0x0022c7 thru 0x005951 (0x368b bytes) 000359 1770447540.872859 read 0x000dfc thru 0x003954 (0x2b59 bytes) 000360 1770447540.875550 trunc from 0x005952 to 0x00b1b1 (0x5860 bytes) 000361 1770447540.931182 mapread 0x0071d2 thru 0x00b1b1 (0x3fe0 bytes) 000362 1770447540.942447 MIRROR_RESYNC 10 mirrors 000363 1770447542.171128 mapread 0x0042e4 thru 0x009b89 (0x58a6 bytes) 000364 1770447542.190191 trunc from 0x00b1b2 to 0x02ed3d (0x23b8c bytes) 000367 1770447542.267032 mapwrite 0x005890 thru 0x006d80 (0x14f1 bytes) 000368 1770447542.372997 mapread 0x002424 thru 0x003bc5 (0x17a2 bytes) 000370 1770447542.374188 write 0x02c3ff thru 0x033434 (0x7036 bytes) 000371 1770447542.381714 trunc from 0x033435 to 0x03984f (0x641b bytes) 000375 1770447542.402017 read 0x026823 thru 0x028fd7 (0x27b5 bytes) 000376 1770447542.402904 mapwrite 0x026f7f thru 0x036414 (0xf496 bytes) 000381 1770447542.466129 read 0x00a8b5 thru 0x01913a (0xe886 bytes) 000383 1770447542.484956 MIRROR_RESYNC 10 mirrors 000384 1770447543.444214 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) 000385 1770447543.872043 read 0x0023ed thru 0x005236 (0x2e4a bytes) 000387 1770447543.877025 mapread 0x002074 thru 0x0117a3 (0xf730 bytes) 000388 1770447543.908380 trunc from 0x039850 to 0x020665 (0x191ea bytes) 000389 1770447543.995098 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x20666): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20666): Mirror 4: 0xe929220 (stale) Mirror 5: 0xe929220 (stale) Mirror 8: 0xe929220 (stale) Mirror 10: 0xe929220 (stale) Mirror 11: 0xe929220 (stale) Mirror 12: 0xe929220 (stale) Mirror 13: 0xe929220 (stale) Mirror 14: 0xe929220 (stale) CRC-32 checksum value for chunk [0, 0x20666): Mirror 1: 0xe929220 Mirror 15: 0xe929220 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20666: skipped 000391 1770447544.719219 mapread 0x01688b thru 0x020665 (0x9ddb bytes) 000392 1770447544.762740 mapread 0x011013 thru 0x01380c (0x27fa bytes) 000395 1770447544.798203 write 0x02d8b8 thru 0x02e6c2 (0xe0b bytes) 000398 1770447544.900806 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e6c3): Mirror 4: 0xc4e1270f (stale) Mirror 8: 0xc4e1270f (stale) Mirror 10: 0xc4e1270f (stale) Mirror 11: 0xc4e1270f (stale) Mirror 12: 0xc4e1270f (stale) Mirror 13: 0xc4e1270f (stale) Mirror 14: 0xc4e1270f (stale) Mirror 15: 0xd7393d58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e6c3): Mirror 4: 0xc4e1270f (stale) Mirror 8: 0xc4e1270f (stale) Mirror 10: 0xc4e1270f (stale) Mirror 11: 0xc4e1270f (stale) Mirror 12: 0xc4e1270f (stale) Mirror 13: 0xc4e1270f (stale) Mirror 14: 0xc4e1270f (stale) Mirror 15: 0xd7393d58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e6c3: skipped 000399 1770447545.276855 mapread 0x024906 thru 0x026a86 (0x2181 bytes) 000400 1770447545.299049 read 0x025848 thru 0x0284de (0x2c97 bytes) 000404 1770447545.304769 trunc from 0x02e6c3 to 0x00ae90 (0x23832 bytes) 000405 1770447545.319932 read 0x0056db thru 0x00ae3e (0x5764 bytes) 000406 1770447545.330923 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xae91): Mirror 4: 0x486885b9 (stale) Mirror 8: 0x486885b9 (stale) Mirror 11: 0x486885b9 (stale) Mirror 12: 0x486885b9 (stale) Mirror 13: 0x486885b9 (stale) Mirror 14: 0x486885b9 (stale) Mirror 15: 0x486885b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae91): Mirror 4: 0x486885b9 (stale) Mirror 8: 0x486885b9 (stale) Mirror 11: 0x486885b9 (stale) Mirror 12: 0x486885b9 (stale) Mirror 13: 0x486885b9 (stale) Mirror 14: 0x486885b9 (stale) Mirror 15: 0x486885b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae91: skipped 000407 1770447545.618124 write 0x03fde2 thru 0x03ffff (0x21e bytes) 000409 1770447545.619507 write 0x03160c thru 0x038932 (0x7327 bytes) 000411 1770447545.629116 read 0x03468c thru 0x03ffff (0xb974 bytes) 000415 1770447545.635663 mapread 0x024414 thru 0x02569b (0x1288 bytes) 000416 1770447545.643160 trunc from 0x040000 to 0x02befa (0x14105 bytes) 000417 1770447545.657181 mapwrite 0x00419f thru 0x00f112 (0xaf74 bytes) 000418 1770447545.714359 write 0x030aad thru 0x0340f6 (0x364a bytes) 000420 1770447545.718465 read 0x004b32 thru 0x00f378 (0xa847 bytes) 000421 1770447545.722407 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x340f7): Mirror 8: 0xfe289b9d (stale) Mirror 11: 0xfe289b9d (stale) Mirror 12: 0xfe289b9d (stale) Mirror 13: 0xfe289b9d (stale) Mirror 14: 0xfe289b9d (stale) Mirror 15: 0xad1ba2df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340f7): Mirror 8: 0xfe289b9d (stale) Mirror 11: 0xfe289b9d (stale) Mirror 12: 0xfe289b9d (stale) Mirror 13: 0xfe289b9d (stale) Mirror 14: 0xfe289b9d (stale) Mirror 15: 0xad1ba2df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340f7: skipped 000423 1770447546.073591 write 0x0035b1 thru 0x0101bf (0xcc0f bytes) 000424 1770447546.082061 mapwrite 0x0002b5 thru 0x00328b (0x2fd7 bytes) 000426 1770447546.145420 mapwrite 0x028f7e thru 0x031b31 (0x8bb4 bytes) 000427 1770447546.201725 trunc from 0x0340f7 to 0x03fa94 (0xb99e bytes) 000431 1770447546.217157 trunc from 0x03fa95 to 0x02b322 (0x14772 bytes) 000433 1770447546.235146 mapwrite 0x004bc4 thru 0x00d3be (0x87fb bytes) 000436 1770447546.276599 MIRROR_RESYNC 7 mirrors 000437 1770447547.244323 trunc from 0x02b323 to 0x01d8be (0xda64 bytes) 000443 1770447547.325559 write 0x0054f3 thru 0x0136e9 (0xe1f7 bytes) 000444 1770447547.334330 write 0x036306 thru 0x03ffff (0x9cfa bytes) 000445 1770447547.335851 mapwrite 0x03c8ce thru 0x03ffff (0x3732 bytes) 000447 1770447547.389478 trunc from 0x040000 to 0x005329 (0x3acd6 bytes) 000448 1770447547.409562 MIRROR_RESYNC 7 mirrors 000449 1770447548.211124 write 0x0337e9 thru 0x03ffff (0xc817 bytes) 000452 1770447548.280084 read 0x03e042 thru 0x03ffff (0x1fbe bytes) 000454 1770447548.281513 write 0x01f327 thru 0x02c4cb (0xd1a5 bytes) 000455 1770447548.293828 write 0x03c7e3 thru 0x03ffff (0x381d bytes) 000457 1770447548.295427 mapwrite 0x037182 thru 0x03fe30 (0x8caf bytes) 000458 1770447548.362464 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3bd4e136 (stale) Mirror 12: 0x3bd4e136 (stale) Mirror 13: 0x3bd4e136 (stale) Mirror 14: 0x3bd4e136 (stale) Mirror 15: 0x3bd4e136 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3bd4e136 (stale) Mirror 12: 0x3bd4e136 (stale) Mirror 13: 0x3bd4e136 (stale) Mirror 14: 0x3bd4e136 (stale) Mirror 15: 0x3bd4e136 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000460 1770447548.682252 read 0x007f74 thru 0x00e2ae (0x633b bytes) 000461 1770447548.689592 trunc from 0x040000 to 0x001dfe (0x3e201 bytes) 000463 1770447548.705345 read 0x00026e thru 0x000b7a (0x90d bytes) 000464 1770447548.718648 trunc from 0x001dff to 0x013bd5 (0x11dd7 bytes) 000465 1770447548.739671 mapread 0x00f291 thru 0x00f3a9 (0x119 bytes) 000467 1770447548.752191 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13bd6): Mirror 8: 0x6194fc61 (stale) Mirror 12: 0x6194fc61 (stale) Mirror 13: 0x6194fc61 (stale) Mirror 15: 0x6194fc61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13bd6): Mirror 8: 0x6194fc61 (stale) Mirror 12: 0x6194fc61 (stale) Mirror 13: 0x6194fc61 (stale) Mirror 15: 0x6194fc61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13bd6: skipped 000468 1770447548.991994 mapwrite 0x006714 thru 0x00d8a7 (0x7194 bytes) 000469 1770447549.043237 read 0x00602d thru 0x00f121 (0x90f5 bytes) 000470 1770447549.044162 mapwrite 0x038200 thru 0x039a02 (0x1803 bytes) 000471 1770447549.095678 write 0x001da2 thru 0x00cfdd (0xb23c bytes) 000472 1770447549.096881 read 0x0356ad thru 0x039a02 (0x4356 bytes) 000473 1770447549.097435 trunc from 0x039a03 to 0x01fd35 (0x19ccd bytes) 000475 1770447549.113371 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1fd36): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fd36): Mirror 8: 0x131a0e6c (stale) Mirror 12: 0x131a0e6c (stale) Mirror 13: 0x131a0e6c (stale) Mirror 15: 0x131a0e6c (stale) CRC-32 checksum value for chunk [0, 0x1fd36): Mirror 1: 0xe58203d6 Mirror 16: 0xe58203d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fd36: skipped 000476 1770447549.868597 mapwrite 0x01780c thru 0x024ed6 (0xd6cb bytes) 000477 1770447550.000920 trunc from 0x024ed7 to 0x016bab (0xe32b bytes) 000479 1770447550.012490 trunc from 0x016bac to 0x0381bc (0x21611 bytes) 000481 1770447550.021050 mapread 0x005c2e thru 0x00a831 (0x4c04 bytes) 000482 1770447550.027863 read 0x00fe39 thru 0x01eca0 (0xee68 bytes) 000486 1770447550.034780 mapread 0x017e48 thru 0x01e0df (0x6298 bytes) 000489 1770447550.036104 trunc from 0x0381bd to 0x031551 (0x6c6b bytes) 000490 1770447550.065748 MIRROR_RESYNC 6 mirrors 000491 1770447550.768975 write 0x034411 thru 0x03851e (0x410e bytes) 000492 1770447550.820913 mapwrite 0x02580a thru 0x030169 (0xa960 bytes) 000493 1770447550.866368 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3851f): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3851f): Mirror 8: 0xc87a6c4e (stale) Mirror 12: 0xc87a6c4e (stale) Mirror 13: 0xc87a6c4e (stale) Mirror 15: 0xc87a6c4e (stale) Mirror 16: 0xc87a6c4e (stale) CRC-32 checksum value for chunk [0, 0x3851f): Mirror 1: 0x3049c45 Mirror 17: 0x3049c45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3851f: skipped 000495 1770447551.424578 mapwrite 0x00f172 thru 0x0111a2 (0x2031 bytes) 000497 1770447551.598419 trunc from 0x03851f to 0x022473 (0x160ab bytes) 000498 1770447551.620266 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x22474): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22474): Mirror 8: 0x6a4e6b22 (stale) Mirror 12: 0x6a4e6b22 (stale) Mirror 13: 0x6a4e6b22 (stale) Mirror 15: 0x6a4e6b22 (stale) Mirror 16: 0x6a4e6b22 (stale) Mirror 17: 0x6a4e6b22 (stale) CRC-32 checksum value for chunk [0, 0x22474): Mirror 1: 0xca4e17ac Mirror 18: 0xca4e17ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22474: skipped 000499 1770447552.355523 MIRROR_RESYNC 8 mirrors 000501 1770447553.372008 write 0x00adda thru 0x017bd6 (0xcdfd bytes) 000503 1770447553.415586 read 0x010403 thru 0x0178c6 (0x74c4 bytes) 000505 1770447553.423582 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x22474): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22474): Mirror 8: 0xca4e17ac (stale) Mirror 12: 0xca4e17ac (stale) Mirror 13: 0xca4e17ac (stale) Mirror 15: 0xca4e17ac (stale) Mirror 16: 0xca4e17ac (stale) Mirror 17: 0xca4e17ac (stale) Mirror 18: 0xca4e17ac (stale) CRC-32 checksum value for chunk [0, 0x22474): Mirror 1: 0x76cd59bd Mirror 19: 0x76cd59bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22474: skipped 000506 1770447554.133423 mapwrite 0x034333 thru 0x03597d (0x164b bytes) 000507 1770447554.227001 read 0x01c89d thru 0x01f8d3 (0x3037 bytes) 000508 1770447554.230164 write 0x022209 thru 0x02de6f (0xbc67 bytes) 000510 1770447554.237922 trunc from 0x03597e to 0x0085a4 (0x2d3d9 bytes) 000511 1770447554.258709 write 0x01b70b thru 0x02173c (0x6032 bytes) 000512 1770447554.260183 trunc from 0x02173d to 0x0099f1 (0x17d4b bytes) 000517 1770447554.277395 trunc from 0x0099f2 to 0x0135aa (0x9bb9 bytes) 000518 1770447554.289471 trunc from 0x0135ab to 0x01efc8 (0xba1e bytes) 000523 1770447554.306737 mapread 0x001f4c thru 0x007fac (0x6061 bytes) 000524 1770447554.317826 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 8: 0x37e20c85 (stale) Mirror 12: 0x37e20c85 (stale) Mirror 13: 0x37e20c85 (stale) Mirror 15: 0x37e20c85 (stale) Mirror 16: 0x37e20c85 (stale) Mirror 17: 0x37e20c85 (stale) Mirror 18: 0x37e20c85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 8: 0x37e20c85 (stale) Mirror 12: 0x37e20c85 (stale) Mirror 13: 0x37e20c85 (stale) Mirror 15: 0x37e20c85 (stale) Mirror 16: 0x37e20c85 (stale) Mirror 17: 0x37e20c85 (stale) Mirror 18: 0x37e20c85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1efc9: skipped 000525 1770447554.621804 mapwrite 0x00c412 thru 0x00ee5a (0x2a49 bytes) 000526 1770447554.682528 mapread 0x006661 thru 0x0115b5 (0xaf55 bytes) 000527 1770447554.693534 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 8: 0x37e20c85 (stale) Mirror 12: 0x37e20c85 (stale) Mirror 13: 0x37e20c85 (stale) Mirror 15: 0x37e20c85 (stale) Mirror 16: 0x37e20c85 (stale) Mirror 17: 0x37e20c85 (stale) Mirror 18: 0x37e20c85 (stale) CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 1: 0xc77a9e Mirror 19: 0xc77a9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1efc9: skipped 000528 1770447555.298565 mapread 0x01b43d thru 0x01efc8 (0x3b8c bytes) 000530 1770447555.345412 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 8: 0x37e20c85 (stale) Mirror 12: 0x37e20c85 (stale) Mirror 13: 0x37e20c85 (stale) Mirror 15: 0x37e20c85 (stale) Mirror 17: 0x37e20c85 (stale) Mirror 18: 0x37e20c85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 8: 0x37e20c85 (stale) Mirror 12: 0x37e20c85 (stale) Mirror 13: 0x37e20c85 (stale) Mirror 15: 0x37e20c85 (stale) Mirror 17: 0x37e20c85 (stale) Mirror 18: 0x37e20c85 (stale) CRC-32 checksum value for chunk [0, 0x1efc9): Mirror 1: 0xc77a9e Mirror 19: 0xc77a9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1efc9: skipped 000531 1770447555.959483 mapread 0x0069c8 thru 0x0083af (0x19e8 bytes) 000535 1770447556.029013 read 0x016393 thru 0x018161 (0x1dcf bytes) 000536 1770447556.042590 trunc from 0x01efc9 to 0x008295 (0x16d33 bytes) 000537 1770447556.135857 read 0x0041af thru 0x008295 (0x40e7 bytes) 000538 1770447556.146365 read 0x0006d1 thru 0x0008a4 (0x1d4 bytes) 000539 1770447556.146965 mapwrite 0x00bb76 thru 0x0161c5 (0xa650 bytes) 000540 1770447556.215917 write 0x013b51 thru 0x013ee6 (0x396 bytes) 000541 1770447556.216523 read 0x01604e thru 0x0161c5 (0x178 bytes) 000544 1770447556.217443 mapwrite 0x0205c0 thru 0x024597 (0x3fd8 bytes) 000545 1770447556.267199 read 0x001448 thru 0x009f31 (0x8aea bytes) 000546 1770447556.268489 trunc from 0x024598 to 0x035b34 (0x1159d bytes) 000549 1770447556.280260 trunc from 0x035b35 to 0x0016a2 (0x34492 bytes) 000550 1770447556.294967 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x16a3): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16a3): Mirror 8: 0x781eecd0 (stale) Mirror 12: 0x781eecd0 (stale) Mirror 13: 0x781eecd0 (stale) Mirror 15: 0x781eecd0 (stale) Mirror 17: 0x781eecd0 (stale) Mirror 18: 0x781eecd0 (stale) Mirror 19: 0x781eecd0 (stale) CRC-32 checksum value for chunk [0, 0x16a3): Mirror 1: 0x781eecd0 Mirror 20: 0x781eecd0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16a3: skipped 000553 1770447556.744250 mapwrite 0x016b3c thru 0x02165a (0xab1f bytes) 000554 1770447556.905817 mapread 0x01c883 thru 0x02165a (0x4dd8 bytes) 000555 1770447556.906683 trunc from 0x02165b to 0x012714 (0xef46 bytes) 000557 1770447556.925864 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x12715): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12715): Mirror 8: 0x7866bb90 (stale) Mirror 12: 0x7866bb90 (stale) Mirror 13: 0x7866bb90 (stale) Mirror 15: 0x7866bb90 (stale) Mirror 17: 0x7866bb90 (stale) Mirror 18: 0x7866bb90 (stale) Mirror 19: 0x103d12c (stale) Mirror 20: 0x31f21ea1 (stale) CRC-32 checksum value for chunk [0, 0x12715): Mirror 1: 0x31f21ea1 Mirror 21: 0x31f21ea1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12715: skipped 000558 1770447557.551412 mapread 0x00e6ce thru 0x012714 (0x4047 bytes) 000561 1770447557.585921 mapwrite 0x010d78 thru 0x01b3c9 (0xa652 bytes) 000562 1770447557.744391 mapread 0x00ef33 thru 0x014883 (0x5951 bytes) 000564 1770447557.748025 mapwrite 0x006d6f thru 0x010b40 (0x9dd2 bytes) 000568 1770447557.847198 write 0x023ede thru 0x02e69c (0xa7bf bytes) 000570 1770447557.849927 read 0x013d49 thru 0x01ae0d (0x70c5 bytes) 000571 1770447557.850536 mapread 0x006a32 thru 0x00d23a (0x6809 bytes) 000572 1770447557.851382 trunc from 0x02e69d to 0x035588 (0x6eec bytes) 000574 1770447557.872779 read 0x011f25 thru 0x013af8 (0x1bd4 bytes) 000575 1770447557.873359 mapread 0x030fff thru 0x035588 (0x458a bytes) 000576 1770447557.890473 read 0x009694 thru 0x00eb6e (0x54db bytes) 000577 1770447557.891565 read 0x02b8fa thru 0x035588 (0x9c8f bytes) 000579 1770447557.897521 mapwrite 0x01174a thru 0x01bf90 (0xa847 bytes) 000581 1770447557.987338 read 0x008878 thru 0x015859 (0xcfe2 bytes) 000582 1770447557.988135 trunc from 0x035589 to 0x01b3b2 (0x1a1d6 bytes) 000584 1770447558.005495 mapread 0x009297 thru 0x00ba80 (0x27ea bytes) 000586 1770447558.006654 mapread 0x00f5cf thru 0x010302 (0xd34 bytes) 000589 1770447558.008297 write 0x021fed thru 0x02b9f5 (0x9a09 bytes) 000590 1770447558.010204 read 0x01c872 thru 0x02485c (0x7feb bytes) 000592 1770447558.024403 read 0x0017b8 thru 0x00444d (0x2c96 bytes) 000593 1770447558.025464 read 0x00cd83 thru 0x01030f (0x358d bytes) 000595 1770447558.026943 trunc from 0x02b9f6 to 0x020286 (0xb76f bytes) 000597 1770447558.061900 mapwrite 0x0272bb thru 0x03421a (0xcf60 bytes) 000599 1770447558.138589 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3421b): Mirror 8: 0xe1c0c15b (stale) Mirror 12: 0xe1c0c15b (stale) Mirror 13: 0xe1c0c15b (stale) Mirror 17: 0xe1c0c15b (stale) Mirror 18: 0xe1c0c15b (stale) Mirror 19: 0x6c7d69d4 (stale) Mirror 20: 0xe2918e6c (stale) Mirror 21: 0xe2918e6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3421b): Mirror 8: 0xe1c0c15b (stale) Mirror 12: 0xe1c0c15b (stale) Mirror 13: 0xe1c0c15b (stale) Mirror 17: 0xe1c0c15b (stale) Mirror 18: 0xe1c0c15b (stale) Mirror 19: 0x6c7d69d4 (stale) Mirror 20: 0xe2918e6c (stale) Mirror 21: 0xe2918e6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3421b: skipped 000603 1770447558.564114 mapwrite 0x015a1a thru 0x0223e2 (0xc9c9 bytes) 000604 1770447558.652422 write 0x0023ed thru 0x00da2f (0xb643 bytes) 000605 1770447558.674097 write 0x019242 thru 0x019def (0xbae bytes) 000607 1770447558.675388 mapread 0x029b6e thru 0x02c0a3 (0x2536 bytes) 000608 1770447558.683443 mapwrite 0x023c23 thru 0x03057d (0xc95b bytes) 000609 1770447558.728661 read 0x009580 thru 0x0167fe (0xd27f bytes) 000610 1770447558.729464 write 0x024ec7 thru 0x027f79 (0x30b3 bytes) 000611 1770447558.731083 read 0x00bf81 thru 0x00c9da (0xa5a bytes) 000612 1770447558.732020 trunc from 0x03421b to 0x00019d (0x3407d bytes) 000613 1770447558.752618 write 0x02c132 thru 0x02e6dd (0x25ac bytes) 000614 1770447558.755107 mapread 0x013a53 thru 0x01d08e (0x963c bytes) 000620 1770447558.775593 mapwrite 0x02ca5b thru 0x037e45 (0xb3eb bytes) 000621 1770447558.826154 read 0x01fbbf thru 0x02520c (0x564e bytes) 000622 1770447558.827016 mapwrite 0x032617 thru 0x03d13b (0xab25 bytes) 000623 1770447558.878073 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d13c): Mirror 12: 0x2d44ee63 (stale) Mirror 13: 0x2d44ee63 (stale) Mirror 17: 0x2d44ee63 (stale) Mirror 18: 0x2d44ee63 (stale) Mirror 19: 0xeeca17fb (stale) Mirror 20: 0xaa1c9f87 (stale) Mirror 21: 0xaa1c9f87 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d13c): Mirror 12: 0x2d44ee63 (stale) Mirror 13: 0x2d44ee63 (stale) Mirror 17: 0x2d44ee63 (stale) Mirror 18: 0x2d44ee63 (stale) Mirror 19: 0xeeca17fb (stale) Mirror 20: 0xaa1c9f87 (stale) Mirror 21: 0xaa1c9f87 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d13c: skipped 000624 1770447559.199757 mapwrite 0x023982 thru 0x032641 (0xecc0 bytes) 000625 1770447559.276873 trunc from 0x03d13c to 0x03c93a (0x801 bytes) 000626 1770447559.293655 write 0x01171f thru 0x01d634 (0xbf16 bytes) 000627 1770447559.322091 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c93b): Mirror 12: 0xff155693 (stale) Mirror 17: 0xff155693 (stale) Mirror 18: 0xff155693 (stale) Mirror 19: 0x9f570cb7 (stale) Mirror 20: 0xd550e045 (stale) Mirror 21: 0xd550e045 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c93b): Mirror 12: 0xff155693 (stale) Mirror 17: 0xff155693 (stale) Mirror 18: 0xff155693 (stale) Mirror 19: 0x9f570cb7 (stale) Mirror 20: 0xd550e045 (stale) Mirror 21: 0xd550e045 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c93b: skipped 000629 1770447559.709968 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c93b): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c93b): Mirror 12: 0xff155693 (stale) Mirror 17: 0xff155693 (stale) Mirror 18: 0xff155693 (stale) Mirror 19: 0x9f570cb7 (stale) Mirror 20: 0xd550e045 (stale) Mirror 21: 0xd550e045 (stale) CRC-32 checksum value for chunk [0, 0x3c93b): Mirror 1: 0x680ebaf Mirror 22: 0x680ebaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c93b: skipped 000630 1770447560.467671 write 0x039ad4 thru 0x03bdf0 (0x231d bytes) 000632 1770447560.573499 mapread 0x012967 thru 0x01999c (0x7036 bytes) 000635 1770447560.585348 MIRROR_RESYNC 8 mirrors 000637 1770447561.670777 mapwrite 0x029046 thru 0x032d1e (0x9cd9 bytes) 000640 1770447561.793914 mapwrite 0x01dfb1 thru 0x02a999 (0xc9e9 bytes) 000641 1770447561.853838 mapread 0x0298e6 thru 0x030770 (0x6e8b bytes) 000642 1770447561.854678 read 0x00b94c thru 0x00eff3 (0x36a8 bytes) 000644 1770447561.863943 mapwrite 0x0362e0 thru 0x03ffff (0x9d20 bytes) 000645 1770447561.951913 mapread 0x0326fd thru 0x03cfd0 (0xa8d4 bytes) 000646 1770447561.953009 trunc from 0x040000 to 0x01f7a9 (0x20856 bytes) 000647 1770447561.970135 mapread 0x0182f3 thru 0x01f7a9 (0x74b7 bytes) 000649 1770447561.971275 trunc from 0x01f7aa to 0x023562 (0x3db9 bytes) 000650 1770447561.986326 read 0x022393 thru 0x023562 (0x11d0 bytes) 000652 1770447561.993493 mapread 0x00a18e thru 0x00c31f (0x2192 bytes) 000655 1770447561.996092 mapread 0x00798b thru 0x00f093 (0x7709 bytes) 000659 1770447561.999363 write 0x013773 thru 0x01a89a (0x7128 bytes) 000662 1770447562.001613 mapwrite 0x033403 thru 0x03ffff (0xcbfd bytes) 000663 1770447562.070551 mapwrite 0x009c64 thru 0x019178 (0xf515 bytes) 000664 1770447562.113920 read 0x006890 thru 0x00c5b8 (0x5d29 bytes) 000665 1770447562.115374 write 0x017ca3 thru 0x01af52 (0x32b0 bytes) 000666 1770447562.116421 write 0x03a609 thru 0x03ffff (0x59f7 bytes) 000668 1770447562.123476 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5eeb8d7c (stale) Mirror 17: 0x5eeb8d7c (stale) Mirror 18: 0x5eeb8d7c (stale) Mirror 19: 0x5eeb8d7c (stale) Mirror 20: 0x5eeb8d7c (stale) Mirror 21: 0x5eeb8d7c (stale) Mirror 22: 0x5eeb8d7c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf406f1d4 Mirror 23: 0xf406f1d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000675 1770447562.778413 write 0x021fb5 thru 0x0242ae (0x22fa bytes) 000677 1770447562.850134 write 0x0063be thru 0x01540c (0xf04f bytes) 000680 1770447562.861790 read 0x01ca1b thru 0x027828 (0xae0e bytes) 000682 1770447562.870314 write 0x030dbb thru 0x03e0f5 (0xd33b bytes) 000683 1770447562.896615 read 0x00a616 thru 0x01006d (0x5a58 bytes) 000686 1770447562.898952 read 0x0367fa thru 0x03ffff (0x9806 bytes) 000687 1770447562.904900 mapread 0x036363 thru 0x03da65 (0x7703 bytes) 000688 1770447562.905799 mapwrite 0x03f6dd thru 0x03ffff (0x923 bytes) 000693 1770447562.967073 mapwrite 0x00e4ab thru 0x019de0 (0xb936 bytes) 000696 1770447563.013030 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5eeb8d7c (stale) Mirror 17: 0x5eeb8d7c (stale) Mirror 19: 0x5eeb8d7c (stale) Mirror 20: 0x5eeb8d7c (stale) Mirror 21: 0x5eeb8d7c (stale) Mirror 22: 0x5eeb8d7c (stale) Mirror 23: 0xf406f1d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5eeb8d7c (stale) Mirror 17: 0x5eeb8d7c (stale) Mirror 19: 0x5eeb8d7c (stale) Mirror 20: 0x5eeb8d7c (stale) Mirror 21: 0x5eeb8d7c (stale) Mirror 22: 0x5eeb8d7c (stale) Mirror 23: 0xf406f1d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000697 1770447563.347374 write 0x038a7a thru 0x03ffff (0x7586 bytes) 000698 1770447563.361344 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5eeb8d7c (stale) Mirror 19: 0x5eeb8d7c (stale) Mirror 20: 0x5eeb8d7c (stale) Mirror 21: 0x5eeb8d7c (stale) Mirror 22: 0x5eeb8d7c (stale) Mirror 23: 0xf406f1d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5eeb8d7c (stale) Mirror 19: 0x5eeb8d7c (stale) Mirror 20: 0x5eeb8d7c (stale) Mirror 21: 0x5eeb8d7c (stale) Mirror 22: 0x5eeb8d7c (stale) Mirror 23: 0xf406f1d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000706 1770447563.676340 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x5eeb8d7c (stale) Mirror 20: 0x5eeb8d7c (stale) Mirror 21: 0x5eeb8d7c (stale) Mirror 22: 0x5eeb8d7c (stale) Mirror 23: 0xf406f1d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x5eeb8d7c (stale) Mirror 20: 0x5eeb8d7c (stale) Mirror 21: 0x5eeb8d7c (stale) Mirror 22: 0x5eeb8d7c (stale) Mirror 23: 0xf406f1d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000707 1770447564.011369 trunc from 0x040000 to 0x039d7c (0x6283 bytes) 000709 1770447564.034777 mapwrite 0x003323 thru 0x00dd02 (0xa9e0 bytes) 000711 1770447564.069085 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39d7d): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d7d): Mirror 19: 0x54ccdc1c (stale) Mirror 20: 0x54ccdc1c (stale) Mirror 21: 0x54ccdc1c (stale) Mirror 22: 0x54ccdc1c (stale) Mirror 23: 0xe1e4d18 (stale) CRC-32 checksum value for chunk [0, 0x39d7d): Mirror 1: 0x5818ac1a Mirror 24: 0x5818ac1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d7d: skipped 000712 1770447564.678556 mapwrite 0x0170bb thru 0x01eb20 (0x7a66 bytes) 000713 1770447564.823274 mapread 0x007001 thru 0x012f82 (0xbf82 bytes) 000717 1770447564.839038 trunc from 0x039d7d to 0x01bb19 (0x1e263 bytes) 000718 1770447564.910008 trunc from 0x01bb1a to 0x009dac (0x11d6d bytes) 000722 1770447564.929880 mapwrite 0x02085a thru 0x0233ca (0x2b71 bytes) 000723 1770447565.022438 read 0x013d23 thru 0x01eda9 (0xb087 bytes) 000724 1770447565.031339 mapread 0x00519d thru 0x005d1f (0xb83 bytes) 000726 1770447565.033229 write 0x02aeea thru 0x030b81 (0x5c98 bytes) 000728 1770447565.037261 mapwrite 0x00da6f thru 0x01900c (0xb59e bytes) 000732 1770447565.102507 write 0x02972e thru 0x0371ff (0xdad2 bytes) 000733 1770447565.114475 read 0x0262fa thru 0x029c01 (0x3908 bytes) 000734 1770447565.115829 trunc from 0x037200 to 0x0036f0 (0x33b0f bytes) 000735 1770447565.141504 mapread 0x000304 thru 0x0035dc (0x32d9 bytes) 000736 1770447565.147946 write 0x000ae8 thru 0x0045de (0x3af7 bytes) 000738 1770447565.150426 read 0x000d3a thru 0x0045de (0x38a5 bytes) 000740 1770447565.156437 MIRROR_RESYNC 7 mirrors 000741 1770447566.089427 mapwrite 0x0015ba thru 0x00a96b (0x93b2 bytes) 000744 1770447566.236975 read 0x00195f thru 0x00a96b (0x900d bytes) 000745 1770447566.241192 MIRROR_RESYNC 7 mirrors 000746 1770447567.032474 read 0x002d88 thru 0x00a96b (0x7be4 bytes) 000751 1770447567.041253 mapwrite 0x00d104 thru 0x01b6a5 (0xe5a2 bytes) 000752 1770447567.184538 write 0x02f271 thru 0x03d7c9 (0xe559 bytes) 000753 1770447567.190452 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d7ca): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7ca): Mirror 19: 0x45643ced (stale) Mirror 20: 0x45643ced (stale) Mirror 21: 0x45643ced (stale) Mirror 22: 0x45643ced (stale) Mirror 23: 0x45643ced (stale) Mirror 24: 0x45643ced (stale) CRC-32 checksum value for chunk [0, 0x3d7ca): Mirror 1: 0x94e57cef Mirror 25: 0x94e57cef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7ca: skipped 000754 1770447567.791310 trunc from 0x03d7ca to 0x017fbe (0x2580b bytes) 000755 1770447567.903054 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17fbf): Mirror 19: 0xa450d4b8 (stale) Mirror 21: 0xa450d4b8 (stale) Mirror 22: 0xa450d4b8 (stale) Mirror 23: 0xa450d4b8 (stale) Mirror 24: 0xa450d4b8 (stale) Mirror 25: 0xaac84e8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17fbf): Mirror 19: 0xa450d4b8 (stale) Mirror 21: 0xa450d4b8 (stale) Mirror 22: 0xa450d4b8 (stale) Mirror 23: 0xa450d4b8 (stale) Mirror 24: 0xa450d4b8 (stale) Mirror 25: 0xaac84e8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17fbf: skipped 000756 1770447568.150988 read 0x015426 thru 0x017fbe (0x2b99 bytes) 000758 1770447568.155275 read 0x006376 thru 0x00e62b (0x82b6 bytes) 000760 1770447568.169743 MIRROR_RESYNC 7 mirrors 000761 1770447569.020492 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) 000766 1770447569.275258 mapread 0x017e63 thru 0x017fbe (0x15c bytes) 000767 1770447569.282280 mapread 0x006066 thru 0x00c7c0 (0x675b bytes) 000769 1770447569.294208 trunc from 0x017fbf to 0x021015 (0x9057 bytes) 000770 1770447569.364291 trunc from 0x021016 to 0x01a227 (0x6dee bytes) 000772 1770447569.381505 mapwrite 0x03023a thru 0x03bba6 (0xb96d bytes) 000774 1770447569.460521 write 0x0102e3 thru 0x01f433 (0xf151 bytes) 000778 1770447569.487428 mapread 0x01a92b thru 0x01f9a1 (0x5077 bytes) 000781 1770447569.491149 write 0x038e93 thru 0x03ffff (0x716d bytes) 000785 1770447569.494987 write 0x0010d6 thru 0x00bd45 (0xac70 bytes) 000787 1770447569.508131 MIRROR_RESYNC 6 mirrors 000788 1770447570.371148 trunc from 0x040000 to 0x018531 (0x27ace bytes) 000792 1770447570.439403 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x18532): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18532): Mirror 19: 0xad5259a9 (stale) Mirror 21: 0xad5259a9 (stale) Mirror 22: 0xad5259a9 (stale) Mirror 23: 0xad5259a9 (stale) Mirror 25: 0xad5259a9 (stale) CRC-32 checksum value for chunk [0, 0x18532): Mirror 1: 0xad5259a9 Mirror 26: 0xad5259a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18532: skipped 000793 1770447571.229497 mapwrite 0x017068 thru 0x024248 (0xd1e1 bytes) 000794 1770447571.396980 write 0x031f26 thru 0x03b499 (0x9574 bytes) 000795 1770447571.398452 mapread 0x0052ff thru 0x00c84b (0x754d bytes) 000796 1770447571.418260 mapread 0x02b6ca thru 0x02e77b (0x30b2 bytes) 000797 1770447571.434892 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b49a): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b49a): Mirror 19: 0x9f45e63d (stale) Mirror 21: 0x9f45e63d (stale) Mirror 22: 0x9f45e63d (stale) Mirror 23: 0x9f45e63d (stale) Mirror 25: 0x9f45e63d (stale) Mirror 26: 0x1588eb29 (stale) CRC-32 checksum value for chunk [0, 0x3b49a): Mirror 1: 0x39d2b37a Mirror 27: 0x39d2b37a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b49a: skipped 000799 1770447572.110829 write 0x03787d thru 0x03dccf (0x6453 bytes) 000804 1770447572.164913 trunc from 0x03dcd0 to 0x01243e (0x2b891 bytes) 000805 1770447572.204716 mapwrite 0x019f70 thru 0x029dea (0xfe7b bytes) 000806 1770447572.271350 mapwrite 0x001cec thru 0x00a800 (0x8b15 bytes) 000807 1770447572.314227 write 0x01d1ec thru 0x021762 (0x4577 bytes) 000814 1770447572.325669 mapread 0x024157 thru 0x029dea (0x5c94 bytes) 000815 1770447572.326564 mapread 0x002521 thru 0x0046c1 (0x21a1 bytes) 000816 1770447572.332132 trunc from 0x029deb to 0x01e336 (0xbab4 bytes) 000818 1770447572.353121 trunc from 0x01e337 to 0x003949 (0x1a9ed bytes) 000820 1770447572.366930 write 0x0117f6 thru 0x0154b0 (0x3cbb bytes) 000821 1770447572.368107 mapwrite 0x00e826 thru 0x01cd14 (0xe4ef bytes) 000822 1770447572.461077 read 0x002d8b thru 0x012926 (0xfb9c bytes) 000823 1770447572.465148 mapread 0x015174 thru 0x01a26c (0x50f9 bytes) 000826 1770447572.467422 read 0x011ccf thru 0x017860 (0x5b92 bytes) 000828 1770447572.469309 trunc from 0x01cd15 to 0x03b4e3 (0x1e7cf bytes) 000829 1770447572.487119 mapwrite 0x00d3e5 thru 0x01261d (0x5239 bytes) 000830 1770447572.529187 trunc from 0x03b4e4 to 0x026064 (0x1547f bytes) 000831 1770447572.541642 trunc from 0x026065 to 0x03478d (0xe729 bytes) 000832 1770447572.554230 write 0x0022ca thru 0x00eb5c (0xc893 bytes) 000833 1770447572.555601 mapread 0x02b131 thru 0x03478d (0x965d bytes) 000834 1770447572.563971 read 0x0015a6 thru 0x00efc1 (0xda1c bytes) 000836 1770447572.564752 write 0x026b32 thru 0x032f82 (0xc451 bytes) 000838 1770447572.565949 mapread 0x00002a thru 0x00e5f2 (0xe5c9 bytes) 000840 1770447572.566850 read 0x0278bc thru 0x02e665 (0x6daa bytes) 000841 1770447572.567428 read 0x01293c thru 0x01dd9d (0xb462 bytes) 000847 1770447572.570174 write 0x03870c thru 0x03dd7b (0x5670 bytes) 000849 1770447572.574419 MIRROR_RESYNC 8 mirrors 000850 1770447573.358472 trunc from 0x03dd7c to 0x003c46 (0x3a135 bytes) 000851 1770447573.411668 read 0x0037f4 thru 0x003c46 (0x453 bytes) 000854 1770447573.415363 mapread 0x003437 thru 0x003c46 (0x810 bytes) 000857 1770447573.419823 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c47): Mirror 19: 0x60cf9587 (stale) Mirror 21: 0x60cf9587 (stale) Mirror 22: 0x60cf9587 (stale) Mirror 23: 0x60cf9587 (stale) Mirror 25: 0x60cf9587 (stale) Mirror 27: 0x60cf9587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c47): Mirror 19: 0x60cf9587 (stale) Mirror 21: 0x60cf9587 (stale) Mirror 22: 0x60cf9587 (stale) Mirror 23: 0x60cf9587 (stale) Mirror 25: 0x60cf9587 (stale) Mirror 27: 0x60cf9587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c47: skipped 000858 1770447573.640698 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c47): Mirror 19: 0x60cf9587 (stale) Mirror 21: 0x60cf9587 (stale) Mirror 22: 0x60cf9587 (stale) Mirror 23: 0x60cf9587 (stale) Mirror 25: 0x60cf9587 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c47): Mirror 19: 0x60cf9587 (stale) Mirror 21: 0x60cf9587 (stale) Mirror 22: 0x60cf9587 (stale) Mirror 23: 0x60cf9587 (stale) Mirror 25: 0x60cf9587 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c47: skipped 000859 1770447573.864008 trunc from 0x003c47 to 0x036b84 (0x32f3e bytes) 000861 1770447573.886200 mapread 0x032a64 thru 0x036b84 (0x4121 bytes) 000862 1770447573.908247 read 0x03666d thru 0x036b84 (0x518 bytes) 000863 1770447573.908646 mapwrite 0x00a3a1 thru 0x00ca62 (0x26c2 bytes) 000865 1770447573.956680 write 0x013c8b thru 0x017879 (0x3bef bytes) 000866 1770447573.967554 write 0x01a39d thru 0x022851 (0x84b5 bytes) 000867 1770447573.971301 write 0x036d26 thru 0x03e965 (0x7c40 bytes) 000868 1770447573.972655 write 0x00f235 thru 0x017005 (0x7dd1 bytes) 000870 1770447573.974277 read 0x02ddce thru 0x03961e (0xb851 bytes) 000871 1770447573.977828 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e966): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e966): Mirror 19: 0xa7e19b29 (stale) Mirror 21: 0xa7e19b29 (stale) Mirror 22: 0xa7e19b29 (stale) Mirror 23: 0xa7e19b29 (stale) Mirror 25: 0xa7e19b29 (stale) CRC-32 checksum value for chunk [0, 0x3e966): Mirror 1: 0x92050f29 Mirror 26: 0x92050f29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e966: skipped 000872 1770447574.650570 trunc from 0x03e966 to 0x030e93 (0xdad2 bytes) 000873 1770447574.720416 trunc from 0x030e94 to 0x027eb8 (0x8fdb bytes) 000874 1770447574.734466 read 0x021ed7 thru 0x02218e (0x2b8 bytes) 000875 1770447574.739586 trunc from 0x027eb9 to 0x030d89 (0x8ed1 bytes) 000878 1770447574.756846 read 0x0071cc thru 0x00f0bc (0x7ef1 bytes) 000881 1770447574.774012 write 0x02aa05 thru 0x033910 (0x8f0c bytes) 000883 1770447574.775803 mapread 0x001e24 thru 0x007718 (0x58f5 bytes) 000884 1770447574.777730 write 0x024f93 thru 0x026dbe (0x1e2c bytes) 000886 1770447574.779065 trunc from 0x033911 to 0x0118d6 (0x2203a bytes) 000887 1770447574.795260 MIRROR_RESYNC 7 mirrors 000888 1770447575.573019 trunc from 0x0118d7 to 0x036577 (0x24ca1 bytes) 000891 1770447575.635222 trunc from 0x036578 to 0x01b609 (0x1af6e bytes) 000892 1770447575.663885 MIRROR_RESYNC 7 mirrors 000893 1770447576.464179 trunc from 0x01b60a to 0x01720a (0x43ff bytes) 000894 1770447576.520850 mapwrite 0x01ba9b thru 0x022f1b (0x7481 bytes) 000895 1770447576.595184 MIRROR_RESYNC 7 mirrors 000896 1770447577.355318 mapread 0x0209c1 thru 0x022f1b (0x255b bytes) 000897 1770447577.381373 MIRROR_EXTEND to 8 mirrors 000898 1770447577.819262 write 0x02389a thru 0x02459b (0xd02 bytes) 000900 1770447577.875361 mapread 0x017f96 thru 0x02459b (0xc606 bytes) 000902 1770447577.905832 read 0x01714f thru 0x02459b (0xd44d bytes) 000903 1770447577.906646 read 0x014f78 thru 0x02459b (0xf624 bytes) 000904 1770447577.907460 trunc from 0x02459c to 0x033dc8 (0xf82d bytes) 000905 1770447577.926677 mapread 0x0283d6 thru 0x029cde (0x1909 bytes) 000911 1770447577.933611 write 0x0100f4 thru 0x01af88 (0xae95 bytes) 000913 1770447577.936882 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33dc9): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dc9): Mirror 19: 0x7f106e3f (stale) Mirror 21: 0x7f106e3f (stale) Mirror 22: 0x7f106e3f (stale) Mirror 23: 0x7f106e3f (stale) Mirror 25: 0x7f106e3f (stale) Mirror 26: 0x7f106e3f (stale) Mirror 27: 0x7f106e3f (stale) CRC-32 checksum value for chunk [0, 0x33dc9): Mirror 1: 0x1955b56d Mirror 28: 0x1955b56d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dc9: skipped 000914 1770447578.538227 read 0x01688d thru 0x0257a4 (0xef18 bytes) 000917 1770447578.567705 mapread 0x01795b thru 0x024cbf (0xd365 bytes) 000918 1770447578.581859 read 0x0096f8 thru 0x00f99a (0x62a3 bytes) 000919 1770447578.592009 read 0x00f21b thru 0x01ea5f (0xf845 bytes) 000922 1770447578.604046 write 0x026e99 thru 0x028658 (0x17c0 bytes) 000924 1770447578.676605 read 0x012a6c thru 0x0187cf (0x5d64 bytes) 000926 1770447578.677891 mapread 0x003509 thru 0x00fed8 (0xc9d0 bytes) 000927 1770447578.678656 mapwrite 0x0393de thru 0x03ffff (0x6c22 bytes) 000928 1770447578.746090 mapread 0x00c629 thru 0x013e72 (0x784a bytes) 000929 1770447578.747529 trunc from 0x040000 to 0x0260d5 (0x19f2a bytes) 000930 1770447578.763143 write 0x011489 thru 0x01334b (0x1ec3 bytes) 000931 1770447578.764128 mapwrite 0x035ede thru 0x035ee0 (0x3 bytes) 000938 1770447578.803309 trunc from 0x035ee1 to 0x012794 (0x2374c bytes) 000939 1770447578.816214 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x12795): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12795): Mirror 19: 0xa0cb13b (stale) Mirror 21: 0xa0cb13b (stale) Mirror 22: 0xa0cb13b (stale) Mirror 23: 0xa0cb13b (stale) Mirror 25: 0xa0cb13b (stale) Mirror 26: 0xa0cb13b (stale) Mirror 27: 0xa0cb13b (stale) Mirror 28: 0xd6b7c43c (stale) CRC-32 checksum value for chunk [0, 0x12795): Mirror 1: 0x90fe8256 Mirror 29: 0x90fe8256 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12795: skipped 000942 1770447579.365261 read 0x011e7d thru 0x012794 (0x918 bytes) 000943 1770447579.372325 trunc from 0x012795 to 0x03a5c6 (0x27e32 bytes) 000945 1770447579.444244 mapwrite 0x027cfc thru 0x02bee6 (0x41eb bytes) 000947 1770447579.480139 read 0x036f42 thru 0x039bed (0x2cac bytes) 000948 1770447579.486398 trunc from 0x03a5c7 to 0x00b089 (0x2f53d bytes) 000949 1770447579.505487 mapwrite 0x03938c thru 0x03ffff (0x6c74 bytes) 000951 1770447579.591245 trunc from 0x040000 to 0x005cbe (0x3a341 bytes) 000952 1770447579.604253 trunc from 0x005cbf to 0x0192eb (0x1362d bytes) 000953 1770447579.617018 mapwrite 0x00991d thru 0x00a9ed (0x10d1 bytes) 000954 1770447579.665752 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x192ec): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x192ec): Mirror 19: 0xeb680ea0 (stale) Mirror 21: 0xeb680ea0 (stale) Mirror 22: 0xeb680ea0 (stale) Mirror 23: 0xeb680ea0 (stale) Mirror 25: 0xeb680ea0 (stale) Mirror 26: 0xeb680ea0 (stale) Mirror 27: 0xeb680ea0 (stale) Mirror 28: 0xdb562184 (stale) Mirror 29: 0xb02dcbfb (stale) CRC-32 checksum value for chunk [0, 0x192ec): Mirror 1: 0x8e685027 Mirror 30: 0x8e685027 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x192ec: skipped 000957 1770447580.264558 mapwrite 0x03a7b9 thru 0x03ffff (0x5847 bytes) 000962 1770447580.419607 read 0x00755a thru 0x016f18 (0xf9bf bytes) 000965 1770447580.427424 write 0x033456 thru 0x039416 (0x5fc1 bytes) 000966 1770447580.436813 read 0x02011c thru 0x027206 (0x70eb bytes) 000967 1770447580.447557 mapread 0x02a3bf thru 0x0318af (0x74f1 bytes) 000968 1770447580.452591 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x25d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x31402b66 (stale) Mirror 21: 0x31402b66 (stale) Mirror 22: 0x31402b66 (stale) Mirror 23: 0x31402b66 (stale) Mirror 25: 0x31402b66 (stale) Mirror 27: 0x31402b66 (stale) Mirror 28: 0x21ae7b6c (stale) Mirror 29: 0x143472c5 (stale) Mirror 30: 0xd2f4de7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x31402b66 (stale) Mirror 21: 0x31402b66 (stale) Mirror 22: 0x31402b66 (stale) Mirror 23: 0x31402b66 (stale) Mirror 25: 0x31402b66 (stale) Mirror 27: 0x31402b66 (stale) Mirror 28: 0x21ae7b6c (stale) Mirror 29: 0x143472c5 (stale) Mirror 30: 0xd2f4de7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000969 1770447580.822643 trunc from 0x040000 to 0x033ddc (0xc223 bytes) 000970 1770447580.842033 MIRROR_RESYNC 10 mirrors 000971 1770447581.847359 write 0x01bfc9 thru 0x025a8a (0x9ac2 bytes) 000972 1770447581.880260 trunc from 0x033ddd to 0x01c9d0 (0x1740c bytes) 000973 1770447581.888856 write 0x01e23d thru 0x01ef27 (0xceb bytes) 000974 1770447581.889553 mapwrite 0x02e97e thru 0x03d8c0 (0xef43 bytes) 000976 1770447581.950304 trunc from 0x03d8c1 to 0x00c54f (0x31371 bytes) 000978 1770447581.966910 write 0x03edae thru 0x03edf5 (0x48 bytes) 000980 1770447581.968340 mapread 0x01f4fa thru 0x02df12 (0xea19 bytes) 000982 1770447581.987712 MIRROR_RESYNC 10 mirrors 000985 1770447582.847109 mapwrite 0x032f28 thru 0x03661d (0x36f6 bytes) 000987 1770447582.959344 read 0x002f63 thru 0x003819 (0x8b7 bytes) 000988 1770447582.964656 mapwrite 0x00851c thru 0x015e97 (0xd97c bytes) 000989 1770447583.020163 mapread 0x01eb91 thru 0x027c24 (0x9094 bytes) 000990 1770447583.032428 mapread 0x001c87 thru 0x0112c5 (0xf63f bytes) 000995 1770447583.036506 trunc from 0x03edf6 to 0x0153bf (0x29a36 bytes) 001000 1770447583.060531 trunc from 0x0153c0 to 0x00d050 (0x836f bytes) All operations completed A-OK! PASS 70a (74s) == sanity-flr test 71: check mirror extend parent fid ==== 01:59:45 (1770447585) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 01:59:47 (1770447587) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137072 s, 15.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x356:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127931 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x356:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x356:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x357:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 01:59:52 (1770447592) Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre2 Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5342496 .. done Extending file size to 7808224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5886400 .. Extending file size to 8044640 .. failed Extending file size to 6198848 .. Extending file size to 6694528 .. Extending file size to 8160928 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8933184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9017632 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9261504 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 55889 55890 55891 55894 55895 done 192.168.204.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg404-client.virtnet /mnt/lustre2 (opts:) 192.168.204.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg404-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (72s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:01:04 (1770447664) Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre2 Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6712384 .. Extending file size to 5015072 .. done Extending file size to 8024736 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7157664 .. Extending file size to 8043936 .. Extending file size to 7170464 .. Extending file size to 8420448 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8307744 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8742784 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9006816 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8638720 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8670496 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9170912 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9262016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9229376 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58237 58238 58239 58240 58243 done 192.168.204.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg404-client.virtnet /mnt/lustre2 (opts:) 192.168.204.104@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg404-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:02:13 (1770447733) Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4352 2204032 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4352 2204032 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0623908 s, 33.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193876 s, 10.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcc71fdb9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg404-client.virtnet /mnt/lustre2 (opts:) PASS 200c (9s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:02:23 (1770447743) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0477004 s, 44.0 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 02:02:27 (1770447747) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2a9:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x368:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x368:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x369:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x36c:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:02:31 (1770447751) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0835076 s, 25.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:02:34 (1770447754) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137299 s, 15.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x370:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:02:38 (1770447758) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165036 s, 12.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x371:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x371:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x372:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:03:04 (1770447784) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg404-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638781 s, 16.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x374:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x373:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x374:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x373:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36f:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:03:51 (1770447831) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg404-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x371:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0842689 s, 24.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x371:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x375:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x371:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x375:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:04:36 (1770447876) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg404-server mds-ost sync done. Creating new pool test_204f oleg404-server: Pool lustre.test_204f created Adding targets to pool oleg404-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg404-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0555542 s, 37.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x376:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x376:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg404-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg404-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg404-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (61s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:05:37 (1770447937) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:05:41 (1770447941) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:05:45 (1770447945) oleg404-server: Pool lustre.test_206 created Adding targets to pool oleg404-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg404-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg404-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg404-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg404-server: Pool lustre.test_206 destroyed PASS 206 (17s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:06:01 (1770447961) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328529 s, 31.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176176 s, 59.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9c3edf26 Mirror 3: 0x9c3edf26 PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:06:06 (1770447966) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:06:08 (1770447968) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:06:10 (1770447970) Starting client: oleg404-client.virtnet: -o user_xattr,flock 192.168.204.104@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0898915 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x7b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2340000 Read this (4 bytes): BBBB 192.168.204.104@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg404-client.virtnet /mnt/lustre2 (opts:) PASS 209a (21s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:06:31 (1770447991) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0147941 s, 554 kB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:06:34 (1770447994) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028726 s, 36.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2ce lmm_fid: [0x200000404:0x2ce:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2ce lmm_fid: [0x200000404:0x2ce:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x381:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834638 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-02-07 02:06:35.000000000 -0500 Modify: 2026-02-07 02:06:35.000000000 -0500 Change: 2026-02-07 02:06:36.000000000 -0500 Birth: 2026-02-07 02:06:34.000000000 -0500 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:06:38 (1770447998) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:06:40 (1770448000) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.369172 s, 111 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0517049 s, 79.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Feb 7 02:06 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2d0:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Feb 7 02:06 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:06:45 (1770448005) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:06:49 (1770448009) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1141 sec =========== 02:06:53 (1770448013) === sanity-flr: start cleanup 02:06:54 (1770448014) === === sanity-flr: finish cleanup 02:06:57 (1770448017) ===