-----============= acceptance-small: sanity-flr ============----- Thu Sep 17 18:15:31 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg155-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:15:50 (1789683350) === oleg155-client.virtnet: executing check_config_client /mnt/lustre oleg155-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg155-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff954346d3b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff954346d3b800.idle_timeout=debug disable quota as required oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:16:25 (1789683385) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7621] == sanity-flr test 0a: lfs mirror create with -N option == 18:16:28 (1789683388) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:16:42 (1789683402) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:16:45 (1789683405) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:16:48 (1789683408) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:17:14 (1789683434) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:17:18 (1789683438) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 18:17:21 (1789683441) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff954346d3b800.stats=clear osc.lustre-OST0001-osc-ffff954346d3b800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:17:29 (1789683449) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:17:41 (1789683461) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:17:50 (1789683470) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:17:54 (1789683474) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:18:02 (1789683482) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 18:18:11 (1789683491) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:18:18 (1789683498) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:18:22 (1789683502) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:18:31 (1789683511) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:18:41 (1789683521) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:18:48 (1789683528) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.62482 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169495 s, 6.2 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:18:59 (1789683539) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.546044 s, 26.9 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:19:08 (1789683548) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:19:20 (1789683560) Stopping /mnt/lustre-ost1 (opts:) on oleg155-server oleg155-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg155-server: oleg155-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg155-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg155-server oleg155-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg155-server: oleg155-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg155-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:20:34 (1789683634) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.495028 s, 19.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg155-server oleg155-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg155-server: oleg155-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg155-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:21:20 (1789683680) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg155-server oleg155-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg155-server: oleg155-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg155-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg155-server oleg155-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg155-server: oleg155-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg155-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:22:51 (1789683771) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.63617 s, 9.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 17 18:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.010s user 0m0.000s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg155-server oleg155-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff954346d3b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.188s user 0m0.001s sys 0m0.172s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg155-server: oleg155-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg155-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg155-server oleg155-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m2.082s user 0m0.000s sys 0m0.202s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg155-server: oleg155-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg155-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg155-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff954346d3b800.ost_server_uuid in FULL state after 0 sec PASS 33b (107s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:24:38 (1789683878) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:24:42 (1789683882) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:24:45 (1789683885) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:24:49 (1789683889) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 18:24:57 (1789683897) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.565862 s, 22.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:25:10 (1789683910) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263277 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590368 s, 17.8 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:25:20 (1789683920) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:25:28 (1789683928) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105517 s, 970 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118458 s, 864 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000975235 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000983278 s, 1.0 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 18:25:38 (1789683938) create test files with size 15747997 .. e14a0b8feb7db3e722c9d471990fc159 - 519d763de6a89dd57aaba937575d5c84 - dfd73a782128b978670163822829e840 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg155-client.virtnet /mnt/lustre (opts:) Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 18:26:08 (1789683968) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:26:26 (1789683986) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:26:35 (1789683995) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20342 s, 5.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630013 s, 16.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 18:26:46 (1789684006) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.322516 s, 6.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.623473 s, 6.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269901 s, 7.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (23s) == sanity-flr test 42: lfs mirror verify ================= 18:27:09 (1789684029) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 18:27:13 (1789684033) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:27:17 (1789684037) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311468 s, 33.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 18:27:25 (1789684045) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:27:29 (1789684049) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:27:43 (1789684063) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.362655 s, 28.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg155-client.virtnet /mnt/lustre (opts:) Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:27:58 (1789684078) llite.lustre-ffff954344266800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.427935 s, 24.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg155-client.virtnet /mnt/lustre (opts:) Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff954346f2d800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:28:15 (1789684095) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:29:02 (1789684142) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:29:35 (1789684175) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 18:29:38 (1789684178) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.201308 s, 15.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.164057 s, 19.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7436db6e (nosync,stale) Mirror 2: 0x7436db6e (nosync,stale) ** verify mirror 2 doesn't change original checksum: cbfb8c6a181131c42f982bfe1358a37d - mirror 1 checksum: 3c5fd089b90e49525b4552bd7bf488b0 - mirror 2 checksum: cbfb8c6a181131c42f982bfe1358a37d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T18:29:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T18:29:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cbfb8c6a181131c42f982bfe1358a37d - mirror 1 checksum: 3c5fd089b90e49525b4552bd7bf488b0 - mirror 2 checksum: 3c5fd089b90e49525b4552bd7bf488b0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T18:29:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T18:29:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:29:51 (1789684191) Failing mds1 on oleg155-server Stopping /mnt/lustre-mds1 (opts:) on oleg155-server 18:29:56 (1789684196) shut down facet: mds1 facet_host: oleg155-server facet_failover_host: oleg155-server Failover mds1 to oleg155-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 18:30:19 (1789684219) targets are mounted 18:30:19 (1789684219) facet_failover done oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283423 s, 37.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:30:37 (1789684237) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:30:41 (1789684241) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:30:44 (1789684244) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0118309 s, 1.4 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:30:53 (1789684253) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:30:57 (1789684257) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0236827 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.093427 s, 43.8 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:31:07 (1789684267) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:31:30 (1789684290) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:32:00 (1789684320) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:32:24 (1789684344) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:32:51 (1789684371) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 18:32:59 (1789684379) mirror create pid 54310 mirror split pid 54311 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7851: 54311 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54310 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 18:34:06 (1789684446) Seed set to 4448 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1789684448.420503 write 0x007a61 thru 0x009322 (0x18c2 bytes) 000003 1789684448.435801 read 0x003568 thru 0x004716 (0x11af bytes) 000004 1789684448.443353 mapread 0x007d81 thru 0x009322 (0x15a2 bytes) 000005 1789684448.446433 read 0x006f18 thru 0x009322 (0x240b bytes) 000006 1789684448.457935 mapwrite 0x011686 thru 0x020dc1 (0xf73c bytes) 000007 1789684448.558510 mapwrite 0x0144ca thru 0x01f7f6 (0xb32d bytes) 000009 1789684448.598449 write 0x0398cd thru 0x03ffff (0x6733 bytes) 000011 1789684448.601592 mapwrite 0x00d91f thru 0x00ea8e (0x1170 bytes) 000012 1789684448.647995 punch from 0x01e1e7 to 0x01f33c (0x1156 bytes) 000014 1789684448.654304 mapwrite 0x024b4b thru 0x02b8bc (0x6d72 bytes) 000017 1789684448.715067 mapwrite 0x025ce6 thru 0x02b8e3 (0x5bfe bytes) 000019 1789684448.757983 write 0x011cbd thru 0x012667 (0x9ab bytes) truncating to largest ever: 0xf38e 000021 1789684448.759302 trunc from 0x040000 to 0x00f38d (0x30c72 bytes) 000022 1789684448.775696 write 0x021dff thru 0x02afe9 (0x91eb bytes) 000024 1789684448.777576 trunc from 0x02afea to 0x0061f5 (0x24df4 bytes) 000026 1789684448.797347 punch from 0x002ba6 to 0x0061f5 (0x3650 bytes) 000028 1789684448.803315 mapwrite 0x028c42 thru 0x035a0c (0xcdcb bytes) 000029 1789684448.919803 punch from 0x025174 to 0x034e62 (0xfcef bytes) 000030 1789684448.923012 trunc from 0x035a0d to 0x00afe6 (0x2aa26 bytes) 000032 1789684448.934099 trunc from 0x00afe7 to 0x009d9b (0x124b bytes) truncating to largest ever: 0x17b21 000034 1789684448.948095 trunc from 0x009d9c to 0x017b20 (0xdd85 bytes) 000036 1789684448.963038 write 0x03c90c thru 0x03ffff (0x36f4 bytes) 000038 1789684448.966705 punch from 0x02aecf to 0x037137 (0xc269 bytes) 000039 1789684448.978479 mapread 0x037b8a thru 0x03e88f (0x6d06 bytes) 000040 1789684449.000098 read 0x01d99a thru 0x0292ee (0xb955 bytes) 000042 1789684449.012833 punch from 0x00ff82 to 0x01f90d (0xf98c bytes) 000044 1789684449.018018 punch from 0x02988d to 0x0376d1 (0xde45 bytes) 000045 1789684449.026731 MIRROR_EXTEND to 2 mirrors 000046 1789684449.924656 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000047 1789684450.069924 write 0x017971 thru 0x01b0ca (0x375a bytes) 000049 1789684450.102572 mapread 0x034735 thru 0x03ffff (0xb8cb bytes) 000051 1789684450.127504 mapread 0x01f5ef thru 0x022f9b (0x39ad bytes) 000053 1789684450.141466 read 0x00efd1 thru 0x01dbf0 (0xec20 bytes) 000055 1789684450.151202 punch from 0x01bb89 to 0x01db74 (0x1fec bytes) 000056 1789684450.165805 MIRROR_EXTEND to 2 mirrors 000058 1789684451.277184 mapwrite 0x0270e9 thru 0x030d76 (0x9c8e bytes) 000060 1789684451.511720 punch from 0x0200dd to 0x021af2 (0x1a16 bytes) 000061 1789684451.525003 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f3fd760 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb5aeb1e6 Mirror 3: 0xb5aeb1e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000062 1789684452.623438 mapread 0x0059c5 thru 0x015796 (0xfdd2 bytes) 000067 1789684452.649040 read 0x01d456 thru 0x02381b (0x63c6 bytes) 000068 1789684452.658996 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f3fd760 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f3fd760 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000069 1789684453.006819 mapwrite 0x01cc8a thru 0x022f1e (0x6295 bytes) truncating to largest ever: 0x3be24 000072 1789684453.084686 trunc from 0x040000 to 0x03be23 (0x41dc bytes) 000073 1789684453.101699 write 0x03dbda thru 0x03ffff (0x2426 bytes) 000076 1789684453.117637 MIRROR_RESYNC 2 mirrors 000078 1789684453.872948 read 0x01276e thru 0x01cd18 (0xa5ab bytes) 000079 1789684453.885176 trunc from 0x040000 to 0x0333f6 (0xcc09 bytes) 000080 1789684453.943660 mapread 0x001e8e thru 0x00ae36 (0x8fa9 bytes) 000081 1789684453.961303 mapwrite 0x0340e4 thru 0x03d9fd (0x991a bytes) 000082 1789684454.033850 punch from 0x034e6c to 0x03d9fd (0x8b92 bytes) 000083 1789684454.039933 mapread 0x023ced thru 0x032e28 (0xf13c bytes) 000084 1789684454.055696 mapread 0x028441 thru 0x0350ee (0xccae bytes) 000085 1789684454.067567 MIRROR_RESYNC 2 mirrors 000086 1789684454.830843 trunc from 0x03d9fe to 0x007d48 (0x35cb5 bytes) 000087 1789684454.894639 mapread 0x001c86 thru 0x007d48 (0x60c3 bytes) 000088 1789684454.913033 write 0x02e780 thru 0x0338ba (0x513b bytes) 000091 1789684454.915511 mapwrite 0x005059 thru 0x00ea47 (0x99ef bytes) 000093 1789684455.012421 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000094 1789684455.166023 punch from 0x008773 to 0x017367 (0xebf5 bytes) 000096 1789684455.172555 trunc from 0x0338bb to 0x006495 (0x2d425 bytes) 000098 1789684455.197359 MIRROR_EXTEND to 2 mirrors 000100 1789684455.924597 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x3e9a0 000103 1789684456.740204 trunc from 0x006496 to 0x03e99f (0x3850a bytes) 000104 1789684456.826226 mapread 0x0321ad thru 0x03e99f (0xc7f3 bytes) 000105 1789684456.852985 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9a0): Mirror 3: 0xa9c98f1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9a0): Mirror 3: 0xa9c98f1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9a0: skipped 000106 1789684457.120783 read 0x0033a9 thru 0x0098b5 (0x650d bytes) 000107 1789684457.133304 punch from 0x03a8f8 to 0x03e99f (0x40a8 bytes) 000109 1789684457.138545 write 0x023056 thru 0x023e92 (0xe3d bytes) 000110 1789684457.148175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000111 1789684457.277663 mapread 0x024b8d thru 0x02fed2 (0xb346 bytes) 000114 1789684457.357546 mapwrite 0x00d27d thru 0x00e7e9 (0x156d bytes) 000115 1789684457.421802 read 0x02691d thru 0x02e021 (0x7705 bytes) 000116 1789684457.422404 write 0x0178f6 thru 0x01c32d (0x4a38 bytes) 000117 1789684457.438533 write 0x016767 thru 0x01a1d4 (0x3a6e bytes) 000118 1789684457.445769 read 0x01b019 thru 0x0222f5 (0x72dd bytes) 000119 1789684457.454474 trunc from 0x03e9a0 to 0x023221 (0x1b77e bytes) 000120 1789684457.474606 mapwrite 0x0184d5 thru 0x02493a (0xc466 bytes) 000121 1789684457.553282 mapread 0x01b67e thru 0x02469c (0x901f bytes) 000123 1789684457.554641 punch from 0x006a4f to 0x00c5f8 (0x5baa bytes) 000124 1789684457.559783 write 0x03889c thru 0x03ac89 (0x23ee bytes) 000126 1789684457.561816 write 0x01b8f8 thru 0x01d17a (0x1883 bytes) 000127 1789684457.563587 mapread 0x030c34 thru 0x03ac89 (0xa056 bytes) 000129 1789684457.598252 mapread 0x026198 thru 0x02f7be (0x9627 bytes) 000130 1789684457.641522 read 0x03913c thru 0x03ac89 (0x1b4e bytes) 000132 1789684457.642555 mapread 0x02a357 thru 0x033fe8 (0x9c92 bytes) 000135 1789684457.645549 write 0x02e98a thru 0x038837 (0x9eae bytes) 000137 1789684457.648013 mapread 0x004918 thru 0x0096f5 (0x4dde bytes) 000140 1789684457.658022 trunc from 0x03ac8a to 0x020460 (0x1a829 bytes) 000141 1789684457.675400 read 0x00489f thru 0x0111a4 (0xc906 bytes) 000142 1789684457.676174 mapwrite 0x014b9a thru 0x016c53 (0x20ba bytes) 000143 1789684457.723825 read 0x00482f thru 0x00cf21 (0x86f3 bytes) 000145 1789684457.726273 trunc from 0x020461 to 0x018bc4 (0x789c bytes) 000147 1789684457.749773 write 0x031491 thru 0x03ffff (0xeb6f bytes) 000148 1789684457.752435 trunc from 0x040000 to 0x004c3b (0x3b3c4 bytes) 000153 1789684457.773237 write 0x027e30 thru 0x0286d7 (0x8a8 bytes) 000156 1789684457.776464 write 0x00dc2a thru 0x010caf (0x3086 bytes) 000158 1789684457.787901 read 0x009300 thru 0x00d414 (0x4115 bytes) 000160 1789684457.789642 mapwrite 0x02eed7 thru 0x03d853 (0xe97d bytes) 000161 1789684457.880525 punch from 0x021f75 to 0x023d97 (0x1e23 bytes) 000162 1789684457.886105 punch from 0x01501a to 0x01891c (0x3903 bytes) truncating to largest ever: 0x3ef16 000163 1789684457.891063 trunc from 0x03d854 to 0x03ef15 (0x16c2 bytes) 000168 1789684457.912243 punch from 0x00c75c to 0x00db73 (0x1418 bytes) 000169 1789684457.917752 punch from 0x019670 to 0x024999 (0xb32a bytes) 000174 1789684457.926574 write 0x027a49 thru 0x02c396 (0x494e bytes) 000175 1789684457.927919 mapread 0x0292b7 thru 0x03727b (0xdfc5 bytes) 000178 1789684457.938405 trunc from 0x03ef16 to 0x01822a (0x26ceb bytes) 000179 1789684457.956638 mapread 0x00d5a7 thru 0x01822a (0xac84 bytes) 000180 1789684457.977424 write 0x01260c thru 0x017375 (0x4d6a bytes) 000181 1789684457.979018 mapread 0x006c84 thru 0x008990 (0x1d0d bytes) 000182 1789684457.989419 mapread 0x015ed6 thru 0x017951 (0x1a7c bytes) 000184 1789684457.990650 read 0x001db5 thru 0x00a5f2 (0x883e bytes) 000185 1789684457.991228 read 0x00af46 thru 0x015aaa (0xab65 bytes) 000186 1789684457.997478 read 0x007908 thru 0x00f2a2 (0x799b bytes) 000187 1789684457.998331 write 0x017a3f thru 0x024b52 (0xd114 bytes) 000188 1789684458.001031 mapwrite 0x0280d6 thru 0x02a455 (0x2380 bytes) 000190 1789684458.089771 trunc from 0x02a456 to 0x03e514 (0x140bf bytes) 000192 1789684458.119727 mapwrite 0x00f7c8 thru 0x0148c8 (0x5101 bytes) 000194 1789684458.225902 write 0x0318c8 thru 0x03960f (0x7d48 bytes) 000195 1789684458.252851 punch from 0x03d809 to 0x03e514 (0xd0c bytes) 000197 1789684458.302084 punch from 0x0139c0 to 0x01c2fa (0x893b bytes) 000198 1789684458.317744 mapread 0x000b8c thru 0x007823 (0x6c98 bytes) 000200 1789684458.329777 trunc from 0x03e515 to 0x012c05 (0x2b90f bytes) 000203 1789684458.379499 punch from 0x005e2e to 0x00e752 (0x8925 bytes) 000204 1789684458.402542 mapread 0x011d90 thru 0x012c05 (0xe76 bytes) 000205 1789684458.403669 mapwrite 0x000388 thru 0x00dfd9 (0xdc52 bytes) 000206 1789684458.607973 punch from 0x0064e0 to 0x00a781 (0x42a2 bytes) 000207 1789684458.613914 read 0x005533 thru 0x012c05 (0xd6d3 bytes) 000209 1789684458.626668 mapread 0x0035fa thru 0x00ac52 (0x7659 bytes) 000212 1789684458.642453 MIRROR_EXTEND to 2 mirrors 000213 1789684459.164914 MIRROR_RESYNC 2 mirrors 000214 1789684459.444969 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000215 1789684459.603404 mapread 0x00d2e4 thru 0x012c05 (0x5922 bytes) 000217 1789684459.637801 read 0x011e4e thru 0x012c05 (0xdb8 bytes) 000218 1789684459.638851 read 0x000d47 thru 0x00c9aa (0xbc64 bytes) 000219 1789684459.648798 mapwrite 0x02818d thru 0x0305fc (0x8470 bytes) 000220 1789684459.841679 mapread 0x0228f1 thru 0x02fdc4 (0xd4d4 bytes) 000221 1789684459.843551 mapread 0x0034c7 thru 0x00cfed (0x9b27 bytes) 000223 1789684459.909293 trunc from 0x0305fd to 0x018e8d (0x1776f bytes) 000225 1789684459.940947 punch from 0x014ca6 to 0x018e8d (0x41e8 bytes) 000226 1789684459.946284 mapwrite 0x03e0fe thru 0x03ffff (0x1f02 bytes) 000231 1789684460.113607 mapread 0x031293 thru 0x034e55 (0x3bc3 bytes) 000234 1789684460.116083 mapwrite 0x0357ea thru 0x039374 (0x3b8b bytes) 000236 1789684460.158426 punch from 0x029f96 to 0x038cb1 (0xed1c bytes) 000238 1789684460.163885 punch from 0x03e7b3 to 0x03ffff (0x184d bytes) 000240 1789684460.169584 mapwrite 0x026e27 thru 0x029bad (0x2d87 bytes) 000241 1789684460.233793 MIRROR_EXTEND to 2 mirrors 000242 1789684460.874222 mapwrite 0x035652 thru 0x03c135 (0x6ae4 bytes) 000244 1789684461.044382 mapwrite 0x009551 thru 0x00bf41 (0x29f1 bytes) 000246 1789684461.107437 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000247 1789684461.264554 trunc from 0x040000 to 0x003245 (0x3cdba bytes) 000248 1789684461.317073 write 0x03d656 thru 0x03ffff (0x29aa bytes) 000249 1789684461.318434 write 0x024d0d thru 0x02f6f5 (0xa9e9 bytes) 000250 1789684461.334316 write 0x00b3cf thru 0x00b8bb (0x4ed bytes) 000252 1789684461.344428 mapread 0x016d10 thru 0x01abc2 (0x3eb3 bytes) 000253 1789684461.362763 read 0x03d4a0 thru 0x03e5fe (0x115f bytes) 000255 1789684461.364847 trunc from 0x040000 to 0x0077a7 (0x38858 bytes) 000256 1789684461.380305 write 0x005db9 thru 0x011374 (0xb5bc bytes) 000257 1789684461.388152 read 0x010d2f thru 0x011374 (0x646 bytes) 000258 1789684461.389174 mapwrite 0x03cb80 thru 0x03ffff (0x3480 bytes) 000259 1789684461.477511 read 0x00df3e thru 0x01064f (0x2712 bytes) 000261 1789684461.479323 mapread 0x000a97 thru 0x002def (0x2359 bytes) 000263 1789684461.481097 trunc from 0x040000 to 0x01616c (0x29e93 bytes) 000264 1789684461.497328 mapread 0x00fa63 thru 0x011840 (0x1dde bytes) 000267 1789684461.500475 trunc from 0x01616d to 0x035ac1 (0x1f955 bytes) 000269 1789684461.524517 mapread 0x02885a thru 0x02f69b (0x6e42 bytes) 000270 1789684461.536468 punch from 0x00adf4 to 0x013c5f (0x8e6c bytes) 000272 1789684461.542730 mapread 0x01f6ca thru 0x02d0af (0xd9e6 bytes) 000274 1789684461.544757 mapread 0x02d39d thru 0x030b6d (0x37d1 bytes) 000276 1789684461.547143 mapread 0x027c9b thru 0x035ac1 (0xde27 bytes) 000277 1789684461.548445 write 0x00f23a thru 0x01c1ab (0xcf72 bytes) 000278 1789684461.554923 trunc from 0x035ac2 to 0x0099b7 (0x2c10a bytes) 000279 1789684461.570015 write 0x0186d8 thru 0x01c086 (0x39af bytes) 000281 1789684461.571487 punch from 0x00f651 to 0x0146ec (0x509c bytes) 000287 1789684461.583461 mapwrite 0x00054d thru 0x005ede (0x5992 bytes) 000290 1789684461.630718 mapwrite 0x003f8c thru 0x010493 (0xc508 bytes) 000292 1789684461.726047 read 0x00ded3 thru 0x015b58 (0x7c86 bytes) 000294 1789684461.737248 mapread 0x013528 thru 0x01c086 (0x8b5f bytes) 000295 1789684461.738901 punch from 0x00f184 to 0x018892 (0x970f bytes) 000297 1789684461.744321 write 0x03d7df thru 0x03ffff (0x2821 bytes) 000298 1789684461.745737 write 0x033142 thru 0x0350e4 (0x1fa3 bytes) 000299 1789684461.753792 read 0x01b0a3 thru 0x01e975 (0x38d3 bytes) 000300 1789684461.754816 punch from 0x03ab00 to 0x03ffff (0x5500 bytes) 000301 1789684461.765762 mapwrite 0x027af9 thru 0x02dc65 (0x616d bytes) 000303 1789684461.812174 mapread 0x017206 thru 0x01c727 (0x5522 bytes) 000305 1789684461.826344 read 0x02aca5 thru 0x0314b5 (0x6811 bytes) 000307 1789684461.828346 mapwrite 0x01f8f8 thru 0x021eff (0x2608 bytes) 000308 1789684461.869251 trunc from 0x040000 to 0x020629 (0x1f9d6 bytes) 000315 1789684461.901419 trunc from 0x02062a to 0x01e876 (0x1db3 bytes) 000316 1789684461.933181 read 0x001829 thru 0x00bf38 (0xa710 bytes) 000317 1789684461.933882 write 0x01f0dc thru 0x02b4b9 (0xc3de bytes) 000318 1789684461.936258 mapwrite 0x02aac2 thru 0x037854 (0xcd93 bytes) 000319 1789684462.054557 read 0x013bc4 thru 0x01c5ac (0x89e9 bytes) 000320 1789684462.056550 mapread 0x01e8c5 thru 0x022809 (0x3f45 bytes) 000321 1789684462.060947 trunc from 0x037855 to 0x0336e1 (0x4173 bytes) 000324 1789684462.109009 trunc from 0x0336e2 to 0x03580f (0x212e bytes) 000327 1789684462.127687 punch from 0x00d3bf to 0x017bd0 (0xa812 bytes) 000331 1789684462.149165 mapwrite 0x02e2e9 thru 0x0343cb (0x60e3 bytes) 000332 1789684462.263510 write 0x005f51 thru 0x006b11 (0xbc1 bytes) 000333 1789684462.265975 trunc from 0x035810 to 0x03c2ec (0x6add bytes) 000334 1789684462.292793 punch from 0x012b11 to 0x017a3c (0x4f2c bytes) 000336 1789684462.303755 read 0x02f19e thru 0x037357 (0x81ba bytes) 000339 1789684462.312703 read 0x023b75 thru 0x028d76 (0x5202 bytes) 000340 1789684462.313317 mapwrite 0x03e59e thru 0x03ffff (0x1a62 bytes) 000343 1789684462.408165 mapwrite 0x010529 thru 0x01fccd (0xf7a5 bytes) 000344 1789684462.558442 punch from 0x03106e to 0x03f282 (0xe215 bytes) 000345 1789684462.566482 read 0x036ceb thru 0x03ffff (0x9315 bytes) 000346 1789684462.593705 mapwrite 0x007a9e thru 0x015934 (0xde97 bytes) 000349 1789684462.691423 write 0x0046d7 thru 0x00f05d (0xa987 bytes) 000350 1789684462.693510 write 0x02f893 thru 0x036e06 (0x7574 bytes) 000353 1789684462.700253 mapwrite 0x01604e thru 0x02130a (0xb2bd bytes) 000355 1789684462.778714 trunc from 0x040000 to 0x00c029 (0x33fd6 bytes) 000357 1789684462.807199 write 0x02b73a thru 0x02bc79 (0x540 bytes) 000358 1789684462.808118 write 0x033f71 thru 0x034942 (0x9d2 bytes) 000360 1789684462.817022 trunc from 0x034943 to 0x024ae6 (0xfe5c bytes) 000361 1789684462.837758 mapread 0x022dc6 thru 0x024ae6 (0x1d21 bytes) 000362 1789684462.854742 mapwrite 0x021297 thru 0x022fd4 (0x1d3e bytes) 000363 1789684462.911210 MIRROR_EXTEND to 2 mirrors 000364 1789684463.649925 write 0x01024b thru 0x010c18 (0x9ce bytes) 000366 1789684463.732466 read 0x015e12 thru 0x024ae6 (0xecd5 bytes) 000367 1789684463.750452 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24ae7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ae7): Mirror 2: 0xe7db5567 (stale) CRC-32 checksum value for chunk [0, 0x24ae7): Mirror 1: 0xae14718f Mirror 3: 0xae14718f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ae7: skipped 000368 1789684464.626279 write 0x019f15 thru 0x021385 (0x7471 bytes) 000369 1789684464.711679 trunc from 0x024ae7 to 0x02b3ee (0x6908 bytes) 000370 1789684464.737427 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: 0xae2ac0e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: 0xae2ac0e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3ef: skipped 000375 1789684465.033465 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: 0xae2ac0e4 (stale) CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 1: 0x6b76f2a4 Mirror 4: 0x6b76f2a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3ef: skipped 000376 1789684466.145892 trunc from 0x02b3ef to 0x005d3a (0x256b4 bytes) 000377 1789684466.283017 mapread 0x000553 thru 0x005d3a (0x57e8 bytes) 000378 1789684466.303310 mapwrite 0x02c5eb thru 0x030dd7 (0x47ed bytes) 000380 1789684466.406145 mapwrite 0x00d078 thru 0x017d73 (0xacfc bytes) 000381 1789684466.477803 write 0x01a4a0 thru 0x023490 (0x8ff1 bytes) 000382 1789684466.485827 read 0x011fc8 thru 0x012d51 (0xd8a bytes) 000383 1789684466.493307 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 3: 0xe55de64a (stale) Mirror 4: 0x437048db (stale) CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 1: 0xe6e1d711 Mirror 5: 0xe6e1d711 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30dd8: skipped 000384 1789684467.407233 mapread 0x002567 thru 0x0057bf (0x3259 bytes) 000385 1789684467.442028 MIRROR_RESYNC 4 mirrors 000386 1789684468.679298 punch from 0x00d841 to 0x01c032 (0xe7f2 bytes) 000387 1789684468.738317 punch from 0x018453 to 0x01e203 (0x5db1 bytes) 000388 1789684468.742980 read 0x02641a thru 0x027c45 (0x182c bytes) 000390 1789684468.749524 mapread 0x00615a thru 0x01598c (0xf833 bytes) 000391 1789684468.787603 punch from 0x003895 to 0x00cadd (0x9249 bytes) 000392 1789684468.795801 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 4: 0xe6e1d711 (stale) Mirror 5: 0xe6e1d711 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 4: 0xe6e1d711 (stale) Mirror 5: 0xe6e1d711 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30dd8: skipped 000393 1789684469.186997 write 0x039ffb thru 0x03ffff (0x6005 bytes) 000394 1789684469.191054 mapwrite 0x002e58 thru 0x00304a (0x1f3 bytes) 000395 1789684469.264090 mapread 0x016bd6 thru 0x018c7a (0x20a5 bytes) 000396 1789684469.274136 punch from 0x01ecbf to 0x021669 (0x29ab bytes) 000397 1789684469.279717 read 0x036096 thru 0x03ffff (0x9f6a bytes) 000398 1789684469.292029 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x54be7e1a (stale) Mirror 5: 0x54be7e1a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5a2b2242 Mirror 6: 0x5a2b2242 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000399 1789684470.653231 punch from 0x0138eb to 0x01e368 (0xaa7e bytes) 000400 1789684470.825316 MIRROR_RESYNC 4 mirrors 000401 1789684473.207076 trunc from 0x040000 to 0x037ec5 (0x813a bytes) 000402 1789684473.293537 punch from 0x01feaa to 0x023857 (0x39ae bytes) 000403 1789684473.302839 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x765032ad (stale) Mirror 6: 0x765032ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x765032ad (stale) Mirror 6: 0x765032ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec6: skipped 000406 1789684473.565496 punch from 0x003e34 to 0x00493b (0xb08 bytes) 000407 1789684473.570345 write 0x0217e1 thru 0x0260e0 (0x4900 bytes) 000408 1789684473.580179 punch from 0x027f93 to 0x0286df (0x74d bytes) 000409 1789684473.592618 write 0x037355 thru 0x037629 (0x2d5 bytes) 000410 1789684473.600906 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x765032ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x765032ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec6: skipped 000411 1789684473.995326 read 0x01c130 thru 0x022dd1 (0x6ca2 bytes) 000412 1789684474.003564 trunc from 0x037ec6 to 0x01592b (0x2259a bytes) 000413 1789684474.020500 mapwrite 0x006003 thru 0x00e037 (0x8035 bytes) 000417 1789684474.103663 trunc from 0x01592c to 0x00cf43 (0x89e8 bytes) 000418 1789684474.128616 punch from 0x008423 to 0x00cf43 (0x4b21 bytes) 000422 1789684474.139467 read 0x003796 thru 0x0064a4 (0x2d0f bytes) 000423 1789684474.140564 trunc from 0x00cf44 to 0x015557 (0x8614 bytes) 000424 1789684474.166790 trunc from 0x015558 to 0x00f39f (0x61b8 bytes) 000425 1789684474.184438 mapread 0x00773e thru 0x00f39f (0x7c62 bytes) 000426 1789684474.193936 mapread 0x003dc3 thru 0x006a1e (0x2c5c bytes) 000428 1789684474.195280 write 0x030d78 thru 0x03458e (0x3817 bytes) 000429 1789684474.196371 write 0x020091 thru 0x02d74a (0xd6ba bytes) 000430 1789684474.204685 punch from 0x002a86 to 0x00849c (0x5a17 bytes) 000431 1789684474.223633 read 0x02b315 thru 0x02e0bd (0x2da9 bytes) 000432 1789684474.224523 mapwrite 0x01618f thru 0x01e6d8 (0x854a bytes) 000433 1789684474.266709 write 0x01919c thru 0x01d062 (0x3ec7 bytes) 000434 1789684474.267965 trunc from 0x03458f to 0x023786 (0x10e08 bytes) 000435 1789684474.280832 punch from 0x010f2f to 0x01ba61 (0xab33 bytes) 000437 1789684474.295558 write 0x03b716 thru 0x03cb9b (0x1486 bytes) 000438 1789684474.296766 mapread 0x021d15 thru 0x031c06 (0xfef2 bytes) 000441 1789684474.323198 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000442 1789684474.482069 trunc from 0x03cb9c to 0x03c616 (0x585 bytes) 000444 1789684474.509633 write 0x03f949 thru 0x03ffff (0x6b7 bytes) 000445 1789684474.510998 write 0x025765 thru 0x030220 (0xaabc bytes) 000446 1789684474.519647 mapwrite 0x00a2ce thru 0x0135ae (0x92e1 bytes) 000448 1789684474.584182 read 0x018983 thru 0x01af63 (0x25e1 bytes) 000449 1789684474.593276 read 0x03b532 thru 0x03ffff (0x4ace bytes) 000450 1789684474.602544 trunc from 0x040000 to 0x028ba7 (0x17458 bytes) 000451 1789684474.617000 write 0x0023f2 thru 0x00d059 (0xac68 bytes) 000454 1789684474.624025 punch from 0x01f232 to 0x028ba7 (0x9976 bytes) 000456 1789684474.637221 write 0x00802e thru 0x016b3a (0xeb0d bytes) 000458 1789684474.640322 mapwrite 0x015c53 thru 0x017efd (0x22ab bytes) 000460 1789684474.691164 mapwrite 0x025f84 thru 0x028703 (0x2780 bytes) 000461 1789684474.737777 mapread 0x017905 thru 0x01d32f (0x5a2b bytes) 000462 1789684474.740038 mapread 0x024e42 thru 0x028ba7 (0x3d66 bytes) 000464 1789684474.744906 punch from 0x0225df to 0x028ba7 (0x65c9 bytes) 000469 1789684474.751646 write 0x031e2b thru 0x035f18 (0x40ee bytes) 000470 1789684474.754143 read 0x012dfb thru 0x01ec33 (0xbe39 bytes) 000472 1789684474.755634 read 0x01ef0e thru 0x029370 (0xa463 bytes) 000474 1789684474.764205 mapread 0x010587 thru 0x012823 (0x229d bytes) 000475 1789684474.765199 trunc from 0x035f19 to 0x001cb8 (0x34260 bytes) 000476 1789684474.780084 write 0x030d8c thru 0x031ea2 (0x1117 bytes) 000479 1789684474.782345 read 0x00dcf4 thru 0x01943a (0xb747 bytes) 000480 1789684474.793724 punch from 0x011d1d to 0x01f6df (0xd9c3 bytes) 000481 1789684474.804716 trunc from 0x031ea3 to 0x009410 (0x28a92 bytes) 000483 1789684474.818018 mapread 0x000f95 thru 0x009410 (0x847c bytes) 000488 1789684474.821037 mapwrite 0x020a85 thru 0x02feff (0xf47b bytes) 000490 1789684475.018067 punch from 0x02660b to 0x02feff (0x98f5 bytes) 000492 1789684475.023578 write 0x0028e8 thru 0x00ef5a (0xc673 bytes) 000493 1789684475.030052 read 0x0109a5 thru 0x01cc2b (0xc287 bytes) 000494 1789684475.044190 write 0x01add1 thru 0x01d1c8 (0x23f8 bytes) 000496 1789684475.046261 write 0x035e95 thru 0x036323 (0x48f bytes) 000497 1789684475.047334 write 0x01c64c thru 0x029cef (0xd6a4 bytes) 000498 1789684475.049404 write 0x03cb7e thru 0x03ffff (0x3482 bytes) 000500 1789684475.058515 MIRROR_EXTEND to 2 mirrors 000501 1789684476.094063 punch from 0x039faa to 0x03ffff (0x6056 bytes) 000503 1789684476.286753 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcac7cfc9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x71eb307e Mirror 3: 0x71eb307e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000504 1789684477.474005 write 0x00ba08 thru 0x01389e (0x7e97 bytes) 000505 1789684477.561091 mapwrite 0x02a05a thru 0x035681 (0xb628 bytes) 000507 1789684477.657690 write 0x02111c thru 0x023cf1 (0x2bd6 bytes) 000508 1789684477.679638 mapread 0x02b675 thru 0x03045e (0x4dea bytes) 000509 1789684477.680891 trunc from 0x040000 to 0x0195fc (0x26a03 bytes) 000510 1789684477.711530 trunc from 0x0195fd to 0x03db09 (0x2450d bytes) 000512 1789684477.728539 mapwrite 0x03ab12 thru 0x03ffff (0x54ee bytes) 000513 1789684477.810041 punch from 0x001e5d to 0x01136c (0xf510 bytes) 000514 1789684477.815634 write 0x004c4f thru 0x00884f (0x3c01 bytes) 000515 1789684477.826965 trunc from 0x040000 to 0x01ae90 (0x2516f bytes) 000516 1789684477.845280 read 0x01a98b thru 0x01ae90 (0x506 bytes) 000517 1789684477.865147 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ae91): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ae91): Mirror 2: 0x89fd5cf1 (stale) Mirror 3: 0x89fd5cf1 (stale) CRC-32 checksum value for chunk [0, 0x1ae91): Mirror 1: 0xe086fc7b Mirror 4: 0xe086fc7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ae91: skipped 000518 1789684479.150577 read 0x019c52 thru 0x01ae90 (0x123f bytes) 000519 1789684479.165559 write 0x03b874 thru 0x03ffff (0x478c bytes) 000520 1789684479.318775 mapread 0x034cae thru 0x03ffff (0xb352 bytes) 000522 1789684479.372000 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcac7cfc9 (stale) Mirror 4: 0xbc7ee53e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcac7cfc9 (stale) Mirror 4: 0xbc7ee53e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000523 1789684479.826752 write 0x0140b1 thru 0x0232cc (0xf21c bytes) 000524 1789684479.858024 mapread 0x0391d4 thru 0x039fc7 (0xdf4 bytes) 000525 1789684479.864011 write 0x009456 thru 0x01530b (0xbeb6 bytes) 000526 1789684479.872326 punch from 0x022760 to 0x02b534 (0x8dd5 bytes) 000528 1789684479.906397 read 0x03f4f0 thru 0x03ffff (0xb10 bytes) 000530 1789684479.915802 mapread 0x01dae7 thru 0x02aa83 (0xcf9d bytes) 000531 1789684479.928458 trunc from 0x040000 to 0x02fa86 (0x10579 bytes) 000532 1789684479.949754 punch from 0x02d68b to 0x02f9ee (0x2364 bytes) 000534 1789684479.955848 mapread 0x00f57d thru 0x019fbf (0xaa43 bytes) 000535 1789684479.957438 mapread 0x0108d6 thru 0x01af78 (0xa6a3 bytes) 000536 1789684479.958645 mapread 0x025922 thru 0x028e0b (0x34ea bytes) 000537 1789684479.959571 write 0x02173d thru 0x027974 (0x6238 bytes) 000538 1789684479.961534 write 0x0279a6 thru 0x031a17 (0xa072 bytes) 000539 1789684479.962960 trunc from 0x031a18 to 0x031502 (0x515 bytes) 000541 1789684479.983068 mapread 0x018f60 thru 0x02061f (0x76c0 bytes) 000542 1789684479.984563 trunc from 0x031503 to 0x03459e (0x309c bytes) 000543 1789684480.000636 trunc from 0x03459f to 0x021ab2 (0x12aec bytes) 000544 1789684480.023408 mapwrite 0x023550 thru 0x031cd7 (0xe788 bytes) 000545 1789684480.087664 mapwrite 0x02ab14 thru 0x0307f6 (0x5ce3 bytes) 000546 1789684480.129497 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31cd8): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31cd8): Mirror 2: 0x319ba03d (stale) Mirror 4: 0x761f3bf3 (stale) CRC-32 checksum value for chunk [0, 0x31cd8): Mirror 1: 0xc497f4f2 Mirror 5: 0xc497f4f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31cd8: skipped 000547 1789684481.222986 mapread 0x00d6c2 thru 0x012040 (0x497f bytes) 000549 1789684481.315256 mapwrite 0x039626 thru 0x03eeb1 (0x588c bytes) 000551 1789684481.545217 mapread 0x012cf1 thru 0x014e1d (0x212d bytes) 000552 1789684481.556093 write 0x0317b1 thru 0x03b076 (0x98c6 bytes) 000553 1789684481.561727 trunc from 0x03eeb2 to 0x0379ac (0x7505 bytes) 000554 1789684481.596943 mapwrite 0x01c272 thru 0x026a01 (0xa790 bytes) 000557 1789684481.686913 read 0x001680 thru 0x00bac2 (0xa443 bytes) 000558 1789684481.708144 punch from 0x036336 to 0x0379ac (0x1677 bytes) 000559 1789684481.722559 mapwrite 0x00ff3a thru 0x0130b1 (0x3178 bytes) 000560 1789684481.770769 mapwrite 0x03d18a thru 0x03ffff (0x2e76 bytes) 000562 1789684481.877777 mapwrite 0x013a2c thru 0x016fab (0x3580 bytes) 000563 1789684481.934165 read 0x01590e thru 0x01c576 (0x6c69 bytes) 000564 1789684481.934769 read 0x024982 thru 0x0252c3 (0x942 bytes) 000565 1789684481.935319 mapwrite 0x018bae thru 0x028a3c (0xfe8f bytes) 000566 1789684482.034163 mapwrite 0x03e0d5 thru 0x03ffff (0x1f2b bytes) 000567 1789684482.140349 read 0x01bb10 thru 0x027aae (0xbf9f bytes) 000568 1789684482.145943 mapread 0x0139ac thru 0x020259 (0xc8ae bytes) 000569 1789684482.157383 write 0x0361c5 thru 0x03ffff (0x9e3b bytes) 000570 1789684482.158712 trunc from 0x040000 to 0x0010ae (0x3ef51 bytes) 000571 1789684482.211066 punch from 0x0007bf to 0x0010ae (0x8f0 bytes) 000573 1789684482.230269 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x10af): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10af): Mirror 2: 0x41ca0ef9 (stale) Mirror 4: 0x41ca0ef9 (stale) Mirror 5: 0x41ca0ef9 (stale) CRC-32 checksum value for chunk [0, 0x10af): Mirror 1: 0x4be136fb Mirror 6: 0x4be136fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10af: skipped 000576 1789684483.376652 read 0x000064 thru 0x0010ae (0x104b bytes) 000577 1789684483.392915 write 0x008783 thru 0x00ebb3 (0x6431 bytes) 000579 1789684483.507189 mapread 0x00e67d thru 0x00ebb3 (0x537 bytes) 000581 1789684483.517836 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xebb4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebb4): Mirror 2: 0x1299eee8 (stale) Mirror 4: 0x3e9dd6b7 (stale) Mirror 5: 0xb80adbdf (stale) Mirror 6: 0x5b598edf (stale) CRC-32 checksum value for chunk [0, 0xebb4): Mirror 1: 0xa7c12270 Mirror 7: 0xa7c12270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebb4: skipped 000582 1789684485.355518 mapwrite 0x000fa4 thru 0x004913 (0x3970 bytes) 000584 1789684485.631412 write 0x039d57 thru 0x03ffff (0x62a9 bytes) 000585 1789684485.635309 trunc from 0x040000 to 0x0191c9 (0x26e36 bytes) 000587 1789684485.704811 punch from 0x011b32 to 0x0191c9 (0x7698 bytes) 000588 1789684485.710435 write 0x009b45 thru 0x009f54 (0x410 bytes) 000589 1789684485.713421 mapwrite 0x00b63f thru 0x019b7a (0xe53c bytes) 000590 1789684485.907580 write 0x00fde6 thru 0x01b2d2 (0xb4ed bytes) 000591 1789684485.913595 read 0x00e68c thru 0x012ac6 (0x443b bytes) 000592 1789684485.916337 mapread 0x009a3c thru 0x01217e (0x8743 bytes) 000593 1789684485.919884 write 0x005e6b thru 0x0070be (0x1254 bytes) 000594 1789684485.928793 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b2d3): Mirror 4: 0x7d39812e (stale) Mirror 5: 0xf686bf33 (stale) Mirror 6: 0x6a4af724 (stale) Mirror 7: 0x8518f2f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b2d3): Mirror 4: 0x7d39812e (stale) Mirror 5: 0xf686bf33 (stale) Mirror 6: 0x6a4af724 (stale) Mirror 7: 0x8518f2f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b2d3: skipped 000595 1789684486.509555 mapread 0x00343c thru 0x009f0b (0x6ad0 bytes) 000597 1789684486.550838 write 0x027119 thru 0x030bdf (0x9ac7 bytes) 000598 1789684486.552279 write 0x035300 thru 0x03ffff (0xad00 bytes) 000599 1789684486.559745 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6e578003 (stale) Mirror 6: 0xd226e743 (stale) Mirror 7: 0xb6e410f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6e578003 (stale) Mirror 6: 0xd226e743 (stale) Mirror 7: 0xb6e410f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1789684486.990735 read 0x038cb8 thru 0x0390c6 (0x40f bytes) 000601 1789684486.995683 trunc from 0x040000 to 0x010c10 (0x2f3ef bytes) 000602 1789684487.018953 punch from 0x0040f6 to 0x00f4e4 (0xb3ef bytes) 000603 1789684487.025611 punch from 0x001dd1 to 0x0082bb (0x64eb bytes) 000604 1789684487.032087 read 0x00b882 thru 0x010c10 (0x538f bytes) 000605 1789684487.040741 write 0x02b89a thru 0x0347c8 (0x8f2f bytes) 000607 1789684487.043159 trunc from 0x0347c9 to 0x00e30e (0x264ba bytes) 000608 1789684487.056631 read 0x00737d thru 0x00e30e (0x6f92 bytes) 000609 1789684487.057353 write 0x02dd8f thru 0x034f28 (0x719a bytes) 000610 1789684487.058555 write 0x0268ff thru 0x030c7a (0xa37c bytes) 000613 1789684487.067300 mapwrite 0x015377 thru 0x01729b (0x1f25 bytes) 000615 1789684487.112846 mapwrite 0x02182e thru 0x02ae0e (0x95e1 bytes) 000616 1789684487.167596 trunc from 0x034f29 to 0x022155 (0x12dd3 bytes) 000617 1789684487.186277 read 0x01b367 thru 0x022155 (0x6def bytes) 000618 1789684487.189024 read 0x00e5f2 thru 0x0103b1 (0x1dc0 bytes) 000621 1789684487.191888 punch from 0x00a91c to 0x00dece (0x35b3 bytes) 000622 1789684487.197786 mapwrite 0x015349 thru 0x015a15 (0x6cd bytes) 000623 1789684487.259362 write 0x00694e thru 0x008538 (0x1beb bytes) 000624 1789684487.260302 read 0x020940 thru 0x022155 (0x1816 bytes) 000625 1789684487.260749 trunc from 0x022156 to 0x002d33 (0x1f422 bytes) 000626 1789684487.295309 MIRROR_RESYNC 4 mirrors 000627 1789684487.976632 punch from 0x001da1 to 0x002d33 (0xf93 bytes) 000628 1789684488.045308 punch from 0x0023fc to 0x002d33 (0x938 bytes) 000629 1789684488.050437 mapread 0x002a2c thru 0x002d33 (0x308 bytes) 000630 1789684488.055686 mapread 0x002af9 thru 0x002d33 (0x23b bytes) 000631 1789684488.061318 write 0x0016df thru 0x00cc74 (0xb596 bytes) 000632 1789684488.066394 mapread 0x009564 thru 0x00cc74 (0x3711 bytes) 000633 1789684488.072007 MIRROR_RESYNC 4 mirrors 000635 1789684488.834761 mapread 0x004a0b thru 0x00cc74 (0x826a bytes) 000636 1789684488.852212 write 0x03e402 thru 0x03ffff (0x1bfe bytes) 000638 1789684488.912006 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x19cc8b4d (stale) Mirror 6: 0x19cc8b4d (stale) Mirror 7: 0x19cc8b4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8df4450 Mirror 8: 0x8df4450 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000639 1789684489.636327 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x19cc8b4d (stale) Mirror 6: 0x19cc8b4d (stale) Mirror 7: 0x19cc8b4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8df4450 Mirror 8: 0x8df4450 Mirror 9: 0x8df4450 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000640 1789684490.402619 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x19cc8b4d (stale) Mirror 6: 0x19cc8b4d (stale) Mirror 7: 0x19cc8b4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8df4450 Mirror 8: 0x8df4450 Mirror 9: 0x8df4450 Mirror 10: 0x8df4450 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000641 1789684491.352248 read 0x0124c9 thru 0x013f89 (0x1ac1 bytes) 000642 1789684491.361315 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x19cc8b4d (stale) Mirror 7: 0x19cc8b4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x19cc8b4d (stale) Mirror 7: 0x19cc8b4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8df4450 Mirror 8: 0x8df4450 Mirror 9: 0x8df4450 Mirror 10: 0x8df4450 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000643 1789684492.077245 mapread 0x01b4ce thru 0x020850 (0x5383 bytes) 000644 1789684492.124049 punch from 0x024ed2 to 0x033c97 (0xedc6 bytes) 000645 1789684492.262571 mapread 0x039f2a thru 0x03a844 (0x91b bytes) 000647 1789684492.282759 trunc from 0x040000 to 0x01d14b (0x22eb4 bytes) 000649 1789684492.308051 read 0x011215 thru 0x017aaa (0x6896 bytes) truncating to largest ever: 0x3fe3f 000650 1789684492.334726 trunc from 0x01d14c to 0x03fe3e (0x22cf3 bytes) 000651 1789684492.375539 trunc from 0x03fe3f to 0x0282a1 (0x17b9d bytes) 000652 1789684492.410193 trunc from 0x0282a2 to 0x028879 (0x5d8 bytes) 000653 1789684492.426541 trunc from 0x02887a to 0x032a0a (0xa191 bytes) 000654 1789684492.441352 MIRROR_RESYNC 6 mirrors 000655 1789684493.609226 punch from 0x0132fb to 0x01389b (0x5a1 bytes) 000656 1789684493.658400 write 0x0207fc thru 0x02bf67 (0xb76c bytes) 000658 1789684493.666968 mapread 0x0081ec thru 0x016adb (0xe8f0 bytes) 000659 1789684493.687233 write 0x034eb0 thru 0x03b090 (0x61e1 bytes) 000660 1789684493.688586 mapwrite 0x03de66 thru 0x03ffff (0x219a bytes) 000661 1789684493.758809 write 0x000734 thru 0x001fbc (0x1889 bytes) 000662 1789684493.770289 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x19cc8b4d (stale) Mirror 7: 0x19cc8b4d (stale) Mirror 8: 0x19cc8b4d (stale) Mirror 9: 0x19cc8b4d (stale) Mirror 10: 0x19cc8b4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ea636d Mirror 11: 0x8ea636d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000663 1789684494.557123 punch from 0x023b12 to 0x029f08 (0x63f7 bytes) 000664 1789684494.637156 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x19cc8b4d (stale) Mirror 7: 0x19cc8b4d (stale) Mirror 8: 0x19cc8b4d (stale) Mirror 9: 0x19cc8b4d (stale) Mirror 10: 0x19cc8b4d (stale) Mirror 11: 0x8ea636d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc37f5c96 Mirror 12: 0xc37f5c96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000665 1789684495.532218 mapread 0x024c90 thru 0x0349bb (0xfd2c bytes) 000667 1789684495.569637 mapwrite 0x01bc2c thru 0x0211b4 (0x5589 bytes) 000668 1789684495.704725 trunc from 0x040000 to 0x0066df (0x39920 bytes) 000671 1789684495.725111 MIRROR_RESYNC 8 mirrors 000672 1789684496.511835 write 0x002854 thru 0x00ebc0 (0xc36d bytes) 000673 1789684496.547003 mapread 0x006908 thru 0x00bc17 (0x5310 bytes) 000676 1789684496.549136 write 0x02c2d3 thru 0x037783 (0xb4b1 bytes) 000677 1789684496.550628 trunc from 0x037784 to 0x00a918 (0x2ce6b bytes) 000679 1789684496.562525 punch from 0x0011a5 to 0x008dd2 (0x7c2e bytes) 000680 1789684496.571054 mapread 0x00a1f9 thru 0x00a918 (0x720 bytes) 000681 1789684496.571736 trunc from 0x00a919 to 0x0111c3 (0x68ab bytes) 000682 1789684496.582230 trunc from 0x0111c4 to 0x03c57f (0x2b3bc bytes) 000683 1789684496.598248 mapread 0x01eb62 thru 0x02b170 (0xc60f bytes) 000684 1789684496.618562 MIRROR_RESYNC 8 mirrors 000685 1789684497.960082 MIRROR_RESYNC 8 mirrors 000686 1789684498.252645 trunc from 0x03c580 to 0x02a5d6 (0x11fa9 bytes) 000688 1789684498.297716 trunc from 0x02a5d7 to 0x039fb9 (0xf9e3 bytes) 000690 1789684498.310285 mapread 0x0362bb thru 0x039fb9 (0x3cff bytes) 000691 1789684498.321575 punch from 0x02bfb3 to 0x02eca2 (0x2cf0 bytes) 000692 1789684498.323930 read 0x019b3e thru 0x021645 (0x7b08 bytes) 000696 1789684498.329543 write 0x02d0c6 thru 0x02f849 (0x2784 bytes) 000697 1789684498.335793 trunc from 0x039fba to 0x000432 (0x39b87 bytes) 000698 1789684498.347783 read 0x00030a thru 0x000432 (0x129 bytes) 000699 1789684498.351257 write 0x021d9e thru 0x029ea1 (0x8104 bytes) 000701 1789684498.353144 punch from 0x026c29 to 0x029ea1 (0x3279 bytes) 000702 1789684498.363474 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 5: 0x664c8887 (stale) Mirror 7: 0x664c8887 (stale) Mirror 8: 0x664c8887 (stale) Mirror 9: 0x664c8887 (stale) Mirror 10: 0x664c8887 (stale) Mirror 11: 0x664c8887 (stale) Mirror 12: 0x664c8887 (stale) CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 1: 0x3a89ed84 Mirror 13: 0x3a89ed84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ea2: skipped 000703 1789684499.177281 MIRROR_RESYNC 9 mirrors 000704 1789684500.890575 punch from 0x00beed to 0x0135e4 (0x76f8 bytes) 000705 1789684501.007825 mapread 0x025c5a thru 0x029ea1 (0x4248 bytes) 000706 1789684501.032718 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 5: 0x3a89ed84 (stale) Mirror 7: 0x3a89ed84 (stale) Mirror 8: 0x3a89ed84 (stale) Mirror 9: 0x3a89ed84 (stale) Mirror 10: 0x3a89ed84 (stale) Mirror 11: 0x3a89ed84 (stale) Mirror 12: 0x3a89ed84 (stale) Mirror 13: 0x3a89ed84 (stale) CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 1: 0x3a89ed84 Mirror 14: 0x3a89ed84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ea2: skipped 000707 1789684501.982659 read 0x01b696 thru 0x0233f3 (0x7d5e bytes) 000708 1789684501.989673 write 0x0013c8 thru 0x00671c (0x5355 bytes) 000709 1789684502.048259 read 0x00111d thru 0x008519 (0x73fd bytes) 000710 1789684502.055721 trunc from 0x029ea2 to 0x030661 (0x67c0 bytes) 000711 1789684502.072757 trunc from 0x030662 to 0x00d096 (0x235cb bytes) 000712 1789684502.087402 write 0x032747 thru 0x036d11 (0x45cb bytes) 000713 1789684502.090662 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36d12): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d12): Mirror 5: 0x3ff9a6b (stale) Mirror 7: 0x3ff9a6b (stale) Mirror 8: 0x3ff9a6b (stale) Mirror 9: 0x3ff9a6b (stale) Mirror 10: 0x3ff9a6b (stale) Mirror 11: 0x3ff9a6b (stale) Mirror 12: 0x3ff9a6b (stale) Mirror 13: 0x3ff9a6b (stale) Mirror 14: 0x3ff9a6b (stale) CRC-32 checksum value for chunk [0, 0x36d12): Mirror 1: 0xafa8ed68 Mirror 15: 0xafa8ed68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d12: skipped 000717 1789684503.394673 write 0x0193c8 thru 0x02187b (0x84b4 bytes) 000718 1789684503.580108 mapwrite 0x012d9b thru 0x0222b8 (0xf51e bytes) 000720 1789684503.757693 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0x3ff9a6b (stale) Mirror 8: 0x3ff9a6b (stale) Mirror 9: 0x3ff9a6b (stale) Mirror 10: 0x3ff9a6b (stale) Mirror 11: 0x3ff9a6b (stale) Mirror 12: 0x3ff9a6b (stale) Mirror 13: 0x3ff9a6b (stale) Mirror 14: 0x3ff9a6b (stale) Mirror 15: 0xafa8ed68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0x3ff9a6b (stale) Mirror 8: 0x3ff9a6b (stale) Mirror 9: 0x3ff9a6b (stale) Mirror 10: 0x3ff9a6b (stale) Mirror 11: 0x3ff9a6b (stale) Mirror 12: 0x3ff9a6b (stale) Mirror 13: 0x3ff9a6b (stale) Mirror 14: 0x3ff9a6b (stale) Mirror 15: 0xafa8ed68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d12: skipped 000721 1789684504.580598 read 0x036880 thru 0x036d11 (0x492 bytes) 000723 1789684504.585287 mapread 0x0155fb thru 0x01a825 (0x522b bytes) 000726 1789684504.610679 mapread 0x01b834 thru 0x021855 (0x6022 bytes) 000727 1789684504.621313 mapread 0x006b65 thru 0x015a64 (0xef00 bytes) 000728 1789684504.635253 mapwrite 0x01d3b5 thru 0x0256c0 (0x830c bytes) 000729 1789684504.721190 read 0x028c04 thru 0x02c095 (0x3492 bytes) 000730 1789684504.732776 read 0x0244f8 thru 0x026b48 (0x2651 bytes) 000731 1789684504.739806 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0x3ff9a6b (stale) Mirror 8: 0x3ff9a6b (stale) Mirror 9: 0x3ff9a6b (stale) Mirror 10: 0x3ff9a6b (stale) Mirror 11: 0x3ff9a6b (stale) Mirror 12: 0x3ff9a6b (stale) Mirror 13: 0x3ff9a6b (stale) Mirror 14: 0x3ff9a6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0x3ff9a6b (stale) Mirror 8: 0x3ff9a6b (stale) Mirror 9: 0x3ff9a6b (stale) Mirror 10: 0x3ff9a6b (stale) Mirror 11: 0x3ff9a6b (stale) Mirror 12: 0x3ff9a6b (stale) Mirror 13: 0x3ff9a6b (stale) Mirror 14: 0x3ff9a6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d12: skipped 000732 1789684505.460787 mapread 0x036095 thru 0x036d11 (0xc7d bytes) 000733 1789684505.468888 read 0x015218 thru 0x01d755 (0x853e bytes) 000734 1789684505.478308 mapread 0x0157f8 thru 0x01c19d (0x69a6 bytes) 000736 1789684505.479906 punch from 0x00cee4 to 0x016b57 (0x9c74 bytes) 000737 1789684505.486100 mapread 0x02836a thru 0x036d11 (0xe9a8 bytes) 000738 1789684505.508639 read 0x0097bf thru 0x0139de (0xa220 bytes) 000739 1789684505.517767 trunc from 0x036d12 to 0x027c54 (0xf0bd bytes) 000740 1789684505.555094 trunc from 0x027c55 to 0x03ae7b (0x13227 bytes) 000741 1789684505.575671 read 0x026f00 thru 0x027563 (0x664 bytes) 000742 1789684505.582184 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: 0x41867e5a (stale) Mirror 8: 0x41867e5a (stale) Mirror 9: 0x41867e5a (stale) Mirror 10: 0x41867e5a (stale) Mirror 11: 0x41867e5a (stale) Mirror 12: 0x41867e5a (stale) Mirror 13: 0x41867e5a (stale) Mirror 14: 0x41867e5a (stale) CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 1: 0x73541678 Mirror 15: 0x73541678 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae7c: skipped 000743 1789684507.542524 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: 0x41867e5a (stale) Mirror 8: 0x41867e5a (stale) Mirror 9: 0x41867e5a (stale) Mirror 10: 0x41867e5a (stale) Mirror 11: 0x41867e5a (stale) Mirror 12: 0x41867e5a (stale) Mirror 13: 0x41867e5a (stale) Mirror 14: 0x41867e5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: 0x41867e5a (stale) Mirror 8: 0x41867e5a (stale) Mirror 9: 0x41867e5a (stale) Mirror 10: 0x41867e5a (stale) Mirror 11: 0x41867e5a (stale) Mirror 12: 0x41867e5a (stale) Mirror 13: 0x41867e5a (stale) Mirror 14: 0x41867e5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae7c: skipped 000744 1789684508.426558 punch from 0x031001 to 0x03909d (0x809d bytes) 000745 1789684508.447562 write 0x032b05 thru 0x039869 (0x6d65 bytes) 000746 1789684508.462620 punch from 0x00270a to 0x010f86 (0xe87d bytes) 000747 1789684508.478657 punch from 0x00e78b to 0x014d92 (0x6608 bytes) 000748 1789684508.483619 write 0x022930 thru 0x023187 (0x858 bytes) 000750 1789684508.496116 MIRROR_RESYNC 9 mirrors 000751 1789684511.874723 mapwrite 0x031184 thru 0x037bca (0x6a47 bytes) 000752 1789684512.049618 write 0x0391d1 thru 0x03ffff (0x6e2f bytes) 000753 1789684512.051059 mapread 0x026648 thru 0x0334db (0xce94 bytes) 000754 1789684512.076671 punch from 0x03a584 to 0x03ffff (0x5a7c bytes) 000755 1789684512.103615 MIRROR_RESYNC 9 mirrors 000756 1789684515.073825 write 0x0198f6 thru 0x0298f4 (0xffff bytes) 000759 1789684515.149332 punch from 0x03f422 to 0x03ffff (0xbde bytes) 000760 1789684515.157337 read 0x00d15c thru 0x00e20f (0x10b4 bytes) 000763 1789684515.161713 write 0x02dffe thru 0x0317ca (0x37cd bytes) 000764 1789684515.168126 punch from 0x020ec6 to 0x02efe0 (0xe11b bytes) 000765 1789684515.176127 punch from 0x009def to 0x013f4f (0xa161 bytes) 000766 1789684515.179061 mapwrite 0x0135f2 thru 0x022206 (0xec15 bytes) 000767 1789684515.271789 mapwrite 0x003e47 thru 0x006dc8 (0x2f82 bytes) 000769 1789684515.367457 trunc from 0x040000 to 0x031afd (0xe502 bytes) 000770 1789684515.405523 punch from 0x02d6cf to 0x031afd (0x442f bytes) 000772 1789684515.413609 read 0x01a4b4 thru 0x01faba (0x5607 bytes) 000773 1789684515.424424 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31afe): Mirror 7: 0x13b5131d (stale) Mirror 8: 0x13b5131d (stale) Mirror 10: 0x13b5131d (stale) Mirror 11: 0x13b5131d (stale) Mirror 12: 0x13b5131d (stale) Mirror 13: 0x13b5131d (stale) Mirror 14: 0x13b5131d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31afe): Mirror 7: 0x13b5131d (stale) Mirror 8: 0x13b5131d (stale) Mirror 10: 0x13b5131d (stale) Mirror 11: 0x13b5131d (stale) Mirror 12: 0x13b5131d (stale) Mirror 13: 0x13b5131d (stale) Mirror 14: 0x13b5131d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31afe: skipped 000776 1789684516.003799 read 0x012d04 thru 0x014c96 (0x1f93 bytes) 000777 1789684516.010084 read 0x008ccb thru 0x01612a (0xd460 bytes) 000778 1789684516.019690 mapread 0x01a59e thru 0x027fa7 (0xda0a bytes) 000780 1789684516.021401 read 0x022afa thru 0x031afd (0xf004 bytes) 000782 1789684516.023177 punch from 0x00b903 to 0x018a3e (0xd13c bytes) 000784 1789684516.028574 mapread 0x00e4da thru 0x01e167 (0xfc8e bytes) 000785 1789684516.037371 read 0x03037a thru 0x031498 (0x111f bytes) 000786 1789684516.038060 punch from 0x011f6c to 0x01f151 (0xd1e6 bytes) 000788 1789684516.045768 mapread 0x022dd3 thru 0x02e259 (0xb487 bytes) 000790 1789684516.048461 write 0x03534f thru 0x03e1fc (0x8eae bytes) 000791 1789684516.050276 mapwrite 0x02d7fe thru 0x030d3e (0x3541 bytes) 000792 1789684516.101695 read 0x01ae56 thru 0x01fd72 (0x4f1d bytes) 000794 1789684516.110611 write 0x00c002 thru 0x018fc7 (0xcfc6 bytes) 000796 1789684516.112312 punch from 0x0122d0 to 0x01e378 (0xc0a9 bytes) 000797 1789684516.128330 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: 0xeb83d01d (stale) Mirror 8: 0xeb83d01d (stale) Mirror 10: 0xeb83d01d (stale) Mirror 11: 0xeb83d01d (stale) Mirror 12: 0xeb83d01d (stale) Mirror 13: 0xeb83d01d (stale) Mirror 14: 0xeb83d01d (stale) CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 1: 0x1a820a0a Mirror 15: 0x1a820a0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1fd: skipped 000798 1789684517.080294 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: 0xeb83d01d (stale) Mirror 8: 0xeb83d01d (stale) Mirror 10: 0xeb83d01d (stale) Mirror 11: 0xeb83d01d (stale) Mirror 12: 0xeb83d01d (stale) Mirror 14: 0xeb83d01d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: 0xeb83d01d (stale) Mirror 8: 0xeb83d01d (stale) Mirror 10: 0xeb83d01d (stale) Mirror 11: 0xeb83d01d (stale) Mirror 12: 0xeb83d01d (stale) Mirror 14: 0xeb83d01d (stale) CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 1: 0x1a820a0a Mirror 15: 0x1a820a0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1fd: skipped 000799 1789684517.817203 mapwrite 0x007365 thru 0x00b24a (0x3ee6 bytes) 000800 1789684517.997715 mapwrite 0x008b8c thru 0x00cfe1 (0x4456 bytes) 000801 1789684518.036307 punch from 0x031b69 to 0x0342e2 (0x277a bytes) 000805 1789684518.041642 punch from 0x020ddb to 0x021d5c (0xf82 bytes) 000808 1789684518.046976 trunc from 0x03e1fd to 0x00e838 (0x2f9c4 bytes) 000812 1789684518.108344 mapwrite 0x00ec79 thru 0x01d675 (0xe9fd bytes) 000814 1789684518.202287 trunc from 0x01d676 to 0x00d829 (0xfe4c bytes) 000816 1789684518.218883 mapread 0x0054c9 thru 0x00d829 (0x8361 bytes) 000819 1789684518.221417 punch from 0x004a05 to 0x00b126 (0x6722 bytes) 000822 1789684518.226842 mapwrite 0x02017a thru 0x021e10 (0x1c97 bytes) 000824 1789684518.281508 read 0x0054a9 thru 0x008a02 (0x355a bytes) 000826 1789684518.293535 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x21e11): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21e11): Mirror 7: 0x508a52e9 (stale) Mirror 8: 0x508a52e9 (stale) Mirror 10: 0x508a52e9 (stale) Mirror 11: 0x508a52e9 (stale) Mirror 12: 0x508a52e9 (stale) Mirror 14: 0x508a52e9 (stale) Mirror 15: 0x3d044d38 (stale) CRC-32 checksum value for chunk [0, 0x21e11): Mirror 1: 0xb5d308a9 Mirror 16: 0xb5d308a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21e11: skipped 000827 1789684519.059684 punch from 0x00cb6d to 0x010681 (0x3b15 bytes) 000828 1789684519.121216 trunc from 0x021e11 to 0x03052b (0xe71b bytes) 000829 1789684519.167194 MIRROR_RESYNC 9 mirrors 000830 1789684521.193893 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) 000831 1789684521.583037 trunc from 0x03052c to 0x027cfa (0x8831 bytes) 000832 1789684521.684851 punch from 0x01c14e to 0x027cfa (0xbbad bytes) 000835 1789684521.693374 write 0x01ad88 thru 0x0248f4 (0x9b6d bytes) 000836 1789684521.713835 trunc from 0x027cfb to 0x01fb06 (0x81f4 bytes) 000838 1789684521.789934 read 0x00fadb thru 0x01d989 (0xdeaf bytes) 000841 1789684521.806023 read 0x0001bd thru 0x00fe03 (0xfc47 bytes) 000842 1789684521.822888 read 0x00c998 thru 0x01a449 (0xdab2 bytes) 000843 1789684521.823962 trunc from 0x01fb07 to 0x039b54 (0x1a04e bytes) 000844 1789684521.856357 trunc from 0x039b55 to 0x010c5a (0x28efa bytes) 000845 1789684521.885207 read 0x00dc40 thru 0x010c5a (0x301b bytes) 000846 1789684521.886587 trunc from 0x010c5b to 0x016232 (0x55d8 bytes) 000847 1789684521.916404 mapwrite 0x01b19e thru 0x0287f6 (0xd659 bytes) 000849 1789684522.063787 trunc from 0x0287f7 to 0x01a3a0 (0xe456 bytes) 000852 1789684522.078259 trunc from 0x01a3a1 to 0x01db4e (0x37ae bytes) 000853 1789684522.098670 mapwrite 0x0335ce thru 0x03607a (0x2aad bytes) 000854 1789684522.175502 read 0x02840a thru 0x034af0 (0xc6e7 bytes) 000855 1789684522.184242 punch from 0x0328fc to 0x03607a (0x377f bytes) 000857 1789684522.189116 trunc from 0x03607b to 0x0102dc (0x25d9e bytes) 000858 1789684522.203505 trunc from 0x0102dd to 0x01d29a (0xcfbe bytes) 000859 1789684522.220645 punch from 0x0148a6 to 0x01d29a (0x89f5 bytes) 000862 1789684522.229680 MIRROR_RESYNC 8 mirrors 000863 1789684523.683977 punch from 0x00b31f to 0x018e38 (0xdb1a bytes) 000867 1789684523.759462 mapwrite 0x01b3b6 thru 0x0228b2 (0x74fd bytes) 000868 1789684523.855623 mapwrite 0x014933 thru 0x0241f7 (0xf8c5 bytes) 000869 1789684523.922620 mapread 0x0105ed thru 0x0126d4 (0x20e8 bytes) 000870 1789684523.924668 read 0x007896 thru 0x00949f (0x1c0a bytes) 000871 1789684523.931176 write 0x0344fa thru 0x03b5f2 (0x70f9 bytes) 000872 1789684523.936653 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b5f3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5f3): Mirror 7: 0x7c4da2f4 (stale) Mirror 8: 0x7c4da2f4 (stale) Mirror 10: 0x7c4da2f4 (stale) Mirror 12: 0x7c4da2f4 (stale) Mirror 14: 0x7c4da2f4 (stale) Mirror 15: 0x7c4da2f4 (stale) Mirror 16: 0x7c4da2f4 (stale) CRC-32 checksum value for chunk [0, 0x3b5f3): Mirror 1: 0x6a11d779 Mirror 17: 0x6a11d779 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5f3: skipped 000873 1789684525.185081 read 0x01c79b thru 0x025eb6 (0x971c bytes) 000874 1789684525.204807 write 0x0302a1 thru 0x0362ec (0x604c bytes) 000875 1789684525.297352 punch from 0x01fa00 to 0x02e661 (0xec62 bytes) 000876 1789684525.312332 trunc from 0x03b5f3 to 0x03f99e (0x43ac bytes) 000878 1789684525.345290 trunc from 0x03f99f to 0x0381d0 (0x77ce bytes) 000879 1789684525.363801 mapread 0x0033f3 thru 0x00fcd6 (0xc8e4 bytes) 000880 1789684525.382289 read 0x018260 thru 0x022631 (0xa3d2 bytes) 000882 1789684525.394591 punch from 0x02253d to 0x023644 (0x1108 bytes) 000883 1789684525.399576 write 0x02c711 thru 0x03a4a0 (0xdd90 bytes) 000884 1789684525.400906 trunc from 0x03a4a1 to 0x004b9e (0x35902 bytes) 000885 1789684525.433161 MIRROR_RESYNC 9 mirrors 000886 1789684526.680265 punch from 0x001bc5 to 0x004b9e (0x2fda bytes) 000887 1789684526.755764 trunc from 0x004b9f to 0x0274cf (0x22931 bytes) 000888 1789684526.776253 mapread 0x013f2a thru 0x01dee3 (0x9fba bytes) 000892 1789684526.797234 mapread 0x012312 thru 0x019498 (0x7187 bytes) 000894 1789684526.799652 write 0x01753f thru 0x019ab7 (0x2579 bytes) 000895 1789684526.801661 punch from 0x02401a to 0x02721e (0x3205 bytes) 000896 1789684526.811984 mapread 0x01cf67 thru 0x0274cf (0xa569 bytes) 000897 1789684526.825907 mapread 0x002e89 thru 0x0119a6 (0xeb1e bytes) 000898 1789684526.836170 mapwrite 0x00531e thru 0x00f335 (0xa018 bytes) 000900 1789684526.906321 MIRROR_RESYNC 9 mirrors 000902 1789684529.332658 read 0x012a01 thru 0x0203cc (0xd9cc bytes) 000903 1789684529.345617 MIRROR_EXTEND to 10 mirrors 000904 1789684530.233483 MIRROR_EXTEND to 11 mirrors 000905 1789684530.932192 punch from 0x003b8b to 0x00b96c (0x7de2 bytes) 000906 1789684531.014330 read 0x024d40 thru 0x0274cf (0x2790 bytes) 000907 1789684531.036612 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x274d0): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x274d0): Mirror 7: 0xb376d355 (stale) Mirror 8: 0xb376d355 (stale) Mirror 10: 0xb376d355 (stale) Mirror 12: 0xb376d355 (stale) Mirror 14: 0xb376d355 (stale) Mirror 15: 0xb376d355 (stale) Mirror 16: 0xb376d355 (stale) Mirror 17: 0xb376d355 (stale) Mirror 18: 0xb376d355 (stale) Mirror 19: 0xb376d355 (stale) CRC-32 checksum value for chunk [0, 0x274d0): Mirror 1: 0xc9bb8379 Mirror 20: 0xc9bb8379 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x274d0: skipped 000909 1789684532.176551 read 0x006735 thru 0x00a3fc (0x3cc8 bytes) 000911 1789684532.195496 read 0x00efaa thru 0x01d39d (0xe3f4 bytes) 000912 1789684532.214693 trunc from 0x0274d0 to 0x03c236 (0x14d67 bytes) 000917 1789684532.296391 mapwrite 0x0303e0 thru 0x03216e (0x1d8f bytes) 000918 1789684532.346781 mapread 0x008bdf thru 0x0161e6 (0xd608 bytes) 000919 1789684532.406587 mapwrite 0x01bc8a thru 0x024b57 (0x8ece bytes) 000920 1789684532.493253 mapread 0x03ae2f thru 0x03c236 (0x1408 bytes) 000921 1789684532.498177 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c237): Mirror 7: 0xa1f21b8d (stale) Mirror 8: 0xa1f21b8d (stale) Mirror 10: 0xa1f21b8d (stale) Mirror 12: 0xa1f21b8d (stale) Mirror 14: 0xa1f21b8d (stale) Mirror 15: 0xa1f21b8d (stale) Mirror 17: 0xa1f21b8d (stale) Mirror 18: 0xa1f21b8d (stale) Mirror 19: 0xa1f21b8d (stale) Mirror 20: 0xe342f246 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c237): Mirror 7: 0xa1f21b8d (stale) Mirror 8: 0xa1f21b8d (stale) Mirror 10: 0xa1f21b8d (stale) Mirror 12: 0xa1f21b8d (stale) Mirror 14: 0xa1f21b8d (stale) Mirror 15: 0xa1f21b8d (stale) Mirror 17: 0xa1f21b8d (stale) Mirror 18: 0xa1f21b8d (stale) Mirror 19: 0xa1f21b8d (stale) Mirror 20: 0xe342f246 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c237: skipped 000922 1789684532.954095 write 0x000337 thru 0x000aec (0x7b6 bytes) 000923 1789684532.959151 write 0x014e32 thru 0x01ad62 (0x5f31 bytes) 000924 1789684532.966953 trunc from 0x03c237 to 0x003fd6 (0x38260 bytes) 000927 1789684532.988116 trunc from 0x003fd7 to 0x01b7e6 (0x17810 bytes) 000928 1789684533.008254 mapwrite 0x022b9f thru 0x02d4ea (0xa94c bytes) 000929 1789684533.081676 write 0x0327eb thru 0x03e4bd (0xbcd3 bytes) 000930 1789684533.088613 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e4be): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4be): Mirror 7: 0xbfb92dd (stale) Mirror 8: 0xbfb92dd (stale) Mirror 10: 0xbfb92dd (stale) Mirror 12: 0xbfb92dd (stale) Mirror 14: 0xbfb92dd (stale) Mirror 15: 0xbfb92dd (stale) Mirror 17: 0xbfb92dd (stale) Mirror 18: 0xbfb92dd (stale) Mirror 19: 0xbfb92dd (stale) Mirror 20: 0xbe31337b (stale) CRC-32 checksum value for chunk [0, 0x3e4be): Mirror 1: 0x308e1b3c Mirror 21: 0x308e1b3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4be: skipped 000931 1789684534.047116 write 0x0307f3 thru 0x030f13 (0x721 bytes) 000933 1789684534.125774 mapread 0x0145f7 thru 0x01884a (0x4254 bytes) 000934 1789684534.141003 read 0x021eda thru 0x028a75 (0x6b9c bytes) 000936 1789684534.148826 trunc from 0x03e4be to 0x03dec8 (0x5f5 bytes) 000937 1789684534.186069 mapwrite 0x00d2de thru 0x017adb (0xa7fe bytes) 000938 1789684534.251235 write 0x0125cf thru 0x02174e (0xf180 bytes) 000940 1789684534.261819 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 7: 0x8db5c46c (stale) Mirror 8: 0x8db5c46c (stale) Mirror 10: 0x8db5c46c (stale) Mirror 12: 0x8db5c46c (stale) Mirror 14: 0x8db5c46c (stale) Mirror 15: 0x8db5c46c (stale) Mirror 17: 0x8db5c46c (stale) Mirror 18: 0x8db5c46c (stale) Mirror 19: 0x8db5c46c (stale) Mirror 20: 0xc943bf2a (stale) Mirror 21: 0xb37cd1cc (stale) CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 1: 0x5f3d29e6 Mirror 22: 0x5f3d29e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dec9: skipped 000941 1789684535.274607 trunc from 0x03dec9 to 0x01d56e (0x2095a bytes) 000942 1789684535.384177 punch from 0x0047f2 to 0x0109b9 (0xc1c8 bytes) 000943 1789684535.389153 punch from 0x0185f3 to 0x0194f7 (0xf05 bytes) 000944 1789684535.394103 punch from 0x00e6fb to 0x01d56e (0xee74 bytes) 000945 1789684535.399412 mapwrite 0x011398 thru 0x01fb00 (0xe769 bytes) 000946 1789684535.502210 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fb01): Mirror 7: 0x7afab222 (stale) Mirror 8: 0x7afab222 (stale) Mirror 10: 0x7afab222 (stale) Mirror 12: 0x7afab222 (stale) Mirror 14: 0x7afab222 (stale) Mirror 15: 0x7afab222 (stale) Mirror 17: 0x7afab222 (stale) Mirror 19: 0x7afab222 (stale) Mirror 20: 0x57afd68b (stale) Mirror 21: 0xee9a7e3a (stale) Mirror 22: 0x97ad7433 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb01): Mirror 7: 0x7afab222 (stale) Mirror 8: 0x7afab222 (stale) Mirror 10: 0x7afab222 (stale) Mirror 12: 0x7afab222 (stale) Mirror 14: 0x7afab222 (stale) Mirror 15: 0x7afab222 (stale) Mirror 17: 0x7afab222 (stale) Mirror 19: 0x7afab222 (stale) Mirror 20: 0x57afd68b (stale) Mirror 21: 0xee9a7e3a (stale) Mirror 22: 0x97ad7433 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb01: skipped 000947 1789684536.025315 read 0x00956b thru 0x0182bb (0xed51 bytes) 000948 1789684536.034079 mapwrite 0x0065d5 thru 0x0067a3 (0x1cf bytes) 000950 1789684536.076647 punch from 0x015742 to 0x018b53 (0x3412 bytes) 000952 1789684536.082213 mapwrite 0x03dd27 thru 0x03ffff (0x22d9 bytes) 000953 1789684536.147662 mapwrite 0x038ca9 thru 0x039538 (0x890 bytes) 000954 1789684536.187196 write 0x013cb2 thru 0x022d38 (0xf087 bytes) 000957 1789684536.193940 write 0x0163e7 thru 0x01eba2 (0x87bc bytes) 000959 1789684536.196529 trunc from 0x040000 to 0x0088a2 (0x3775d bytes) 000960 1789684536.211453 write 0x022fab thru 0x02c56e (0x95c4 bytes) 000961 1789684536.213431 punch from 0x021c52 to 0x02c51e (0xa8cd bytes) 000962 1789684536.225164 mapwrite 0x00509a thru 0x013edb (0xee42 bytes) 000963 1789684536.278390 write 0x00e76d thru 0x0118a9 (0x313d bytes) 000964 1789684536.279564 punch from 0x01491e to 0x022682 (0xdd65 bytes) 000965 1789684536.288717 mapread 0x00b7ca thru 0x019dad (0xe5e4 bytes) 000966 1789684536.296921 punch from 0x02b963 to 0x02c56e (0xc0c bytes) 000967 1789684536.301728 mapread 0x007161 thru 0x014031 (0xced1 bytes) 000968 1789684536.302790 read 0x009b1c thru 0x00eea5 (0x538a bytes) 000970 1789684536.304518 mapwrite 0x033e0c thru 0x03615a (0x234f bytes) 000971 1789684536.364483 mapwrite 0x03e47a thru 0x03ffff (0x1b86 bytes) 000972 1789684536.453378 mapread 0x029a6b thru 0x038aa7 (0xf03d bytes) 000975 1789684536.459001 punch from 0x00889a to 0x016c72 (0xe3d9 bytes) 000976 1789684536.464761 read 0x030853 thru 0x036f25 (0x66d3 bytes) 000978 1789684536.466386 read 0x02656a thru 0x02661f (0xb6 bytes) 000979 1789684536.468891 write 0x03128e thru 0x03f02f (0xdda2 bytes) 000980 1789684536.474843 mapwrite 0x03c263 thru 0x03cf84 (0xd22 bytes) 000981 1789684536.529619 write 0x01fcd1 thru 0x0250a7 (0x53d7 bytes) 000983 1789684536.541278 read 0x00c5bc thru 0x00cfcc (0xa11 bytes) 000984 1789684536.564615 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9b5c1820 (stale) Mirror 8: 0x9b5c1820 (stale) Mirror 12: 0x9b5c1820 (stale) Mirror 14: 0x9b5c1820 (stale) Mirror 15: 0x9b5c1820 (stale) Mirror 17: 0x9b5c1820 (stale) Mirror 19: 0x9b5c1820 (stale) Mirror 20: 0x539495de (stale) Mirror 21: 0x4ec9460b (stale) Mirror 22: 0x696de34e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9b5c1820 (stale) Mirror 8: 0x9b5c1820 (stale) Mirror 12: 0x9b5c1820 (stale) Mirror 14: 0x9b5c1820 (stale) Mirror 15: 0x9b5c1820 (stale) Mirror 17: 0x9b5c1820 (stale) Mirror 19: 0x9b5c1820 (stale) Mirror 20: 0x539495de (stale) Mirror 21: 0x4ec9460b (stale) Mirror 22: 0x696de34e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000986 1789684537.135481 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9b5c1820 (stale) Mirror 8: 0x9b5c1820 (stale) Mirror 14: 0x9b5c1820 (stale) Mirror 15: 0x9b5c1820 (stale) Mirror 17: 0x9b5c1820 (stale) Mirror 19: 0x9b5c1820 (stale) Mirror 20: 0x539495de (stale) Mirror 21: 0x4ec9460b (stale) Mirror 22: 0x696de34e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9b5c1820 (stale) Mirror 8: 0x9b5c1820 (stale) Mirror 14: 0x9b5c1820 (stale) Mirror 15: 0x9b5c1820 (stale) Mirror 17: 0x9b5c1820 (stale) Mirror 19: 0x9b5c1820 (stale) Mirror 20: 0x539495de (stale) Mirror 21: 0x4ec9460b (stale) Mirror 22: 0x696de34e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1789684537.524307 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9b5c1820 (stale) Mirror 8: 0x9b5c1820 (stale) Mirror 14: 0x9b5c1820 (stale) Mirror 15: 0x9b5c1820 (stale) Mirror 17: 0x9b5c1820 (stale) Mirror 19: 0x9b5c1820 (stale) Mirror 20: 0x539495de (stale) Mirror 21: 0x4ec9460b (stale) Mirror 22: 0x696de34e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x400006d4 Mirror 23: 0x400006d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1789684538.617611 trunc from 0x040000 to 0x02d3cc (0x12c33 bytes) 000990 1789684538.703789 read 0x001a41 thru 0x004b06 (0x30c6 bytes) 000991 1789684538.710188 punch from 0x024e8c to 0x02b8aa (0x6a1f bytes) 000992 1789684538.714584 mapread 0x00b3cb thru 0x0165bf (0xb1f5 bytes) 000993 1789684538.736461 trunc from 0x02d3cd to 0x03d35c (0xff90 bytes) 000994 1789684538.758539 MIRROR_RESYNC 11 mirrors 000995 1789684541.075681 punch from 0x00b8f4 to 0x00e152 (0x285f bytes) 000996 1789684541.222195 read 0x034e04 thru 0x03767d (0x287a bytes) 000997 1789684541.231809 mapwrite 0x022976 thru 0x030c89 (0xe314 bytes) 000999 1789684541.351044 read 0x010cad thru 0x018dca (0x811e bytes) 001000 1789684541.378295 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3d35d): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d35d): Mirror 7: 0x342d70ab (stale) Mirror 8: 0x342d70ab (stale) Mirror 14: 0x342d70ab (stale) Mirror 15: 0x342d70ab (stale) Mirror 17: 0x342d70ab (stale) Mirror 19: 0x342d70ab (stale) Mirror 20: 0x342d70ab (stale) Mirror 21: 0x342d70ab (stale) Mirror 22: 0x342d70ab (stale) Mirror 23: 0x342d70ab (stale) CRC-32 checksum value for chunk [0, 0x3d35d): Mirror 1: 0x6ab9eb25 Mirror 24: 0x6ab9eb25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d35d: skipped All operations completed A-OK! PASS 70a (101s) == sanity-flr test 71: check mirror extend parent fid ==== 18:35:47 (1789684547) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 18:35:51 (1789684551) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252019 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185598 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 18:36:00 (1789684560) Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre2 Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6674432 .. Extending file size to 6315552 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8360064 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7729696 .. Extending file size to 8514496 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8527200 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9404416 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8469792 .. Extending file size to 8859072 .. Extending file size to 8919520 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57433 57434 57436 57437 57438 failed 192.168.201.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg155-client.virtnet /mnt/lustre2 (opts:) 192.168.201.155@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg155-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:37:16 (1789684636) Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre2 Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7787296 .. Extending file size to 1997344 .. done Extending file size to 3582432 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8985056 .. failed Extending file size to 8689760 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9318432 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8807040 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8911744 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59087 59088 59089 59090 59094 done 192.168.201.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg155-client.virtnet /mnt/lustre2 (opts:) 192.168.201.155@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg155-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:38:33 (1789684713) Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103474 s, 20.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.373548 s, 5.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xed56b48d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg155-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:38:49 (1789684729) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114647 s, 18.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 18:38:56 (1789684736) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:39:02 (1789684742) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.185394 s, 11.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:39:09 (1789684749) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220274 s, 9.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:39:17 (1789684757) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177988 s, 11.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:39:48 (1789684788) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg155-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051863 s, 20.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:40:43 (1789684843) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg155-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245562 s, 8.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x336:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:41:41 (1789684901) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg155-server mds-ost sync done. Creating new pool test_204f oleg155-server: Pool lustre.test_204f created Adding targets to pool oleg155-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg155-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193113 s, 10.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x319:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg155-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg155-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg155-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:42:51 (1789684971) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:42:56 (1789684976) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:43:01 (1789684981) oleg155-server: Pool lustre.test_206 created Adding targets to pool oleg155-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg155-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg155-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg155-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg155-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:43:22 (1789685002) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462979 s, 22.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808498 s, 13.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1a766d6d Mirror 3: 0x1a766d6d PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:43:29 (1789685009) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:43:31 (1789685011) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:43:34 (1789685014) Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0661571 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x720000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x850000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x850000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x850000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x14f0000 Read this (4 bytes): BBBB 192.168.201.155@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg155-client.virtnet /mnt/lustre2 (opts:) PASS 209a (27s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:44:01 (1789685041) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0316373 s, 259 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:44:06 (1789685046) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458416 s, 22.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x240 lmm_fid: [0x200000404:0x240:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x240 lmm_fid: [0x200000404:0x240:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x323:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834496 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 18:44:08.000000000 -0400 Modify: 2026-09-17 18:44:08.000000000 -0400 Change: 2026-09-17 18:44:09.000000000 -0400 Birth: 2026-09-17 18:44:09.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:44:12 (1789685052) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:44:15 (1789685055) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.479099 s, 85.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0470765 s, 87.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 17 18:44 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x242:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 17 18:44 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:44:21 (1789685061) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:44:27 (1789685067) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1742 sec =========== 18:44:35 (1789685075) === sanity-flr: start cleanup 18:44:36 (1789685076) === === sanity-flr: finish cleanup 18:44:41 (1789685081) ===