-----============= acceptance-small: sanity-flr ============----- Fri Aug 7 07:10:44 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg654-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:11:03 (1786101063) === oleg654-client.virtnet: executing check_config_client /mnt/lustre oleg654-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg654-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89b905974800.idle_timeout=debug osc.lustre-OST0001-osc-ffff89b905974800.idle_timeout=debug disable quota as required oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:11:41 (1786101101) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 07:11:43 (1786101103) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:11:57 (1786101117) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:11:59 (1786101119) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:12:02 (1786101122) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:12:23 (1786101143) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:12:26 (1786101146) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:12:29 (1786101149) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89b905974800.stats=clear osc.lustre-OST0001-osc-ffff89b905974800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:12:36 (1786101156) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:12:44 (1786101164) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:12:50 (1786101170) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:12:53 (1786101173) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:12:59 (1786101179) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 07:13:07 (1786101187) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:13:14 (1786101194) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:13:21 (1786101201) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:13:28 (1786101208) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:13:37 (1786101217) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:13:43 (1786101223) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184507 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517294 s, 20.3 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:13:49 (1786101229) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.280594 s, 29.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:13:57 (1786101237) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:14:07 (1786101247) Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg654-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (67s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:15:15 (1786101315) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.633617 s, 14.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:15:56 (1786101356) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg654-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:17:10 (1786101430) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.7449 s, 19.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 7 07:17 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.928s user 0m0.001s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89b905974800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.762s user 0m0.000s sys 0m0.167s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg654-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg654-server oleg654-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.342s user 0m0.001s sys 0m0.154s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg654-server: oleg654-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg654-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg654-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89b905974800.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:18:34 (1786101514) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:18:37 (1786101517) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:18:40 (1786101520) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 07:18:43 (1786101523) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 07:18:49 (1786101529) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.347877 s, 36.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:18:59 (1786101539) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376301 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372878 s, 28.1 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:19:05 (1786101545) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:19:12 (1786101552) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115684 s, 885 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00167189 s, 612 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00183071 s, 559 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105416 s, 971 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 07:19:20 (1786101560) create test files with size 15748320 .. 27f70f2e39b891b5e086509968f615bb - cb3553b790dfe314670487c305aa71cf - c2b7900996ad42734984bff74fc52561 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg654-client.virtnet /mnt/lustre (opts:) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 07:19:40 (1786101580) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:19:51 (1786101591) PASS 39 (5s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:19:56 (1786101596) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730558 s, 14.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058309 s, 18.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 07:20:03 (1786101603) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0742277 s, 28.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.324527 s, 12.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116569 s, 18.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 07:20:16 (1786101616) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 07:20:18 (1786101618) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:20:21 (1786101621) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228335 s, 45.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 07:20:27 (1786101627) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.389363 s, 8.1 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:20:35 (1786101635) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:20:44 (1786101644) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31083 s, 33.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg654-client.virtnet /mnt/lustre (opts:) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:20:55 (1786101655) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff89b914240800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.291138 s, 36.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg654-client.virtnet /mnt/lustre (opts:) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff89b925768800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:21:09 (1786101669) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:21:41 (1786101701) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:22:13 (1786101733) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 07:22:16 (1786101736) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.223417 s, 14.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.11528 s, 27.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbf5496c (nosync,stale) Mirror 2: 0xbf5496c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4933ef8f1b0970ea8c6585b87ac166ba - mirror 1 checksum: dd798393458d64aab3fe902bee9343f9 - mirror 2 checksum: 4933ef8f1b0970ea8c6585b87ac166ba - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-07T07:22:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-07T07:22:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4933ef8f1b0970ea8c6585b87ac166ba - mirror 1 checksum: dd798393458d64aab3fe902bee9343f9 - mirror 2 checksum: dd798393458d64aab3fe902bee9343f9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-07T07:22:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-07T07:22:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:22:24 (1786101744) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241978 s, 43.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:22:31 (1786101751) Failing mds1 on oleg654-server Stopping /mnt/lustre-mds1 (opts:) on oleg654-server 07:22:43 (1786101763) shut down facet: mds1 facet_host: oleg654-server facet_failover_host: oleg654-server Failover mds1 to oleg654-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 07:23:16 (1786101796) targets are mounted 07:23:16 (1786101796) facet_failover done oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275509 s, 38.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:23:32 (1786101812) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0249912 s, 819 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0104225 s, 2.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:23:41 (1786101821) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0394972 s, 519 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00999182 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:23:49 (1786101829) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00968543 s, 1.7 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:23:55 (1786101835) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0518417 s, 395 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:24:02 (1786101842) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00530559 s, 772 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0191418 s, 214 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:24:08 (1786101848) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:24:26 (1786101866) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:24:53 (1786101893) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:25:16 (1786101916) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:25:38 (1786101938) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 07:25:43 (1786101943) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57517 mirror split pid 57518 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57517 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57518 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 07:26:49 (1786102009) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2011 skipping zero size read 000003 1786102011.798618 write 0x02b5ad thru 0x036665 (0xb0b9 bytes) 000005 1786102011.807026 read 0x015788 thru 0x021f10 (0xc789 bytes) truncating to largest ever: 0x3baac 000006 1786102011.813696 trunc from 0x036666 to 0x03baab (0x5446 bytes) 000007 1786102011.829213 trunc from 0x03baac to 0x00324e (0x3885d bytes) 000010 1786102011.852994 MIRROR_EXTEND to 2 mirrors 000011 1786102012.396528 MIRROR_RESYNC 2 mirrors 000012 1786102012.783981 fallocate 0x007c6c thru 0x00bcf3 (0x4088 bytes) 000013 1786102012.808652 trunc from 0x003f44 to 0x00bcf3 (0x7db0 bytes) 000014 1786102012.819797 punch from 0x009e80 to 0x00bcf3 (0x1e74 bytes) 000015 1786102012.827555 mapwrite 0x03eba0 thru 0x03ffff (0x1460 bytes) 000017 1786102012.903415 write 0x03fa64 thru 0x03ffff (0x59c bytes) 000018 1786102012.904967 trunc from 0x040000 to 0x00c409 (0x33bf6 bytes) 000019 1786102012.930448 punch from 0x0069e6 to 0x00c409 (0x5a24 bytes) 000020 1786102012.941045 mapread 0x00ab8c thru 0x00c409 (0x187e bytes) 000021 1786102012.958614 mapread 0x00bc3d thru 0x00c409 (0x7cd bytes) 000022 1786102012.959629 read 0x00422d thru 0x00c409 (0x81dd bytes) 000023 1786102012.967718 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000026 1786102013.143113 fallocate 0x01181f thru 0x01e865 (0xd047 bytes) 000029 1786102013.151775 read 0x00bd03 thru 0x00c409 (0x707 bytes) 000030 1786102013.157043 trunc from 0x0011d7 to 0x00ad4e (0x9b78 bytes) 000031 1786102013.161509 mapwrite 0x030106 thru 0x031388 (0x1283 bytes) 000032 1786102013.212210 punch from 0x01b272 to 0x029a06 (0xe795 bytes) 000034 1786102013.221718 read 0x01765b thru 0x0200f9 (0x8a9f bytes) 000035 1786102013.233739 trunc from 0x031389 to 0x0254c7 (0xbec1 bytes) 000036 1786102013.246494 mapread 0x0006a0 thru 0x0103c4 (0xfd25 bytes) 000037 1786102013.248071 punch from 0x01f4ca to 0x0254c7 (0x5ffe bytes) 000038 1786102013.252773 mapread 0x017de4 thru 0x024ff5 (0xd212 bytes) 000039 1786102013.275685 mapread 0x012457 thru 0x020d61 (0xe90b bytes) 000040 1786102013.277370 mapwrite 0x02a856 thru 0x03a538 (0xfce3 bytes) fallocating to largest ever: 0x40000 000041 1786102013.349577 fallocate 0x03644a thru 0x03ffff (0x9bb6 bytes) 000042 1786102013.353352 trunc from 0x014cb3 to 0x0182be (0x360c bytes) 000043 1786102013.357679 mapwrite 0x005301 thru 0x00e187 (0x8e87 bytes) 000044 1786102013.373069 mapread 0x002d05 thru 0x00e188 (0xb484 bytes) 000045 1786102013.374581 trunc from 0x040000 to 0x0017ac (0x3e853 bytes) 000047 1786102013.389524 mapwrite 0x027836 thru 0x034c14 (0xd3df bytes) 000048 1786102013.499869 punch from 0x024d48 to 0x02a3f1 (0x56aa bytes) 000049 1786102013.504787 trunc from 0x034c15 to 0x002edb (0x31d39 bytes) 000050 1786102013.517640 trunc from 0x001e55 to 0x002edb (0x1087 bytes) 000052 1786102013.524114 trunc from 0x002edc to 0x02da1e (0x2ab43 bytes) 000053 1786102013.537536 fallocate 0x026b28 thru 0x03293a (0xbe13 bytes) 000054 1786102013.541541 mapwrite 0x007acb thru 0x00e241 (0x6777 bytes) 000057 1786102013.602523 fallocate 0x00f5e7 thru 0x01949f (0x9eb9 bytes) 000058 1786102013.608578 trunc from 0x02b520 to 0x02da1e (0x24ff bytes) 000060 1786102013.614851 read 0x00c8be thru 0x00e7fc (0x1f3f bytes) 000061 1786102013.616346 fallocate 0x02d445 thru 0x02f799 (0x2355 bytes) 000062 1786102013.621776 write 0x03378f thru 0x03d51f (0x9d91 bytes) 000063 1786102013.624650 trunc from 0x038329 to 0x03d51f (0x51f7 bytes) 000064 1786102013.638441 trunc from 0x03d520 to 0x00b0a6 (0x32479 bytes) 000065 1786102013.655854 write 0x02593b thru 0x031447 (0xbb0d bytes) 000067 1786102013.658887 read 0x00b55c thru 0x01a57d (0xf022 bytes) 000068 1786102013.667160 mapread 0x027315 thru 0x031447 (0xa133 bytes) 000069 1786102013.668652 write 0x006166 thru 0x00d5e7 (0x7482 bytes) 000070 1786102013.671160 write 0x01359c thru 0x0212f3 (0xdd58 bytes) 000071 1786102013.673457 fallocate 0x037681 thru 0x03a222 (0x2ba2 bytes) 000072 1786102013.680786 punch from 0x020d90 to 0x02df31 (0xd1a2 bytes) 000073 1786102013.700072 punch from 0x01d2d4 to 0x029dce (0xcafb bytes) 000075 1786102013.706012 fallocate 0x02292c thru 0x028aca (0x619f bytes) 000076 1786102013.712133 trunc from 0x02ce84 to 0x0349a2 (0x7b1f bytes) 000078 1786102013.718566 write 0x009e84 thru 0x00bd67 (0x1ee4 bytes) 000079 1786102013.720362 trunc from 0x03a223 to 0x0155fb (0x24c27 bytes) 000081 1786102013.735654 trunc from 0x0155fc to 0x000a2b (0x14bd0 bytes) 000082 1786102013.747901 read 0x00039a thru 0x000a2b (0x692 bytes) 000083 1786102013.748651 trunc from 0x0006c8 to 0x000a2b (0x364 bytes) 000084 1786102013.753603 read 0x00035c thru 0x000a2b (0x6d0 bytes) 000085 1786102013.758851 punch from 0x000924 to 0x000a2b (0x108 bytes) 000086 1786102013.763107 write 0x0189f5 thru 0x022884 (0x9e90 bytes) 000087 1786102013.768281 trunc from 0x00080f to 0x001da9 (0x159b bytes) 000088 1786102013.777592 mapread 0x013b2b thru 0x01961a (0x5af0 bytes) 000090 1786102013.792616 fallocate 0x033c9b thru 0x03ffff (0xc365 bytes) 000091 1786102013.798482 trunc from 0x040000 to 0x00e26f (0x31d90 bytes) 000092 1786102013.827566 write 0x009d61 thru 0x0172ce (0xd56e bytes) 000093 1786102013.837713 mapwrite 0x0236a9 thru 0x02c5ae (0x8f06 bytes) 000094 1786102013.888484 trunc from 0x01d1e1 to 0x01f2f4 (0x2114 bytes) 000095 1786102013.893244 read 0x00d766 thru 0x00e348 (0xbe3 bytes) 000096 1786102013.893954 punch from 0x00bdfe to 0x014d93 (0x8f96 bytes) 000097 1786102013.899967 read 0x0142e3 thru 0x017661 (0x337f bytes) 000098 1786102013.910107 read 0x021084 thru 0x02c5ae (0xb52b bytes) 000099 1786102013.911805 mapread 0x010bcf thru 0x01116d (0x59f bytes) 000100 1786102013.915370 trunc from 0x02c5af to 0x033be1 (0x7633 bytes) 000101 1786102013.935527 trunc from 0x033be2 to 0x03cd1d (0x913c bytes) 000102 1786102013.955946 trunc from 0x03cd1e to 0x01c092 (0x20c8b bytes) 000105 1786102013.985284 mapwrite 0x0214bd thru 0x02d0aa (0xbbee bytes) 000106 1786102014.070576 write 0x005cac thru 0x00dd9f (0x80f4 bytes) 000107 1786102014.074307 punch from 0x01b2d6 to 0x02073a (0x5465 bytes) 000108 1786102014.084130 trunc from 0x0153f4 to 0x019d2a (0x4937 bytes) 000109 1786102014.089646 trunc from 0x021f23 to 0x0293f6 (0x74d4 bytes) 000110 1786102014.096458 trunc from 0x0251d8 to 0x027f97 (0x2dc0 bytes) 000111 1786102014.104576 fallocate 0x03a1dd thru 0x03ef83 (0x4da7 bytes) 000112 1786102014.117481 MIRROR_EXTEND to 2 mirrors 000113 1786102014.640701 write 0x01b53c thru 0x025a0f (0xa4d4 bytes) 000114 1786102014.682544 write 0x0053bf thru 0x00f160 (0x9da2 bytes) 000115 1786102014.695674 read 0x0112f1 thru 0x020bfc (0xf90c bytes) 000116 1786102014.705961 punch from 0x00b91b to 0x015f7d (0xa663 bytes) 000117 1786102014.720856 trunc from 0x02d0ab to 0x01419e (0x18f0c bytes) 000118 1786102014.758004 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1419f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1419f): Mirror 2: 0xff17ee16 (stale) CRC-32 checksum value for chunk [0, 0x1419f): Mirror 1: 0x4794800c Mirror 3: 0x4794800c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1419f: skipped 000119 1786102015.603127 trunc from 0x00fd7a to 0x013d09 (0x3f90 bytes) 000120 1786102015.651741 trunc from 0x005020 to 0x0063c1 (0x13a2 bytes) 000121 1786102015.657011 mapread 0x001aed thru 0x00c495 (0xa9a9 bytes) 000122 1786102015.680646 read 0x001475 thru 0x0042f8 (0x2e84 bytes) 000123 1786102015.681499 read 0x00df30 thru 0x01419e (0x626f bytes) 000124 1786102015.698994 read 0x0060f2 thru 0x01419e (0xe0ad bytes) 000125 1786102015.700771 trunc from 0x01419f to 0x03b653 (0x274b5 bytes) 000126 1786102015.751898 punch from 0x03a280 to 0x03b653 (0x13d4 bytes) 000127 1786102015.760653 fallocate 0x032237 thru 0x03863a (0x6404 bytes) 000128 1786102015.771511 read 0x013756 thru 0x019b2e (0x63d9 bytes) 000129 1786102015.785928 trunc from 0x03b654 to 0x0378ab (0x3da8 bytes) 000130 1786102015.801663 mapwrite 0x021316 thru 0x02dc98 (0xc983 bytes) 000131 1786102015.833011 fallocate 0x01981e thru 0x01e633 (0x4e16 bytes) 000132 1786102015.838751 trunc from 0x0378ac to 0x01a509 (0x1d3a2 bytes) 000133 1786102015.871686 punch from 0x013d60 to 0x017023 (0x32c4 bytes) 000134 1786102015.888700 fallocate 0x00e1a8 thru 0x017165 (0x8fbe bytes) 000135 1786102015.893075 mapread 0x0101db thru 0x01a509 (0xa32f bytes) 000136 1786102015.912256 mapwrite 0x036460 thru 0x03ffff (0x9ba0 bytes) 000137 1786102015.996949 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ee9b746 (stale) Mirror 3: 0x91e83808 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x772f1263 Mirror 4: 0x772f1263 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000138 1786102016.846300 fallocate 0x00ddcc thru 0x012559 (0x478e bytes) 000139 1786102016.881277 mapwrite 0x03e47e thru 0x03ffff (0x1b82 bytes) 000141 1786102016.903019 fallocate 0x01758e thru 0x01a765 (0x31d8 bytes) 000142 1786102016.908304 mapwrite 0x020304 thru 0x02c8d8 (0xc5d5 bytes) 000143 1786102016.940118 mapwrite 0x001f7e thru 0x010c8d (0xed10 bytes) 000144 1786102016.986437 write 0x01db89 thru 0x02cd8c (0xf204 bytes) 000146 1786102016.996691 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ee9b746 (stale) Mirror 3: 0x91e83808 (stale) Mirror 4: 0x772f1263 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ca6077 Mirror 5: 0x7ca6077 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000147 1786102017.911475 mapread 0x001014 thru 0x00d711 (0xc6fe bytes) 000148 1786102017.946542 fallocate 0x037987 thru 0x03ffff (0x8679 bytes) 000149 1786102017.986036 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ee9b746 (stale) Mirror 3: 0x91e83808 (stale) Mirror 4: 0x772f1263 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ee9b746 (stale) Mirror 3: 0x91e83808 (stale) Mirror 4: 0x772f1263 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1786102018.384698 fallocate 0x00a888 thru 0x00f183 (0x48fc bytes) 000151 1786102018.390422 read 0x01a376 thru 0x0297cf (0xf45a bytes) 000152 1786102018.411616 mapread 0x01d73c thru 0x0223fb (0x4cc0 bytes) 000153 1786102018.418169 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ee9b746 (stale) Mirror 3: 0x91e83808 (stale) Mirror 4: 0x772f1263 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ca6077 Mirror 5: 0x7ca6077 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000154 1786102019.299694 MIRROR_RESYNC 5 mirrors 000155 1786102020.289072 trunc from 0x040000 to 0x027886 (0x18779 bytes) 000156 1786102020.349407 read 0x01ad72 thru 0x027886 (0xcb15 bytes) 000158 1786102020.360417 trunc from 0x027887 to 0x016665 (0x11221 bytes) 000160 1786102020.382308 mapwrite 0x016589 thru 0x01bd18 (0x5790 bytes) 000161 1786102020.429305 trunc from 0x01bd19 to 0x0146d3 (0x7645 bytes) 000162 1786102020.445716 trunc from 0x0146d4 to 0x023b96 (0xf4c3 bytes) 000163 1786102020.488299 write 0x03149c thru 0x035d31 (0x4896 bytes) 000164 1786102020.491997 trunc from 0x008f75 to 0x016a9f (0xdb2b bytes) 000166 1786102020.502482 read 0x01f5c6 thru 0x024265 (0x4ca0 bytes) 000167 1786102020.516767 MIRROR_RESYNC 5 mirrors 000168 1786102021.444836 trunc from 0x00d2cd to 0x011ba6 (0x48da bytes) 000170 1786102021.487633 trunc from 0x011055 to 0x014631 (0x35dd bytes) 000171 1786102021.501789 trunc from 0x02e0a4 to 0x035d31 (0x7c8e bytes) 000172 1786102021.515213 mapread 0x0174ec thru 0x01a911 (0x3426 bytes) 000173 1786102021.542640 trunc from 0x015786 to 0x015d14 (0x58f bytes) 000174 1786102021.546770 trunc from 0x01aa05 to 0x01c527 (0x1b23 bytes) 000175 1786102021.554447 mapwrite 0x01410e thru 0x01514b (0x103e bytes) 000176 1786102021.634683 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35d32): Mirror 2: 0x120fa5e6 (stale) Mirror 4: 0x120fa5e6 (stale) Mirror 5: 0x120fa5e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d32): Mirror 2: 0x120fa5e6 (stale) Mirror 4: 0x120fa5e6 (stale) Mirror 5: 0x120fa5e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d32: skipped 000177 1786102022.049913 write 0x00d468 thru 0x01cedc (0xfa75 bytes) 000179 1786102022.070738 trunc from 0x035d32 to 0x026710 (0xf621 bytes) 000180 1786102022.091081 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: 0xb6cfc484 (stale) Mirror 4: 0xb6cfc484 (stale) Mirror 5: 0xb6cfc484 (stale) CRC-32 checksum value for chunk [0, 0x26711): Mirror 1: 0x6b497dc1 Mirror 6: 0x6b497dc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26711: skipped 000181 1786102022.906032 trunc from 0x004096 to 0x006063 (0x1fce bytes) 000182 1786102022.970140 read 0x020350 thru 0x026710 (0x63c1 bytes) 000183 1786102022.985140 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: 0xb6cfc484 (stale) Mirror 4: 0xb6cfc484 (stale) Mirror 5: 0xb6cfc484 (stale) Mirror 6: 0x6b497dc1 (stale) CRC-32 checksum value for chunk [0, 0x26711): Mirror 1: 0xde636710 Mirror 7: 0xde636710 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26711: skipped 000185 1786102024.004516 mapread 0x023ff9 thru 0x024128 (0x130 bytes) 000186 1786102024.065201 trunc from 0x00e328 to 0x016a61 (0x873a bytes) 000187 1786102024.131546 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: 0xb6cfc484 (stale) Mirror 4: 0xb6cfc484 (stale) Mirror 5: 0xb6cfc484 (stale) Mirror 6: 0x6b497dc1 (stale) Mirror 7: 0xde636710 (stale) CRC-32 checksum value for chunk [0, 0x26711): Mirror 1: 0x5f008f81 Mirror 8: 0x5f008f81 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26711: skipped 000188 1786102025.228659 mapread 0x006273 thru 0x008273 (0x2001 bytes) 000189 1786102025.260024 read 0x0155b5 thru 0x017203 (0x1c4f bytes) 000190 1786102025.271802 trunc from 0x00a061 to 0x00fb55 (0x5af5 bytes) 000191 1786102025.319887 fallocate 0x013a36 thru 0x01495f (0xf2a bytes) 000192 1786102025.327644 mapread 0x002fb4 thru 0x006e00 (0x3e4d bytes) 000193 1786102025.340924 mapread 0x0219e1 thru 0x026710 (0x4d30 bytes) 000194 1786102025.355050 fallocate 0x00a8e2 thru 0x01341b (0x8b3a bytes) 000195 1786102025.364380 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: 0xb6cfc484 (stale) Mirror 4: 0xb6cfc484 (stale) Mirror 5: 0xb6cfc484 (stale) Mirror 6: 0x6b497dc1 (stale) Mirror 8: 0x5f008f81 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26711): Mirror 2: 0xb6cfc484 (stale) Mirror 4: 0xb6cfc484 (stale) Mirror 5: 0xb6cfc484 (stale) Mirror 6: 0x6b497dc1 (stale) Mirror 8: 0x5f008f81 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26711: skipped 000196 1786102025.882060 MIRROR_RESYNC 6 mirrors 000197 1786102027.747119 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000198 1786102028.203228 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000199 1786102028.614286 mapwrite 0x038f86 thru 0x03ffff (0x707a bytes) 000201 1786102028.715515 read 0x02268f thru 0x02d91c (0xb28e bytes) 000202 1786102028.735212 trunc from 0x040000 to 0x02c8ed (0x13712 bytes) 000204 1786102028.768716 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c8ee): Mirror 2: 0x95c8b4a2 (stale) Mirror 4: 0x95c8b4a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c8ee): Mirror 2: 0x95c8b4a2 (stale) Mirror 4: 0x95c8b4a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c8ee: skipped 000205 1786102029.128701 mapread 0x010e0f thru 0x01bb81 (0xad73 bytes) 000206 1786102029.181784 trunc from 0x02c8ee to 0x026774 (0x6179 bytes) 000208 1786102029.218432 read 0x002028 thru 0x009ff6 (0x7fcf bytes) 000209 1786102029.229410 read 0x010512 thru 0x017056 (0x6b45 bytes) 000210 1786102029.230397 punch from 0x00cab9 to 0x011e58 (0x53a0 bytes) 000211 1786102029.234430 read 0x00ddf7 thru 0x010884 (0x2a8e bytes) 000212 1786102029.242442 fallocate 0x031640 thru 0x03a7aa (0x916b bytes) 000213 1786102029.247134 write 0x0305c8 thru 0x0386aa (0x80e3 bytes) 000214 1786102029.248721 trunc from 0x02bd6f to 0x0386aa (0xc93c bytes) 000215 1786102029.261568 write 0x02197d thru 0x023a79 (0x20fd bytes) 000216 1786102029.262717 read 0x02df40 thru 0x031331 (0x33f2 bytes) 000217 1786102029.271195 trunc from 0x025968 to 0x033a5d (0xe0f6 bytes) 000220 1786102029.277306 read 0x00b564 thru 0x00d4c6 (0x1f63 bytes) 000221 1786102029.278409 mapwrite 0x032c92 thru 0x03d1ac (0xa51b bytes) 000222 1786102029.366342 mapread 0x00651b thru 0x014d14 (0xe7fa bytes) 000224 1786102029.375456 mapread 0x037fb5 thru 0x0399ab (0x19f7 bytes) 000226 1786102029.390351 MIRROR_RESYNC 3 mirrors 000227 1786102030.520612 trunc from 0x03d1ad to 0x018554 (0x24c58 bytes) 000228 1786102030.590058 mapwrite 0x036683 thru 0x03e88a (0x8208 bytes) 000230 1786102030.709043 trunc from 0x03e88b to 0x01de7e (0x20a0c bytes) 000231 1786102030.722557 fallocate 0x01f9b9 thru 0x02e2ce (0xe916 bytes) 000232 1786102030.728986 read 0x01b0aa thru 0x01de7e (0x2dd5 bytes) 000233 1786102030.734971 fallocate 0x00b334 thru 0x015b15 (0xa7e2 bytes) 000234 1786102030.743418 write 0x030f63 thru 0x033d3f (0x2ddd bytes) 000236 1786102030.745813 trunc from 0x0269d2 to 0x02cb80 (0x61af bytes) 000237 1786102030.750086 trunc from 0x023f67 to 0x025895 (0x192f bytes) 000238 1786102030.756769 trunc from 0x033d40 to 0x03a324 (0x65e5 bytes) 000240 1786102030.773703 mapwrite 0x036440 thru 0x03de38 (0x79f9 bytes) 000241 1786102030.837235 mapwrite 0x0142f6 thru 0x01d0d5 (0x8de0 bytes) 000242 1786102030.865095 trunc from 0x00dc08 to 0x01c9cf (0xedc8 bytes) 000243 1786102030.876478 fallocate 0x0126dd thru 0x012f1c (0x840 bytes) 000244 1786102030.881251 mapwrite 0x037056 thru 0x03ffff (0x8faa bytes) 000245 1786102030.918212 punch from 0x0097c9 to 0x00df2b (0x4763 bytes) 000246 1786102030.924528 fallocate 0x026717 thru 0x02eb2f (0x8419 bytes) 000247 1786102030.932371 mapwrite 0x03cf7b thru 0x03ffff (0x3085 bytes) 000248 1786102030.953095 mapread 0x03f089 thru 0x03ffff (0xf77 bytes) 000249 1786102030.954295 read 0x022b6b thru 0x030130 (0xd5c6 bytes) 000250 1786102030.978376 mapread 0x032c09 thru 0x035201 (0x25f9 bytes) 000251 1786102030.979670 read 0x00388e thru 0x007239 (0x39ac bytes) 000253 1786102030.987721 trunc from 0x040000 to 0x0248bf (0x1b740 bytes) 000255 1786102031.016678 trunc from 0x0248c0 to 0x029ba6 (0x52e7 bytes) 000257 1786102031.067169 fallocate 0x0169d7 thru 0x0207d3 (0x9dfd bytes) 000259 1786102031.077443 punch from 0x020be9 to 0x029ba6 (0x8fbe bytes) 000260 1786102031.089555 fallocate 0x00ec01 thru 0x016ec5 (0x82c5 bytes) 000261 1786102031.100071 fallocate 0x00ed79 thru 0x00f91f (0xba7 bytes) 000262 1786102031.106809 trunc from 0x025213 to 0x029ba6 (0x4994 bytes) 000263 1786102031.116886 mapread 0x00a9d5 thru 0x011003 (0x662f bytes) 000264 1786102031.119246 mapread 0x0181c4 thru 0x027783 (0xf5c0 bytes) 000265 1786102031.137998 mapread 0x009c52 thru 0x014b53 (0xaf02 bytes) 000266 1786102031.139217 trunc from 0x015955 to 0x02017a (0xa826 bytes) 000267 1786102031.150652 read 0x019558 thru 0x01a83c (0x12e5 bytes) 000268 1786102031.163281 mapwrite 0x02cf62 thru 0x0374ff (0xa59e bytes) 000269 1786102031.230637 read 0x00ab64 thru 0x012818 (0x7cb5 bytes) 000270 1786102031.232002 fallocate 0x0240c5 thru 0x02b2ba (0x71f6 bytes) 000272 1786102031.238917 fallocate 0x036774 thru 0x03d8b0 (0x713d bytes) 000273 1786102031.244565 read 0x01bcd4 thru 0x0202cc (0x45f9 bytes) 000274 1786102031.245268 mapwrite 0x026c78 thru 0x02b062 (0x43eb bytes) 000276 1786102031.279255 MIRROR_RESYNC 3 mirrors 000277 1786102032.084839 fallocate 0x03a9f3 thru 0x03aff3 (0x601 bytes) 000278 1786102032.110452 write 0x004ead thru 0x01208a (0xd1de bytes) 000279 1786102032.123043 punch from 0x00ccf6 to 0x015533 (0x883e bytes) 000280 1786102032.140727 trunc from 0x037500 to 0x0331ce (0x4331 bytes) 000282 1786102032.174714 trunc from 0x0279e7 to 0x0331ce (0xb7e8 bytes) 000283 1786102032.180625 trunc from 0x013eb1 to 0x0184ce (0x461e bytes) 000284 1786102032.184616 mapread 0x017f84 thru 0x02337d (0xb3fa bytes) 000285 1786102032.207924 trunc from 0x025d91 to 0x02aaf5 (0x4d65 bytes) 000287 1786102032.213239 mapwrite 0x00c70f thru 0x00d43d (0xd2f bytes) 000288 1786102032.231399 mapread 0x0058d2 thru 0x009c00 (0x432f bytes) 000289 1786102032.232714 write 0x00c526 thru 0x00e95b (0x2436 bytes) 000290 1786102032.234060 mapread 0x011c19 thru 0x0204df (0xe8c7 bytes) 000291 1786102032.236941 write 0x026099 thru 0x034911 (0xe879 bytes) 000292 1786102032.244263 write 0x001085 thru 0x004bbe (0x3b3a bytes) 000293 1786102032.251698 mapwrite 0x029a70 thru 0x02d1fb (0x378c bytes) 000294 1786102032.270610 mapwrite 0x01c1ae thru 0x027a36 (0xb889 bytes) 000295 1786102032.295239 write 0x01e87a thru 0x028016 (0x979d bytes) 000296 1786102032.296759 punch from 0x030847 to 0x034911 (0x40cb bytes) 000297 1786102032.302032 trunc from 0x034912 to 0x00520f (0x2f702 bytes) 000298 1786102032.320530 mapwrite 0x0078d7 thru 0x00a919 (0x3043 bytes) 000299 1786102032.365938 punch from 0x004e2d to 0x00a919 (0x5aed bytes) 000300 1786102032.371300 mapread 0x0099f8 thru 0x00a919 (0xf22 bytes) 000301 1786102032.381720 fallocate 0x00fc8e thru 0x0163f7 (0x676a bytes) 000302 1786102032.388229 trunc from 0x00a91a to 0x022de8 (0x184cf bytes) 000303 1786102032.402724 trunc from 0x022de9 to 0x02c47c (0x9694 bytes) 000304 1786102032.435224 punch from 0x01dce1 to 0x022bee (0x4f0e bytes) 000305 1786102032.441477 mapread 0x0258cc thru 0x026b78 (0x12ad bytes) 000306 1786102032.455863 trunc from 0x020b42 to 0x0251bd (0x467c bytes) 000307 1786102032.465433 trunc from 0x00de71 to 0x01ca47 (0xebd7 bytes) 000308 1786102032.473833 write 0x019e07 thru 0x026ddb (0xcfd5 bytes) 000309 1786102032.482160 read 0x00681f thru 0x00f2d1 (0x8ab3 bytes) 000311 1786102032.492708 write 0x009cab thru 0x017277 (0xd5cd bytes) 000312 1786102032.505201 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c47d): Mirror 4: 0x16635353 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c47d): Mirror 4: 0x16635353 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c47d: skipped 000314 1786102032.844143 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c47d): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c47d): Mirror 4: 0x16635353 (stale) CRC-32 checksum value for chunk [0, 0x2c47d): Mirror 1: 0x4888e85 Mirror 5: 0x4888e85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c47d: skipped 000315 1786102033.731700 trunc from 0x02c47d to 0x01105b (0x1b421 bytes) 000316 1786102033.790634 trunc from 0x0033e7 to 0x007c15 (0x482f bytes) 000317 1786102033.797617 read 0x00e572 thru 0x01105b (0x2aea bytes) 000318 1786102033.806206 trunc from 0x00b13f to 0x01105b (0x5f1d bytes) 000319 1786102033.825511 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1105c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1105c): Mirror 4: 0x3d084545 (stale) Mirror 5: 0xb90c099e (stale) CRC-32 checksum value for chunk [0, 0x1105c): Mirror 1: 0x120df0af Mirror 6: 0x120df0af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1105c: skipped 000320 1786102034.981004 MIRROR_RESYNC 4 mirrors 000322 1786102035.814345 punch from 0x004375 to 0x010fd3 (0xcc5f bytes) 000323 1786102035.852853 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1105c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1105c): Mirror 4: 0x120df0af (stale) Mirror 5: 0x120df0af (stale) Mirror 6: 0x120df0af (stale) CRC-32 checksum value for chunk [0, 0x1105c): Mirror 1: 0x5fc1fa00 Mirror 7: 0x5fc1fa00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1105c: skipped 000324 1786102036.715964 read 0x00b653 thru 0x01105b (0x5a09 bytes) 000325 1786102036.730016 mapwrite 0x03540f thru 0x03ffff (0xabf1 bytes) 000326 1786102036.824284 punch from 0x019c6c to 0x025cba (0xc04f bytes) 000328 1786102036.829508 trunc from 0x0058db to 0x014149 (0xe86f bytes) 000329 1786102036.834527 mapread 0x0367f1 thru 0x03872a (0x1f3a bytes) 000330 1786102036.836016 fallocate 0x035e23 thru 0x03ffff (0xa1dd bytes) 000331 1786102036.840999 trunc from 0x02ba4f to 0x032414 (0x69c6 bytes) 000332 1786102036.845871 read 0x03d8a5 thru 0x03ffff (0x275b bytes) 000333 1786102036.847212 mapwrite 0x00ac7c thru 0x01a657 (0xf9dc bytes) 000334 1786102036.886298 punch from 0x011840 to 0x01f40b (0xdbcc bytes) 000335 1786102036.891249 mapwrite 0x023da2 thru 0x03279e (0xe9fd bytes) 000336 1786102036.910264 write 0x037a1c thru 0x03ffff (0x85e4 bytes) 000337 1786102036.911626 trunc from 0x040000 to 0x03a636 (0x59c9 bytes) 000338 1786102036.934168 trunc from 0x03a637 to 0x02b40d (0xf229 bytes) 000339 1786102036.958582 MIRROR_RESYNC 5 mirrors 000340 1786102038.321675 mapread 0x00e2a5 thru 0x015d07 (0x7a63 bytes) 000341 1786102038.338104 mapwrite 0x02dd66 thru 0x032e8c (0x5127 bytes) 000342 1786102038.409696 fallocate 0x008481 thru 0x009226 (0xda6 bytes) 000343 1786102038.415571 mapread 0x02fc76 thru 0x032e8c (0x3217 bytes) 000344 1786102038.418205 punch from 0x024b68 to 0x0325c9 (0xda62 bytes) 000345 1786102038.423485 punch from 0x02d9be to 0x032e8c (0x54cf bytes) 000346 1786102038.430962 trunc from 0x030832 to 0x031227 (0x9f6 bytes) 000348 1786102038.446779 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32e8d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e8d): Mirror 4: 0x21790336 (stale) Mirror 5: 0x21790336 (stale) Mirror 6: 0x21790336 (stale) Mirror 7: 0x21790336 (stale) CRC-32 checksum value for chunk [0, 0x32e8d): Mirror 1: 0xe3c2e8af Mirror 8: 0xe3c2e8af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e8d: skipped 000349 1786102039.325353 mapwrite 0x03a660 thru 0x03ffff (0x59a0 bytes) 000350 1786102039.459176 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x83b8bdba (stale) Mirror 5: 0x83b8bdba (stale) Mirror 6: 0x83b8bdba (stale) Mirror 7: 0x83b8bdba (stale) Mirror 8: 0xe394582 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd99ada11 Mirror 9: 0xd99ada11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000351 1786102040.297450 mapread 0x008025 thru 0x00bdd1 (0x3dad bytes) 000352 1786102040.370105 fallocate 0x0288fd thru 0x034be1 (0xc2e5 bytes) 000353 1786102040.424630 mapread 0x003688 thru 0x0061b7 (0x2b30 bytes) 000354 1786102040.459659 fallocate 0x02f1b2 thru 0x032699 (0x34e8 bytes) 000355 1786102040.472140 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x83b8bdba (stale) Mirror 6: 0x83b8bdba (stale) Mirror 7: 0x83b8bdba (stale) Mirror 8: 0xe394582 (stale) Mirror 9: 0xd99ada11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x83b8bdba (stale) Mirror 6: 0x83b8bdba (stale) Mirror 7: 0x83b8bdba (stale) Mirror 8: 0xe394582 (stale) Mirror 9: 0xd99ada11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000356 1786102040.914550 trunc from 0x02eaf8 to 0x0322c4 (0x37cd bytes) 000357 1786102040.920354 fallocate 0x015ac1 thru 0x01f425 (0x9965 bytes) 000358 1786102040.925355 write 0x01b0b7 thru 0x022011 (0x6f5b bytes) 000360 1786102040.942243 punch from 0x00849b to 0x016984 (0xe4ea bytes) 000361 1786102040.947049 trunc from 0x040000 to 0x00fe1b (0x301e4 bytes) 000362 1786102040.985567 fallocate 0x03413b thru 0x03efa6 (0xae6c bytes) 000363 1786102040.989637 trunc from 0x00fe1c to 0x02d4f4 (0x1d6d9 bytes) 000364 1786102041.006542 punch from 0x02893e to 0x02d3a4 (0x4a67 bytes) 000366 1786102041.017403 fallocate 0x0098f8 thru 0x00d2d3 (0x39dc bytes) 000367 1786102041.026567 punch from 0x00b02a to 0x01a35b (0xf332 bytes) 000368 1786102041.031805 punch from 0x00913d to 0x0134f1 (0xa3b5 bytes) 000369 1786102041.037620 mapwrite 0x00be6e thru 0x01390f (0x7aa2 bytes) 000370 1786102041.100130 punch from 0x00aadf to 0x00c7f2 (0x1d14 bytes) 000371 1786102041.107156 trunc from 0x02d4f5 to 0x00ec6e (0x1e886 bytes) 000372 1786102041.126366 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xec6f): Mirror 6: 0x84bfabaa (stale) Mirror 7: 0x84bfabaa (stale) Mirror 8: 0x84bfabaa (stale) Mirror 9: 0x84bfabaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xec6f): Mirror 6: 0x84bfabaa (stale) Mirror 7: 0x84bfabaa (stale) Mirror 8: 0x84bfabaa (stale) Mirror 9: 0x84bfabaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xec6f: skipped 000373 1786102041.619593 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xec6f): Mirror 7: 0x84bfabaa (stale) Mirror 8: 0x84bfabaa (stale) Mirror 9: 0x84bfabaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xec6f): Mirror 7: 0x84bfabaa (stale) Mirror 8: 0x84bfabaa (stale) Mirror 9: 0x84bfabaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xec6f: skipped 000374 1786102042.051104 mapread 0x00aa47 thru 0x00ec6e (0x4228 bytes) 000375 1786102042.069904 punch from 0x00bdb9 to 0x00c96b (0xbb3 bytes) 000376 1786102042.081462 punch from 0x00d7d0 to 0x00ec6e (0x149f bytes) 000377 1786102042.086167 mapwrite 0x031dd6 thru 0x03bb49 (0x9d74 bytes) 000378 1786102042.125565 write 0x016ea4 thru 0x021107 (0xa264 bytes) 000379 1786102042.133760 write 0x03ce18 thru 0x03ffff (0x31e8 bytes) 000380 1786102042.136194 punch from 0x01edf0 to 0x02e9ca (0xfbdb bytes) 000381 1786102042.146956 write 0x01ac4c thru 0x021fa0 (0x7355 bytes) 000382 1786102042.152103 fallocate 0x02d053 thru 0x03377d (0x672b bytes) 000383 1786102042.156818 mapread 0x00a466 thru 0x011af1 (0x768c bytes) 000384 1786102042.165666 mapwrite 0x00498e thru 0x00b9d8 (0x704b bytes) 000385 1786102042.186511 mapwrite 0x012beb thru 0x018f09 (0x631f bytes) 000386 1786102042.208165 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x83b8bdba (stale) Mirror 8: 0xe394582 (stale) Mirror 9: 0xd99ada11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9b3dde49 Mirror 10: 0x9b3dde49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000387 1786102043.027916 punch from 0x0315d8 to 0x036f86 (0x59af bytes) 000389 1786102043.059276 trunc from 0x040000 to 0x032b1d (0xd4e2 bytes) 000390 1786102043.101965 read 0x031392 thru 0x032b1d (0x178c bytes) 000391 1786102043.116305 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32b1e): Mirror 7: 0x4b338a11 (stale) Mirror 8: 0x3b438e09 (stale) Mirror 10: 0xdb34377b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32b1e): Mirror 7: 0x4b338a11 (stale) Mirror 8: 0x3b438e09 (stale) Mirror 10: 0xdb34377b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32b1e: skipped 000392 1786102043.477732 fallocate 0x018c5a thru 0x024282 (0xb629 bytes) 000393 1786102043.488338 trunc from 0x032b1e to 0x01652c (0x1c5f1 bytes) 000394 1786102043.526804 MIRROR_RESYNC 4 mirrors 000395 1786102044.472885 mapread 0x00ffe5 thru 0x01652c (0x6548 bytes) 000396 1786102044.498647 MIRROR_EXTEND to 5 mirrors 000397 1786102045.061155 trunc from 0x01652d to 0x00c2db (0xa251 bytes) 000398 1786102045.135765 punch from 0x00548f to 0x00c2db (0x6e4d bytes) 000399 1786102045.145616 fallocate 0x0296e3 thru 0x0357b3 (0xc0d1 bytes) 000400 1786102045.155465 read 0x02fc8d thru 0x0357b3 (0x5b27 bytes) 000403 1786102045.179680 punch from 0x000063 to 0x00f2be (0xf25c bytes) 000404 1786102045.191802 punch from 0x00277d to 0x00af28 (0x87ac bytes) 000405 1786102045.197714 trunc from 0x016a0f to 0x019f07 (0x34f9 bytes) 000406 1786102045.204608 mapread 0x02a27a thru 0x0357b3 (0xb53a bytes) 000407 1786102045.228366 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x357b4): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x357b4): Mirror 7: 0x6fc90203 (stale) Mirror 8: 0x6fc90203 (stale) Mirror 10: 0x6fc90203 (stale) Mirror 11: 0x6fc90203 (stale) CRC-32 checksum value for chunk [0, 0x357b4): Mirror 1: 0x30babba0 Mirror 12: 0x30babba0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x357b4: skipped 000408 1786102045.925242 fallocate 0x011bcd thru 0x01b3fb (0x982f bytes) 000410 1786102045.982760 mapread 0x0103ec thru 0x0136f0 (0x3305 bytes) 000411 1786102046.000825 mapread 0x0230ec thru 0x02cc07 (0x9b1c bytes) 000412 1786102046.018973 mapread 0x027f5c thru 0x02cef0 (0x4f95 bytes) 000413 1786102046.024592 write 0x015d21 thru 0x01c2ac (0x658c bytes) 000414 1786102046.031274 punch from 0x018ef3 to 0x024186 (0xb294 bytes) 000415 1786102046.044766 trunc from 0x013132 to 0x0154a7 (0x2376 bytes) 000416 1786102046.051624 trunc from 0x0342ba to 0x0357b3 (0x14fa bytes) 000417 1786102046.058026 trunc from 0x0080cb to 0x015536 (0xd46c bytes) 000418 1786102046.063756 trunc from 0x0357b4 to 0x01e854 (0x16f5f bytes) 000419 1786102046.115965 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1e855): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e855): Mirror 7: 0x520e06f3 (stale) Mirror 8: 0x520e06f3 (stale) Mirror 10: 0x520e06f3 (stale) Mirror 11: 0x520e06f3 (stale) Mirror 12: 0x9bf0a57a (stale) CRC-32 checksum value for chunk [0, 0x1e855): Mirror 1: 0xb18a60df Mirror 13: 0xb18a60df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e855: skipped 000420 1786102046.961211 mapwrite 0x00efcf thru 0x013901 (0x4933 bytes) 000421 1786102047.081298 trunc from 0x00b184 to 0x01044d (0x52ca bytes) 000422 1786102047.086246 fallocate 0x0084fe thru 0x00a633 (0x2136 bytes) 000423 1786102047.091388 write 0x02e921 thru 0x032494 (0x3b74 bytes) 000424 1786102047.095890 write 0x032b2b thru 0x03ffff (0xd4d5 bytes) 000425 1786102047.099426 mapwrite 0x016288 thru 0x01c241 (0x5fba bytes) 000426 1786102047.127447 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa8770ef5 (stale) Mirror 8: 0xa8770ef5 (stale) Mirror 10: 0xa8770ef5 (stale) Mirror 11: 0xa8770ef5 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe2eb086f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf0a5821 Mirror 14: 0xaf0a5821 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000427 1786102047.800942 mapwrite 0x03a855 thru 0x03ffff (0x57ab bytes) 000428 1786102047.932432 mapread 0x0219ea thru 0x0255b6 (0x3bcd bytes) 000429 1786102047.952207 punch from 0x01fd08 to 0x02b629 (0xb922 bytes) 000431 1786102047.960969 fallocate 0x035f22 thru 0x03ffff (0xa0de bytes) 000432 1786102047.967491 write 0x03dd9a thru 0x03ffff (0x2266 bytes) 000433 1786102047.980054 mapread 0x007f04 thru 0x012201 (0xa2fe bytes) 000434 1786102048.002159 trunc from 0x040000 to 0x03cd73 (0x328c bytes) 000435 1786102048.060695 trunc from 0x03cd74 to 0x010c41 (0x2c132 bytes) 000437 1786102048.079510 trunc from 0x010c42 to 0x0393f8 (0x287b7 bytes) 000438 1786102048.104817 trunc from 0x0393f9 to 0x02bd12 (0xd6e6 bytes) 000439 1786102048.128442 fallocate 0x03acfd thru 0x03e86c (0x3b70 bytes) 000440 1786102048.135740 mapwrite 0x035570 thru 0x03ce03 (0x7894 bytes) 000441 1786102048.197249 mapread 0x011898 thru 0x018f58 (0x76c1 bytes) 000442 1786102048.209729 mapread 0x035733 thru 0x03704f (0x191d bytes) 000443 1786102048.210804 trunc from 0x03ce04 to 0x00131b (0x3bae8 bytes) 000444 1786102048.229625 trunc from 0x00131c to 0x021a70 (0x20755 bytes) 000445 1786102048.251565 trunc from 0x021a71 to 0x031e63 (0x103f3 bytes) 000446 1786102048.281921 fallocate 0x019dab thru 0x02787e (0xdad4 bytes) 000447 1786102048.292771 fallocate 0x03dd30 thru 0x03ffff (0x22d0 bytes) 000448 1786102048.301983 fallocate 0x0095e8 thru 0x0110f3 (0x7b0c bytes) 000450 1786102048.309979 write 0x0109b9 thru 0x0178b9 (0x6f01 bytes) 000451 1786102048.325793 write 0x01e699 thru 0x02c156 (0xdabe bytes) 000452 1786102048.339585 trunc from 0x00c5be to 0x01bc35 (0xf678 bytes) 000453 1786102048.360538 trunc from 0x040000 to 0x003a32 (0x3c5cd bytes) 000454 1786102048.372452 trunc from 0x003a33 to 0x00444d (0xa1b bytes) 000455 1786102048.390306 trunc from 0x000695 to 0x00444d (0x3db9 bytes) 000456 1786102048.397630 write 0x039597 thru 0x03ffff (0x6a69 bytes) 000457 1786102048.400331 trunc from 0x040000 to 0x01c98c (0x23673 bytes) 000458 1786102048.415399 mapread 0x00d755 thru 0x00db0b (0x3b7 bytes) 000460 1786102048.430066 read 0x012b1e thru 0x018e67 (0x634a bytes) 000461 1786102048.440138 fallocate 0x025ea9 thru 0x029235 (0x338d bytes) 000463 1786102048.447099 trunc from 0x01be6c to 0x0263d4 (0xa569 bytes) 000464 1786102048.457335 write 0x023bdd thru 0x0304d8 (0xc8fc bytes) 000465 1786102048.473696 mapread 0x00dc0b thru 0x0137ed (0x5be3 bytes) 000466 1786102048.482191 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x304d9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x304d9): Mirror 7: 0x99c63f99 (stale) Mirror 8: 0x99c63f99 (stale) Mirror 10: 0x99c63f99 (stale) Mirror 11: 0x99c63f99 (stale) Mirror 12: 0xfe933b81 (stale) Mirror 13: 0x6f685660 (stale) Mirror 14: 0xb1e4387 (stale) CRC-32 checksum value for chunk [0, 0x304d9): Mirror 1: 0x3b744599 Mirror 15: 0x3b744599 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x304d9: skipped 000467 1786102049.262520 punch from 0x02e9a1 to 0x0304d8 (0x1b38 bytes) 000468 1786102049.292956 trunc from 0x016523 to 0x022e61 (0xc93f bytes) 000469 1786102049.303293 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x304d9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x304d9): Mirror 7: 0x99c63f99 (stale) Mirror 8: 0x99c63f99 (stale) Mirror 10: 0x99c63f99 (stale) Mirror 11: 0x99c63f99 (stale) Mirror 12: 0xfe933b81 (stale) Mirror 13: 0x6f685660 (stale) Mirror 14: 0xb1e4387 (stale) Mirror 15: 0x3b744599 (stale) CRC-32 checksum value for chunk [0, 0x304d9): Mirror 1: 0x8772b057 Mirror 16: 0x8772b057 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x304d9: skipped 000470 1786102050.253611 mapwrite 0x02cdf5 thru 0x033d59 (0x6f65 bytes) 000471 1786102050.343103 mapread 0x030458 thru 0x033d59 (0x3902 bytes) 000473 1786102050.347156 punch from 0x005856 to 0x008a84 (0x322f bytes) 000474 1786102050.353852 trunc from 0x02672c to 0x033d59 (0xd62e bytes) 000475 1786102050.359540 trunc from 0x0147e9 to 0x01dfad (0x97c5 bytes) 000476 1786102050.365615 trunc from 0x033d5a to 0x00047f (0x338da bytes) 000477 1786102050.385165 write 0x012cfd thru 0x01f06b (0xc36f bytes) 000478 1786102050.387380 fallocate 0x0050ce thru 0x00640e (0x1341 bytes) 000479 1786102050.392811 mapread 0x00e852 thru 0x018b3b (0xa2ea bytes) 000480 1786102050.409737 mapread 0x019512 thru 0x01f06b (0x5b5a bytes) 000482 1786102050.412718 mapread 0x001bbf thru 0x003290 (0x16d2 bytes) 000483 1786102050.419804 mapwrite 0x026ab7 thru 0x0356fa (0xec44 bytes) 000484 1786102050.481893 mapread 0x02fc7d thru 0x0308f4 (0xc78 bytes) 000485 1786102050.497415 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x356fb): Mirror 7: 0x330e18f1 (stale) Mirror 8: 0x330e18f1 (stale) Mirror 10: 0x330e18f1 (stale) Mirror 11: 0x330e18f1 (stale) Mirror 13: 0x8633be1c (stale) Mirror 14: 0xce8d2e4c (stale) Mirror 15: 0xce153eaf (stale) Mirror 16: 0xe9146ba1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x356fb): Mirror 7: 0x330e18f1 (stale) Mirror 8: 0x330e18f1 (stale) Mirror 10: 0x330e18f1 (stale) Mirror 11: 0x330e18f1 (stale) Mirror 13: 0x8633be1c (stale) Mirror 14: 0xce8d2e4c (stale) Mirror 15: 0xce153eaf (stale) Mirror 16: 0xe9146ba1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x356fb: skipped 000486 1786102051.010226 fallocate 0x027ff6 thru 0x02983f (0x184a bytes) 000487 1786102051.016416 trunc from 0x027688 to 0x02bb1a (0x4493 bytes) 000488 1786102051.022949 read 0x02e00c thru 0x0356fa (0x76ef bytes) 000489 1786102051.041218 mapread 0x00829d thru 0x011add (0x9841 bytes) 000490 1786102051.069076 punch from 0x031fc0 to 0x0356fa (0x373b bytes) 000492 1786102051.081134 trunc from 0x00e0d5 to 0x015a8d (0x79b9 bytes) 000493 1786102051.095380 write 0x02c2dc thru 0x03ba11 (0xf736 bytes) 000494 1786102051.129619 fallocate 0x00afd2 thru 0x012ead (0x7edc bytes) 000495 1786102051.143097 fallocate 0x016a88 thru 0x01d74e (0x6cc7 bytes) 000496 1786102051.176725 write 0x0090a5 thru 0x00f2d3 (0x622f bytes) 000497 1786102051.203148 MIRROR_RESYNC 9 mirrors 000498 1786102053.569122 mapwrite 0x03e33b thru 0x03ffff (0x1cc5 bytes) 000500 1786102053.640900 write 0x034e4d thru 0x036099 (0x124d bytes) 000501 1786102053.654577 mapwrite 0x02d1c3 thru 0x02f02f (0x1e6d bytes) 000502 1786102053.672251 mapwrite 0x0135c7 thru 0x0189a6 (0x53e0 bytes) 000503 1786102053.690110 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf3c7f2ef (stale) Mirror 8: 0xf3c7f2ef (stale) Mirror 10: 0xf3c7f2ef (stale) Mirror 11: 0xf3c7f2ef (stale) Mirror 13: 0xf3c7f2ef (stale) Mirror 14: 0xf3c7f2ef (stale) Mirror 15: 0xf3c7f2ef (stale) Mirror 16: 0xf3c7f2ef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x33ae73f4 Mirror 17: 0x33ae73f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000504 1786102054.744208 punch from 0x01214b to 0x015fe6 (0x3e9c bytes) 000505 1786102054.785957 write 0x007d62 thru 0x00f453 (0x76f2 bytes) 000506 1786102054.801892 trunc from 0x040000 to 0x03ad88 (0x5277 bytes) 000507 1786102054.834980 MIRROR_RESYNC 10 mirrors 000508 1786102058.457418 trunc from 0x00b7e8 to 0x018596 (0xcdaf bytes) 000509 1786102058.504779 mapwrite 0x000e21 thru 0x004767 (0x3947 bytes) 000510 1786102058.557267 trunc from 0x03ad89 to 0x03885e (0x252a bytes) 000511 1786102058.583707 mapwrite 0x02014c thru 0x0278f0 (0x77a5 bytes) 000512 1786102058.612569 punch from 0x0059de to 0x0122c2 (0xc8e5 bytes) 000513 1786102058.618511 mapwrite 0x03a6c1 thru 0x03ffff (0x593f bytes) 000514 1786102058.670088 punch from 0x011181 to 0x0186a3 (0x7523 bytes) 000515 1786102058.699294 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6ff4d97e (stale) Mirror 8: 0x6ff4d97e (stale) Mirror 10: 0x6ff4d97e (stale) Mirror 11: 0x6ff4d97e (stale) Mirror 13: 0x6ff4d97e (stale) Mirror 14: 0x6ff4d97e (stale) Mirror 15: 0x6ff4d97e (stale) Mirror 16: 0x6ff4d97e (stale) Mirror 17: 0x6ff4d97e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2daa53d2 Mirror 18: 0x2daa53d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000516 1786102060.745395 mapwrite 0x00c64b thru 0x018850 (0xc206 bytes) 000517 1786102061.038119 fallocate 0x029954 thru 0x0319d6 (0x8083 bytes) 000518 1786102061.068808 trunc from 0x040000 to 0x036cf6 (0x9309 bytes) 000519 1786102061.113663 punch from 0x0309fd to 0x036cf6 (0x62fa bytes) 000520 1786102061.130507 read 0x01d8c3 thru 0x022349 (0x4a87 bytes) 000521 1786102061.154576 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36cf7): Mirror 7: 0xaa74396a (stale) Mirror 8: 0xaa74396a (stale) Mirror 10: 0xaa74396a (stale) Mirror 11: 0xaa74396a (stale) Mirror 14: 0xaa74396a (stale) Mirror 15: 0xaa74396a (stale) Mirror 16: 0xaa74396a (stale) Mirror 17: 0xaa74396a (stale) Mirror 18: 0x1eb7dd30 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cf7): Mirror 7: 0xaa74396a (stale) Mirror 8: 0xaa74396a (stale) Mirror 10: 0xaa74396a (stale) Mirror 11: 0xaa74396a (stale) Mirror 14: 0xaa74396a (stale) Mirror 15: 0xaa74396a (stale) Mirror 16: 0xaa74396a (stale) Mirror 17: 0xaa74396a (stale) Mirror 18: 0x1eb7dd30 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cf7: skipped 000522 1786102062.281555 write 0x03ee13 thru 0x03ffff (0x11ed bytes) 000523 1786102062.299470 mapread 0x001bbc thru 0x010891 (0xecd6 bytes) 000524 1786102062.351307 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6ff4d97e (stale) Mirror 8: 0x6ff4d97e (stale) Mirror 10: 0x6ff4d97e (stale) Mirror 11: 0x6ff4d97e (stale) Mirror 14: 0x6ff4d97e (stale) Mirror 15: 0x6ff4d97e (stale) Mirror 16: 0x6ff4d97e (stale) Mirror 18: 0x2daa53d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6ff4d97e (stale) Mirror 8: 0x6ff4d97e (stale) Mirror 10: 0x6ff4d97e (stale) Mirror 11: 0x6ff4d97e (stale) Mirror 14: 0x6ff4d97e (stale) Mirror 15: 0x6ff4d97e (stale) Mirror 16: 0x6ff4d97e (stale) Mirror 18: 0x2daa53d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000525 1786102063.702235 mapread 0x032730 thru 0x03b1ef (0x8ac0 bytes) 000527 1786102063.752023 mapread 0x01b671 thru 0x0256be (0xa04e bytes) 000528 1786102063.804123 mapwrite 0x03a693 thru 0x03d65d (0x2fcb bytes) 000529 1786102063.855968 trunc from 0x00cb97 to 0x00f75c (0x2bc6 bytes) 000530 1786102063.878285 mapwrite 0x020558 thru 0x021788 (0x1231 bytes) 000531 1786102063.909581 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6ff4d97e (stale) Mirror 8: 0x6ff4d97e (stale) Mirror 10: 0x6ff4d97e (stale) Mirror 14: 0x6ff4d97e (stale) Mirror 15: 0x6ff4d97e (stale) Mirror 16: 0x6ff4d97e (stale) Mirror 18: 0x2daa53d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x6ff4d97e (stale) Mirror 8: 0x6ff4d97e (stale) Mirror 10: 0x6ff4d97e (stale) Mirror 14: 0x6ff4d97e (stale) Mirror 15: 0x6ff4d97e (stale) Mirror 16: 0x6ff4d97e (stale) Mirror 18: 0x2daa53d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000532 1786102064.866091 mapread 0x01eac1 thru 0x02105f (0x259f bytes) 000533 1786102064.913166 punch from 0x027def to 0x02ffd5 (0x81e7 bytes) 000534 1786102064.934300 mapwrite 0x00d14c thru 0x00f52d (0x23e2 bytes) 000537 1786102065.006813 fallocate 0x031fb6 thru 0x03abf1 (0x8c3c bytes) 000538 1786102065.016087 trunc from 0x040000 to 0x020fff (0x1f000 bytes) 000540 1786102065.052981 trunc from 0x021000 to 0x014223 (0xcddc bytes) 000541 1786102065.097031 trunc from 0x00eec6 to 0x00fed6 (0x1011 bytes) 000542 1786102065.119007 fallocate 0x00ee93 thru 0x0186e1 (0x984f bytes) 000544 1786102065.144741 mapwrite 0x01d89f thru 0x025b36 (0x8298 bytes) 000545 1786102065.258730 read 0x019725 thru 0x0245fb (0xaed7 bytes) 000546 1786102065.269220 trunc from 0x01584d to 0x01cf08 (0x76bc bytes) 000547 1786102065.286231 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x25b37): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b37): Mirror 7: 0xf25d3adb (stale) Mirror 8: 0xf25d3adb (stale) Mirror 10: 0xf25d3adb (stale) Mirror 14: 0xf25d3adb (stale) Mirror 15: 0xf25d3adb (stale) Mirror 16: 0xf25d3adb (stale) Mirror 18: 0xb73cad73 (stale) CRC-32 checksum value for chunk [0, 0x25b37): Mirror 1: 0x7ad995c1 Mirror 19: 0x7ad995c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b37: skipped 000548 1786102066.638470 mapread 0x018f32 thru 0x025b36 (0xcc05 bytes) 000549 1786102066.676257 read 0x010392 thru 0x015108 (0x4d77 bytes) 000550 1786102066.684470 trunc from 0x025b37 to 0x022793 (0x33a3 bytes) 000551 1786102066.837258 mapwrite 0x011e2d thru 0x019576 (0x774a bytes) 000552 1786102066.949416 trunc from 0x016372 to 0x0164ce (0x15d bytes) 000553 1786102066.968067 trunc from 0x022794 to 0x033ae7 (0x11354 bytes) 000554 1786102067.017691 trunc from 0x033ae8 to 0x011fed (0x21afa bytes) 000555 1786102067.050506 punch from 0x0107d4 to 0x011fed (0x181a bytes) 000556 1786102067.059656 trunc from 0x011fee to 0x02d902 (0x1b915 bytes) 000557 1786102067.077195 trunc from 0x02d903 to 0x006188 (0x2777a bytes) 000559 1786102067.099091 mapwrite 0x03a94e thru 0x03aac2 (0x175 bytes) 000560 1786102067.200154 fallocate 0x0202bd thru 0x0283ae (0x80f2 bytes) 000561 1786102067.225164 MIRROR_RESYNC 9 mirrors 000562 1786102069.069950 read 0x000d45 thru 0x00e36d (0xd629 bytes) 000563 1786102069.110865 read 0x007f30 thru 0x013af5 (0xbbc6 bytes) 000564 1786102069.135418 MIRROR_RESYNC 9 mirrors 000565 1786102069.795973 mapread 0x02b4e3 thru 0x039b47 (0xe665 bytes) 000566 1786102069.799209 trunc from 0x03aac3 to 0x02c5e1 (0xe4e1 bytes) 000567 1786102069.940266 trunc from 0x00f39a to 0x018761 (0x93c8 bytes) 000568 1786102069.954952 trunc from 0x02c5e2 to 0x02faa2 (0x34c1 bytes) 000569 1786102070.021070 trunc from 0x02faa3 to 0x035e3a (0x6398 bytes) 000570 1786102070.075456 trunc from 0x00c8ad to 0x01a746 (0xde9a bytes) 000571 1786102070.095085 write 0x012577 thru 0x014d1a (0x27a4 bytes) 000572 1786102070.110108 punch from 0x027e1e to 0x02f5cc (0x77af bytes) 000573 1786102070.122227 mapwrite 0x019f66 thru 0x02010c (0x61a7 bytes) 000574 1786102070.213248 mapwrite 0x010d81 thru 0x017831 (0x6ab1 bytes) 000575 1786102070.324563 mapwrite 0x0346bc thru 0x03ffff (0xb944 bytes) 000576 1786102070.507563 write 0x03381f thru 0x03ffff (0xc7e1 bytes) 000577 1786102070.514830 read 0x0116d2 thru 0x016109 (0x4a38 bytes) 000578 1786102070.515683 read 0x00ed75 thru 0x019dac (0xb038 bytes) 000579 1786102070.516654 write 0x019f48 thru 0x023d2c (0x9de5 bytes) 000580 1786102070.559218 fallocate 0x005a1e thru 0x00cad6 (0x70b9 bytes) 000581 1786102070.578640 mapread 0x023edb thru 0x02cbce (0x8cf4 bytes) 000582 1786102070.613269 mapread 0x001422 thru 0x002df8 (0x19d7 bytes) 000583 1786102070.617397 mapread 0x00b457 thru 0x016a8d (0xb637 bytes) 000584 1786102070.619945 trunc from 0x009661 to 0x00ff22 (0x68c2 bytes) 000585 1786102070.655142 MIRROR_RESYNC 9 mirrors 000586 1786102073.444498 mapwrite 0x027b0d thru 0x02aa1d (0x2f11 bytes) 000587 1786102073.618525 trunc from 0x00bd00 to 0x0103bb (0x46bc bytes) 000588 1786102073.634427 read 0x000c54 thru 0x00e25f (0xd60c bytes) 000589 1786102073.654919 fallocate 0x03c558 thru 0x03ffff (0x3aa8 bytes) 000590 1786102073.677648 write 0x02e4d8 thru 0x036fe3 (0x8b0c bytes) 000591 1786102073.705957 trunc from 0x040000 to 0x01fc69 (0x20396 bytes) 000592 1786102073.754292 trunc from 0x01fc6a to 0x001679 (0x1e5f0 bytes) 000593 1786102073.781952 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x167a): Mirror 7: 0xb5f633d9 (stale) Mirror 8: 0xb5f633d9 (stale) Mirror 14: 0xb5f633d9 (stale) Mirror 15: 0xb5f633d9 (stale) Mirror 16: 0xb5f633d9 (stale) Mirror 18: 0xb5f633d9 (stale) Mirror 19: 0xb5f633d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x167a): Mirror 7: 0xb5f633d9 (stale) Mirror 8: 0xb5f633d9 (stale) Mirror 14: 0xb5f633d9 (stale) Mirror 15: 0xb5f633d9 (stale) Mirror 16: 0xb5f633d9 (stale) Mirror 18: 0xb5f633d9 (stale) Mirror 19: 0xb5f633d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x167a: skipped 000594 1786102074.261359 punch from 0x000707 to 0x001679 (0xf73 bytes) 000595 1786102074.274672 trunc from 0x0006c1 to 0x001679 (0xfb9 bytes) 000597 1786102074.281575 trunc from 0x00167a to 0x00f452 (0xddd9 bytes) 000598 1786102074.332977 punch from 0x00a18d to 0x00f452 (0x52c6 bytes) 000600 1786102074.350929 read 0x004f44 thru 0x00f452 (0xa50f bytes) 000601 1786102074.363305 fallocate 0x02b110 thru 0x02eecf (0x3dc0 bytes) 000602 1786102074.374509 mapwrite 0x01a9c2 thru 0x023fc2 (0x9601 bytes) 000603 1786102074.490137 trunc from 0x023fc3 to 0x00966e (0x1a954 bytes) 000604 1786102074.512359 mapwrite 0x016cc8 thru 0x021c33 (0xaf6c bytes) 000605 1786102074.565744 punch from 0x01d0c6 to 0x021c33 (0x4b6e bytes) 000606 1786102074.570699 trunc from 0x003803 to 0x00697b (0x3179 bytes) 000607 1786102074.575355 punch from 0x011665 to 0x019306 (0x7ca2 bytes) 000608 1786102074.581150 fallocate 0x02596c thru 0x033d4d (0xe3e2 bytes) 000609 1786102074.586155 write 0x0254d8 thru 0x0328bb (0xd3e4 bytes) 000610 1786102074.596456 fallocate 0x004e8d thru 0x010e68 (0xbfdc bytes) 000612 1786102074.601828 write 0x03d796 thru 0x03ed03 (0x156e bytes) 000613 1786102074.603725 read 0x0397bd thru 0x03ed03 (0x5547 bytes) 000614 1786102074.619025 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed04): Mirror 7: 0x9166434c (stale) Mirror 14: 0x9166434c (stale) Mirror 15: 0x9166434c (stale) Mirror 16: 0x9166434c (stale) Mirror 18: 0x9166434c (stale) Mirror 19: 0x9166434c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed04): Mirror 7: 0x9166434c (stale) Mirror 14: 0x9166434c (stale) Mirror 15: 0x9166434c (stale) Mirror 16: 0x9166434c (stale) Mirror 18: 0x9166434c (stale) Mirror 19: 0x9166434c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed04: skipped 000615 1786102075.261207 punch from 0x02920a to 0x032042 (0x8e39 bytes) 000616 1786102075.267946 trunc from 0x00277c to 0x00c06d (0x98f2 bytes) 000617 1786102075.273267 punch from 0x02e3ee to 0x03dc5c (0xf86f bytes) 000618 1786102075.281242 trunc from 0x03ed04 to 0x03d4c8 (0x183b bytes) 000619 1786102075.302941 punch from 0x02a340 to 0x03948d (0xf14e bytes) 000621 1786102075.310040 read 0x015a42 thru 0x0216aa (0xbc69 bytes) 000623 1786102075.338260 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 7: 0x8e4229cb (stale) Mirror 14: 0x8e4229cb (stale) Mirror 15: 0x8e4229cb (stale) Mirror 16: 0x8e4229cb (stale) Mirror 18: 0x8e4229cb (stale) Mirror 19: 0x8e4229cb (stale) CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 1: 0x684a081c Mirror 20: 0x684a081c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4c9: skipped 000624 1786102077.069041 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 7: 0x8e4229cb (stale) Mirror 14: 0x8e4229cb (stale) Mirror 15: 0x8e4229cb (stale) Mirror 16: 0x8e4229cb (stale) Mirror 18: 0x8e4229cb (stale) Mirror 19: 0x8e4229cb (stale) CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 1: 0x684a081c Mirror 20: 0x684a081c Mirror 21: 0x684a081c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4c9: skipped 000625 1786102078.779711 mapread 0x03974c thru 0x03d4c8 (0x3d7d bytes) 000626 1786102078.795434 punch from 0x0071e9 to 0x00fa13 (0x882b bytes) 000627 1786102078.843837 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 7: 0x8e4229cb (stale) Mirror 14: 0x8e4229cb (stale) Mirror 15: 0x8e4229cb (stale) Mirror 16: 0x8e4229cb (stale) Mirror 18: 0x8e4229cb (stale) Mirror 19: 0x8e4229cb (stale) Mirror 20: 0x684a081c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4c9): Mirror 7: 0x8e4229cb (stale) Mirror 14: 0x8e4229cb (stale) Mirror 15: 0x8e4229cb (stale) Mirror 16: 0x8e4229cb (stale) Mirror 18: 0x8e4229cb (stale) Mirror 19: 0x8e4229cb (stale) Mirror 20: 0x684a081c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4c9: skipped 000629 1786102079.443355 write 0x026448 thru 0x028c0c (0x27c5 bytes) 000630 1786102079.467071 mapread 0x006f8f thru 0x0129b8 (0xba2a bytes) 000631 1786102079.524411 read 0x038a60 thru 0x03d4c8 (0x4a69 bytes) 000633 1786102079.546926 read 0x0017f2 thru 0x0041f5 (0x2a04 bytes) 000634 1786102079.547726 mapread 0x035276 thru 0x03d4c8 (0x8253 bytes) 000635 1786102079.577339 punch from 0x024bc4 to 0x02ebba (0x9ff7 bytes) 000636 1786102079.595486 write 0x02852f thru 0x0317f3 (0x92c5 bytes) 000638 1786102079.624839 fallocate 0x0339b2 thru 0x03ffff (0xc64e bytes) 000639 1786102079.631843 trunc from 0x00e141 to 0x00ecaf (0xb6f bytes) 000640 1786102079.647926 read 0x038c67 thru 0x03d586 (0x4920 bytes) 000641 1786102079.649202 trunc from 0x010217 to 0x013421 (0x320b bytes) 000642 1786102079.664575 mapread 0x0294b6 thru 0x031823 (0x836e bytes) 000643 1786102079.687406 MIRROR_RESYNC 8 mirrors 000644 1786102082.519068 punch from 0x03b1d2 to 0x03ffff (0x4e2e bytes) 000645 1786102082.566686 trunc from 0x040000 to 0x003eff (0x3c100 bytes) 000646 1786102082.581573 write 0x00d7a9 thru 0x0179f0 (0xa248 bytes) 000647 1786102082.584613 fallocate 0x0317fa thru 0x03f71d (0xdf24 bytes) 000648 1786102082.607330 trunc from 0x03f71e to 0x031ea3 (0xd87a bytes) 000649 1786102082.618133 write 0x039451 thru 0x03ffff (0x6baf bytes) 000650 1786102082.619390 trunc from 0x040000 to 0x017dca (0x28235 bytes) 000651 1786102082.646684 read 0x00f698 thru 0x017dca (0x8733 bytes) 000653 1786102082.652127 punch from 0x006360 to 0x00eb29 (0x87ca bytes) 000654 1786102082.678473 fallocate 0x015aae thru 0x01cf64 (0x74b7 bytes) 000655 1786102082.685159 read 0x00ce36 thru 0x0199ec (0xcbb7 bytes) 000656 1786102082.696837 read 0x017dec thru 0x01cf64 (0x5179 bytes) 000657 1786102082.698563 trunc from 0x01cf65 to 0x026c8e (0x9d2a bytes) 000658 1786102082.723866 mapwrite 0x005255 thru 0x00c135 (0x6ee1 bytes) 000659 1786102082.768210 MIRROR_RESYNC 8 mirrors 000660 1786102085.087770 read 0x01e7ce thru 0x026c8e (0x84c1 bytes) 000662 1786102085.104298 mapread 0x024fd3 thru 0x026c8e (0x1cbc bytes) 000664 1786102085.105543 write 0x016f10 thru 0x0213cf (0xa4c0 bytes) 000665 1786102085.167804 mapread 0x01ad18 thru 0x01ed83 (0x406c bytes) 000666 1786102085.173116 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x26c8f): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c8f): Mirror 7: 0x63c90295 (stale) Mirror 14: 0x63c90295 (stale) Mirror 15: 0x63c90295 (stale) Mirror 16: 0x63c90295 (stale) Mirror 18: 0x63c90295 (stale) Mirror 19: 0x63c90295 (stale) Mirror 20: 0x63c90295 (stale) CRC-32 checksum value for chunk [0, 0x26c8f): Mirror 1: 0x3c15bfdb Mirror 21: 0x3c15bfdb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c8f: skipped 000667 1786102086.343710 mapread 0x01a6a9 thru 0x020a42 (0x639a bytes) 000668 1786102086.374204 write 0x033bea thru 0x039036 (0x544d bytes) 000669 1786102086.458989 trunc from 0x012dfd to 0x01aa52 (0x7c56 bytes) 000670 1786102086.467404 punch from 0x016c65 to 0x022bff (0xbf9b bytes) 000671 1786102086.477162 mapwrite 0x032377 thru 0x0327bc (0x446 bytes) 000672 1786102086.541369 fallocate 0x02ba7f thru 0x0366e1 (0xac63 bytes) 000673 1786102086.561877 mapwrite 0x0079d2 thru 0x007d69 (0x398 bytes) 000674 1786102086.612955 mapread 0x02f12c thru 0x034638 (0x550d bytes) 000675 1786102086.614246 mapwrite 0x00c337 thru 0x019dc4 (0xda8e bytes) 000676 1786102086.728524 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x39037): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39037): Mirror 7: 0x754a9053 (stale) Mirror 14: 0x754a9053 (stale) Mirror 15: 0x754a9053 (stale) Mirror 16: 0x754a9053 (stale) Mirror 18: 0x754a9053 (stale) Mirror 19: 0x754a9053 (stale) Mirror 20: 0x754a9053 (stale) Mirror 21: 0x4cdc81b1 (stale) CRC-32 checksum value for chunk [0, 0x39037): Mirror 1: 0x32b85fd5 Mirror 22: 0x32b85fd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39037: skipped 000677 1786102088.109780 punch from 0x00d2ac to 0x018f29 (0xbc7e bytes) 000678 1786102088.154081 trunc from 0x030741 to 0x039036 (0x88f6 bytes) 000679 1786102088.158799 trunc from 0x01375f to 0x014b6a (0x140c bytes) 000680 1786102088.165172 trunc from 0x039037 to 0x02c149 (0xceed bytes) 000681 1786102088.192245 mapread 0x02202d thru 0x029dbb (0x7d8f bytes) 000684 1786102088.224303 mapread 0x02859a thru 0x02c149 (0x3bb0 bytes) 000685 1786102088.225666 write 0x02db4c thru 0x03c7dd (0xec92 bytes) 000687 1786102088.228404 mapread 0x018c1a thru 0x023337 (0xa71e bytes) 000688 1786102088.260896 MIRROR_RESYNC 10 mirrors 000689 1786102090.921880 write 0x00ada9 thru 0x0101cb (0x5423 bytes) 000690 1786102090.981217 fallocate 0x009fce thru 0x0107ee (0x6821 bytes) 000691 1786102090.994076 trunc from 0x03c7de to 0x01f47d (0x1d360 bytes) 000693 1786102091.044072 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1f47e): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f47e): Mirror 7: 0x2abda3f6 (stale) Mirror 14: 0x2abda3f6 (stale) Mirror 15: 0x2abda3f6 (stale) Mirror 16: 0x2abda3f6 (stale) Mirror 18: 0x2abda3f6 (stale) Mirror 19: 0x2abda3f6 (stale) Mirror 20: 0x2abda3f6 (stale) Mirror 21: 0x2abda3f6 (stale) Mirror 22: 0x2abda3f6 (stale) CRC-32 checksum value for chunk [0, 0x1f47e): Mirror 1: 0x778a2141 Mirror 23: 0x778a2141 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f47e: skipped 000694 1786102092.938941 punch from 0x001e6b to 0x005412 (0x35a8 bytes) 000695 1786102092.977081 mapwrite 0x028759 thru 0x02e303 (0x5bab bytes) 000696 1786102093.040626 trunc from 0x00ca10 to 0x00e7e8 (0x1dd9 bytes) 000697 1786102093.050712 mapread 0x01a737 thru 0x0222b9 (0x7b83 bytes) 000698 1786102093.069244 trunc from 0x02e304 to 0x017cd9 (0x1662a bytes) 000699 1786102093.089820 trunc from 0x017cda to 0x039d7c (0x220a3 bytes) 000700 1786102093.120540 mapread 0x018d1d thru 0x020ae1 (0x7dc5 bytes) 000701 1786102093.152172 trunc from 0x02a068 to 0x0350b0 (0xb049 bytes) 000702 1786102093.166276 trunc from 0x01bd54 to 0x024e09 (0x90b6 bytes) 000703 1786102093.182052 fallocate 0x011d82 thru 0x016a9f (0x4d1e bytes) 000704 1786102093.193149 fallocate 0x02c346 thru 0x03790e (0xb5c9 bytes) 000705 1786102093.202583 fallocate 0x00380c thru 0x01228e (0xea83 bytes) 000706 1786102093.214834 mapread 0x02efa2 thru 0x039d7c (0xaddb bytes) 000707 1786102093.242197 fallocate 0x026e20 thru 0x0300d4 (0x92b5 bytes) 000710 1786102093.272971 fallocate 0x025142 thru 0x02fc70 (0xab2f bytes) 000711 1786102093.292856 trunc from 0x039d7d to 0x036b2b (0x3251 bytes) 000712 1786102093.307363 trunc from 0x036b2c to 0x01feb5 (0x16c76 bytes) 000713 1786102093.319888 write 0x02d9a9 thru 0x033d58 (0x63b0 bytes) 000714 1786102093.321458 write 0x034b42 thru 0x03d881 (0x8d40 bytes) 000715 1786102093.323236 trunc from 0x00e9ac to 0x0134fc (0x4b51 bytes) 000716 1786102093.327834 mapread 0x026792 thru 0x03627c (0xfaeb bytes) 000717 1786102093.339581 MIRROR_RESYNC 11 mirrors 000718 1786102097.231236 write 0x008500 thru 0x0138b1 (0xb3b2 bytes) 000719 1786102097.284029 read 0x02b293 thru 0x02e5d7 (0x3345 bytes) 000720 1786102097.292138 write 0x02b408 thru 0x02b764 (0x35d bytes) 000721 1786102097.293867 read 0x001424 thru 0x00b463 (0xa040 bytes) 000722 1786102097.309140 punch from 0x01cd39 to 0x02a708 (0xd9d0 bytes) 000724 1786102097.318089 punch from 0x036b03 to 0x036ec5 (0x3c3 bytes) 000726 1786102097.328248 read 0x003447 thru 0x004e4a (0x1a04 bytes) 000727 1786102097.329238 write 0x01e537 thru 0x027aa3 (0x956d bytes) 000728 1786102097.345278 read 0x01ef8a thru 0x01fdf1 (0xe68 bytes) 000729 1786102097.346272 read 0x003003 thru 0x01115c (0xe15a bytes) 000730 1786102097.348633 read 0x0219fe thru 0x02663c (0x4c3f bytes) 000731 1786102097.351172 mapwrite 0x001016 thru 0x002afc (0x1ae7 bytes) 000732 1786102097.370653 write 0x0277b5 thru 0x034635 (0xce81 bytes) 000733 1786102097.372768 trunc from 0x03d882 to 0x029a57 (0x13e2a bytes) 000734 1786102097.391297 trunc from 0x029a58 to 0x024395 (0x56c2 bytes) 000735 1786102097.405857 punch from 0x001f21 to 0x00b954 (0x9a34 bytes) 000736 1786102097.411147 write 0x03cb7d thru 0x03ffff (0x3483 bytes) 000737 1786102097.413002 mapwrite 0x00d373 thru 0x015507 (0x8195 bytes) 000738 1786102097.432706 trunc from 0x038874 to 0x03ffff (0x778c bytes) 000739 1786102097.438340 mapread 0x02f0cd thru 0x031e78 (0x2dac bytes) 000740 1786102097.467955 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9775893c (stale) Mirror 14: 0x9775893c (stale) Mirror 15: 0x9775893c (stale) Mirror 16: 0x9775893c (stale) Mirror 18: 0x9775893c (stale) Mirror 19: 0x9775893c (stale) Mirror 20: 0x9775893c (stale) Mirror 21: 0x9775893c (stale) Mirror 22: 0x9775893c (stale) Mirror 23: 0x9775893c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x654ce9af Mirror 24: 0x654ce9af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000741 1786102098.490967 read 0x028e6d thru 0x02b8a8 (0x2a3c bytes) 000742 1786102098.505542 punch from 0x018cbe to 0x01c2e2 (0x3625 bytes) 000743 1786102098.556035 read 0x0261e5 thru 0x033ab5 (0xd8d1 bytes) 000745 1786102098.580448 fallocate 0x02ee3d thru 0x039500 (0xa6c4 bytes) 000746 1786102098.592152 trunc from 0x040000 to 0x037ad7 (0x8528 bytes) 000747 1786102098.670464 read 0x022cbf thru 0x024cfa (0x203c bytes) 000749 1786102098.674944 trunc from 0x008cb8 to 0x016700 (0xda49 bytes) 000750 1786102098.684723 punch from 0x02d095 to 0x037ad7 (0xaa43 bytes) 000751 1786102098.691744 write 0x007aad thru 0x0084e1 (0xa35 bytes) 000752 1786102098.705774 mapwrite 0x01f681 thru 0x022db9 (0x3739 bytes) 000753 1786102098.751692 MIRROR_RESYNC 12 mirrors 000754 1786102102.363047 mapwrite 0x03f3ed thru 0x03ffff (0xc13 bytes) 000755 1786102102.491781 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb5a4720b (stale) Mirror 14: 0xb5a4720b (stale) Mirror 15: 0xb5a4720b (stale) Mirror 16: 0xb5a4720b (stale) Mirror 18: 0xb5a4720b (stale) Mirror 19: 0xb5a4720b (stale) Mirror 20: 0xb5a4720b (stale) Mirror 21: 0xb5a4720b (stale) Mirror 22: 0xb5a4720b (stale) Mirror 23: 0xb5a4720b (stale) Mirror 24: 0xb5a4720b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2202199c Mirror 25: 0x2202199c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000758 1786102103.857553 trunc from 0x040000 to 0x0315e2 (0xea1d bytes) 000759 1786102103.932193 write 0x01955a thru 0x01ad8d (0x1834 bytes) 000760 1786102103.943197 trunc from 0x0315e3 to 0x0346f9 (0x3117 bytes) 000761 1786102103.966287 mapwrite 0x01887a thru 0x01c082 (0x3809 bytes) 000762 1786102104.010802 mapwrite 0x00e635 thru 0x011331 (0x2cfd bytes) 000764 1786102104.038848 write 0x01b164 thru 0x02036a (0x5207 bytes) 000765 1786102104.041297 write 0x0080cd thru 0x010439 (0x836d bytes) 000766 1786102104.043233 mapwrite 0x027be4 thru 0x02c7e9 (0x4c06 bytes) 000767 1786102104.076867 mapwrite 0x026b99 thru 0x027f06 (0x136e bytes) 000768 1786102104.097365 mapwrite 0x014639 thru 0x01d76a (0x9132 bytes) 000769 1786102104.124289 trunc from 0x026bb3 to 0x033d6d (0xd1bb bytes) 000770 1786102104.132068 mapwrite 0x02f196 thru 0x033905 (0x4770 bytes) 000771 1786102104.184258 trunc from 0x0346fa to 0x033733 (0xfc6 bytes) 000772 1786102104.210607 read 0x024e97 thru 0x027b37 (0x2ca1 bytes) 000773 1786102104.220097 mapwrite 0x03ebb0 thru 0x03ffff (0x1450 bytes) 000774 1786102104.299894 trunc from 0x00ddd6 to 0x015e6c (0x8097 bytes) 000776 1786102104.319840 MIRROR_RESYNC 13 mirrors 000777 1786102109.413843 fallocate 0x012817 thru 0x019f25 (0x770f bytes) 000778 1786102109.506158 mapread 0x02b325 thru 0x02c01a (0xcf6 bytes) 000779 1786102109.526506 MIRROR_RESYNC 13 mirrors 000780 1786102113.854851 trunc from 0x023495 to 0x02c129 (0x8c95 bytes) 000781 1786102113.903932 punch from 0x01148a to 0x017a1b (0x6592 bytes) 000782 1786102113.921534 mapread 0x0371a2 thru 0x03ffff (0x8e5e bytes) 000783 1786102113.951195 mapread 0x03c54f thru 0x03ffff (0x3ab1 bytes) 000784 1786102113.957879 mapwrite 0x032ced thru 0x035917 (0x2c2b bytes) 000786 1786102113.985087 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x24243d6e (stale) Mirror 14: 0x24243d6e (stale) Mirror 15: 0x24243d6e (stale) Mirror 16: 0x24243d6e (stale) Mirror 18: 0x24243d6e (stale) Mirror 19: 0x24243d6e (stale) Mirror 20: 0x24243d6e (stale) Mirror 21: 0x24243d6e (stale) Mirror 22: 0x24243d6e (stale) Mirror 23: 0x24243d6e (stale) Mirror 24: 0x24243d6e (stale) Mirror 25: 0x24243d6e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd446d02e Mirror 26: 0xd446d02e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000787 1786102115.520843 fallocate 0x03e4d5 thru 0x03ffff (0x1b2b bytes) 000788 1786102115.607067 trunc from 0x0389f0 to 0x03d673 (0x4c84 bytes) 000790 1786102115.613201 write 0x03f9d9 thru 0x03fa4c (0x74 bytes) 000791 1786102115.626142 mapwrite 0x00aa74 thru 0x00d48b (0x2a18 bytes) 000792 1786102115.695754 punch from 0x0034c3 to 0x011254 (0xdd92 bytes) 000793 1786102115.706126 read 0x018bec thru 0x020d75 (0x818a bytes) 000794 1786102115.738181 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x24243d6e (stale) Mirror 14: 0x24243d6e (stale) Mirror 16: 0x24243d6e (stale) Mirror 18: 0x24243d6e (stale) Mirror 19: 0x24243d6e (stale) Mirror 20: 0x24243d6e (stale) Mirror 21: 0x24243d6e (stale) Mirror 22: 0x24243d6e (stale) Mirror 23: 0x24243d6e (stale) Mirror 24: 0x24243d6e (stale) Mirror 25: 0x24243d6e (stale) Mirror 26: 0xd446d02e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x24243d6e (stale) Mirror 14: 0x24243d6e (stale) Mirror 16: 0x24243d6e (stale) Mirror 18: 0x24243d6e (stale) Mirror 19: 0x24243d6e (stale) Mirror 20: 0x24243d6e (stale) Mirror 21: 0x24243d6e (stale) Mirror 22: 0x24243d6e (stale) Mirror 23: 0x24243d6e (stale) Mirror 24: 0x24243d6e (stale) Mirror 25: 0x24243d6e (stale) Mirror 26: 0xd446d02e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000796 1786102116.570201 write 0x0124be thru 0x0149af (0x24f2 bytes) 000797 1786102116.582753 trunc from 0x00191a to 0x0113b9 (0xfaa0 bytes) 000798 1786102116.587408 fallocate 0x000aec thru 0x00bbea (0xb0ff bytes) 000799 1786102116.599675 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x24243d6e (stale) Mirror 14: 0x24243d6e (stale) Mirror 16: 0x24243d6e (stale) Mirror 18: 0x24243d6e (stale) Mirror 19: 0x24243d6e (stale) Mirror 20: 0x24243d6e (stale) Mirror 21: 0x24243d6e (stale) Mirror 22: 0x24243d6e (stale) Mirror 23: 0x24243d6e (stale) Mirror 25: 0x24243d6e (stale) Mirror 26: 0xd446d02e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x24243d6e (stale) Mirror 14: 0x24243d6e (stale) Mirror 16: 0x24243d6e (stale) Mirror 18: 0x24243d6e (stale) Mirror 19: 0x24243d6e (stale) Mirror 20: 0x24243d6e (stale) Mirror 21: 0x24243d6e (stale) Mirror 22: 0x24243d6e (stale) Mirror 23: 0x24243d6e (stale) Mirror 25: 0x24243d6e (stale) Mirror 26: 0xd446d02e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000800 1786102117.305080 fallocate 0x000e1e thru 0x005d61 (0x4f44 bytes) 000802 1786102117.317620 MIRROR_RESYNC 12 mirrors 000803 1786102120.892794 trunc from 0x040000 to 0x01c9dc (0x23623 bytes) 000804 1786102120.969403 read 0x01c9ce thru 0x01c9dc (0xf bytes) 000805 1786102120.989488 MIRROR_RESYNC 12 mirrors 000806 1786102123.731640 read 0x00bad5 thru 0x01b8a8 (0xfdd4 bytes) 000807 1786102123.744430 mapwrite 0x01d2b0 thru 0x02c65e (0xf3af bytes) 000808 1786102123.864654 mapwrite 0x014b74 thru 0x021d47 (0xd1d4 bytes) 000809 1786102123.908995 mapwrite 0x032a66 thru 0x0394aa (0x6a45 bytes) 000810 1786102123.969466 fallocate 0x00e785 thru 0x0190c9 (0xa945 bytes) 000811 1786102123.979485 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x394ab): Mirror 14: 0x17cdf156 (stale) Mirror 16: 0x17cdf156 (stale) Mirror 18: 0x17cdf156 (stale) Mirror 19: 0x17cdf156 (stale) Mirror 20: 0x17cdf156 (stale) Mirror 21: 0x17cdf156 (stale) Mirror 22: 0x17cdf156 (stale) Mirror 23: 0x17cdf156 (stale) Mirror 25: 0x17cdf156 (stale) Mirror 26: 0x17cdf156 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394ab): Mirror 14: 0x17cdf156 (stale) Mirror 16: 0x17cdf156 (stale) Mirror 18: 0x17cdf156 (stale) Mirror 19: 0x17cdf156 (stale) Mirror 20: 0x17cdf156 (stale) Mirror 21: 0x17cdf156 (stale) Mirror 22: 0x17cdf156 (stale) Mirror 23: 0x17cdf156 (stale) Mirror 25: 0x17cdf156 (stale) Mirror 26: 0x17cdf156 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394ab: skipped 000812 1786102124.648085 mapwrite 0x0171fc thru 0x0205a3 (0x93a8 bytes) 000813 1786102124.685686 write 0x0066bc thru 0x007ff0 (0x1935 bytes) 000814 1786102124.695831 mapwrite 0x00c3b0 thru 0x014236 (0x7e87 bytes) 000815 1786102124.723003 read 0x02d079 thru 0x033d1d (0x6ca5 bytes) 000816 1786102124.731437 mapwrite 0x019531 thru 0x01fe86 (0x6956 bytes) 000817 1786102124.748668 write 0x0148b7 thru 0x022003 (0xd74d bytes) 000818 1786102124.758140 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x394ab): Mirror 14: 0x17cdf156 (stale) Mirror 16: 0x17cdf156 (stale) Mirror 18: 0x17cdf156 (stale) Mirror 19: 0x17cdf156 (stale) Mirror 21: 0x17cdf156 (stale) Mirror 22: 0x17cdf156 (stale) Mirror 23: 0x17cdf156 (stale) Mirror 25: 0x17cdf156 (stale) Mirror 26: 0x17cdf156 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394ab): Mirror 14: 0x17cdf156 (stale) Mirror 16: 0x17cdf156 (stale) Mirror 18: 0x17cdf156 (stale) Mirror 19: 0x17cdf156 (stale) Mirror 21: 0x17cdf156 (stale) Mirror 22: 0x17cdf156 (stale) Mirror 23: 0x17cdf156 (stale) Mirror 25: 0x17cdf156 (stale) Mirror 26: 0x17cdf156 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394ab: skipped 000819 1786102125.335825 write 0x02bb1d thru 0x036960 (0xae44 bytes) 000820 1786102125.375276 fallocate 0x00984e thru 0x00d676 (0x3e29 bytes) 000821 1786102125.397976 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x394ab): Mirror 14: 0x17cdf156 (stale) Mirror 16: 0x17cdf156 (stale) Mirror 19: 0x17cdf156 (stale) Mirror 21: 0x17cdf156 (stale) Mirror 22: 0x17cdf156 (stale) Mirror 23: 0x17cdf156 (stale) Mirror 25: 0x17cdf156 (stale) Mirror 26: 0x17cdf156 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394ab): Mirror 14: 0x17cdf156 (stale) Mirror 16: 0x17cdf156 (stale) Mirror 19: 0x17cdf156 (stale) Mirror 21: 0x17cdf156 (stale) Mirror 22: 0x17cdf156 (stale) Mirror 23: 0x17cdf156 (stale) Mirror 25: 0x17cdf156 (stale) Mirror 26: 0x17cdf156 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394ab: skipped 000822 1786102125.986859 fallocate 0x02d9b2 thru 0x037608 (0x9c57 bytes) 000823 1786102125.993572 write 0x018620 thru 0x01cee6 (0x48c7 bytes) 000824 1786102126.013777 read 0x03475f thru 0x0394aa (0x4d4c bytes) 000825 1786102126.033647 punch from 0x016937 to 0x01a730 (0x3dfa bytes) 000826 1786102126.047766 punch from 0x011fcc to 0x0178a3 (0x58d8 bytes) 000828 1786102126.054795 trunc from 0x00529a to 0x00a910 (0x5677 bytes) 000829 1786102126.066893 mapwrite 0x02406e thru 0x02ab2e (0x6ac1 bytes) 000830 1786102126.122539 read 0x025075 thru 0x031831 (0xc7bd bytes) 000831 1786102126.152819 trunc from 0x0394ab to 0x03f0d3 (0x5c29 bytes) 000832 1786102126.172766 trunc from 0x03f0d4 to 0x02eacc (0x10607 bytes) 000833 1786102126.209767 trunc from 0x02db6d to 0x02eacc (0xf60 bytes) 000834 1786102126.222716 fallocate 0x005279 thru 0x007901 (0x2689 bytes) 000835 1786102126.232048 read 0x029b14 thru 0x02eacc (0x4fb9 bytes) 000836 1786102126.246692 trunc from 0x02eacd to 0x0146f6 (0x1a3d6 bytes) 000837 1786102126.267842 read 0x001fb4 thru 0x00b926 (0x9973 bytes) 000838 1786102126.269381 read 0x0099da thru 0x0146f6 (0xad1d bytes) 000839 1786102126.271058 fallocate 0x003358 thru 0x00fae8 (0xc791 bytes) 000840 1786102126.281104 punch from 0x004156 to 0x0078d0 (0x377b bytes) 000841 1786102126.286755 fallocate 0x03d35d thru 0x03ffff (0x2ca3 bytes) 000842 1786102126.313067 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x146f7): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x146f7): Mirror 14: 0xfcec3d11 (stale) Mirror 16: 0xfcec3d11 (stale) Mirror 19: 0xfcec3d11 (stale) Mirror 21: 0xfcec3d11 (stale) Mirror 22: 0xfcec3d11 (stale) Mirror 23: 0xfcec3d11 (stale) Mirror 25: 0xfcec3d11 (stale) Mirror 26: 0xfcec3d11 (stale) CRC-32 checksum value for chunk [0, 0x146f7): Mirror 1: 0xb227b3cb Mirror 27: 0xb227b3cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x146f7: skipped 000843 1786102127.540110 read 0x00118d thru 0x006daa (0x5c1e bytes) 000844 1786102127.569443 mapread 0x00fb82 thru 0x0146f6 (0x4b75 bytes) 000845 1786102127.627875 trunc from 0x0146f7 to 0x034028 (0x1f932 bytes) 000847 1786102127.720173 mapread 0x0097d5 thru 0x00bb8b (0x23b7 bytes) 000849 1786102127.732083 mapread 0x01d120 thru 0x02d058 (0xff39 bytes) 000850 1786102127.760120 read 0x02b454 thru 0x034028 (0x8bd5 bytes) 000851 1786102127.778022 trunc from 0x034029 to 0x023a83 (0x105a5 bytes) 000852 1786102127.808825 write 0x01d54f thru 0x022d78 (0x582a bytes) 000853 1786102127.811727 punch from 0x007561 to 0x012e1c (0xb8bc bytes) 000854 1786102127.818706 mapwrite 0x0303e1 thru 0x03d200 (0xce20 bytes) 000855 1786102127.959988 punch from 0x00fd42 to 0x01d786 (0xda45 bytes) 000856 1786102127.983781 read 0x00bc2a thru 0x00e5c3 (0x299a bytes) 000857 1786102128.012992 fallocate 0x01c56a thru 0x02abb7 (0xe64e bytes) 000859 1786102128.031893 read 0x036023 thru 0x03d200 (0x71de bytes) 000860 1786102128.032556 read 0x035d8b thru 0x038b30 (0x2da6 bytes) 000861 1786102128.033184 punch from 0x024eec to 0x027062 (0x2177 bytes) 000862 1786102128.045870 trunc from 0x034a1c to 0x03d200 (0x87e5 bytes) 000863 1786102128.066528 read 0x01f791 thru 0x0250bc (0x592c bytes) 000864 1786102128.090845 punch from 0x0041fa to 0x01398b (0xf792 bytes) 000865 1786102128.101920 mapread 0x02d077 thru 0x02e245 (0x11cf bytes) 000867 1786102128.110434 fallocate 0x0112d5 thru 0x011d0f (0xa3b bytes) 000868 1786102128.121875 mapwrite 0x0234b5 thru 0x025f72 (0x2abe bytes) 000869 1786102128.156159 read 0x013d51 thru 0x021232 (0xd4e2 bytes) 000870 1786102128.169921 write 0x015081 thru 0x015555 (0x4d5 bytes) 000871 1786102128.170983 mapread 0x0286c6 thru 0x0355be (0xcef9 bytes) 000872 1786102128.191476 MIRROR_RESYNC 10 mirrors 000873 1786102131.669364 write 0x027334 thru 0x036df1 (0xfabe bytes) 000874 1786102131.718186 mapwrite 0x034bd1 thru 0x03fd0e (0xb13e bytes) 000875 1786102131.783751 mapwrite 0x029864 thru 0x02b64e (0x1deb bytes) 000876 1786102131.801791 punch from 0x039be7 to 0x03d6d5 (0x3aef bytes) 000877 1786102131.807118 write 0x021eca thru 0x02d634 (0xb76b bytes) 000878 1786102131.824976 mapwrite 0x03a6a0 thru 0x03ffff (0x5960 bytes) 000879 1786102131.915181 trunc from 0x040000 to 0x03f07c (0xf83 bytes) 000881 1786102131.942089 punch from 0x009b52 to 0x016c00 (0xd0af bytes) 000883 1786102131.952871 fallocate 0x020204 thru 0x024cfd (0x4afa bytes) 000884 1786102131.963763 read 0x0223fc thru 0x027e80 (0x5a85 bytes) 000885 1786102131.970948 mapwrite 0x03a9d3 thru 0x03ffff (0x562d bytes) 000887 1786102132.016172 write 0x02d127 thru 0x03941b (0xc2f5 bytes) 000888 1786102132.023266 mapwrite 0x022994 thru 0x02a6e7 (0x7d54 bytes) 000889 1786102132.080771 write 0x006b47 thru 0x0075df (0xa99 bytes) 000890 1786102132.107010 mapwrite 0x034256 thru 0x03ffff (0xbdaa bytes) 000891 1786102132.146885 mapwrite 0x00fc37 thru 0x016e92 (0x725c bytes) 000892 1786102132.190819 trunc from 0x039090 to 0x039e31 (0xda2 bytes) 000893 1786102132.196608 punch from 0x022c9d to 0x022d04 (0x68 bytes) 000894 1786102132.203524 trunc from 0x040000 to 0x0107f8 (0x2f807 bytes) 000895 1786102132.226244 mapwrite 0x02cdb7 thru 0x033a93 (0x6cdd bytes) 000896 1786102132.289128 fallocate 0x002707 thru 0x00773c (0x5036 bytes) 000897 1786102132.298350 punch from 0x0098ea to 0x017273 (0xd98a bytes) 000898 1786102132.304789 read 0x00d421 thru 0x01259d (0x517d bytes) 000899 1786102132.340107 write 0x032d8a thru 0x039fae (0x7225 bytes) 000900 1786102132.342995 read 0x018eea thru 0x0272c3 (0xe3da bytes) 000901 1786102132.344052 write 0x03f591 thru 0x03ffff (0xa6f bytes) 000902 1786102132.346408 read 0x00a75f thru 0x017393 (0xcc35 bytes) 000903 1786102132.351459 MIRROR_RESYNC 10 mirrors 000904 1786102134.886109 trunc from 0x040000 to 0x00651f (0x39ae0 bytes) 000906 1786102134.926284 mapwrite 0x02e392 thru 0x038795 (0xa404 bytes) 000907 1786102135.010122 trunc from 0x006096 to 0x014aff (0xea6a bytes) 000908 1786102135.017217 read 0x00c9d2 thru 0x018bc8 (0xc1f7 bytes) 000909 1786102135.029949 write 0x0114c1 thru 0x011b30 (0x670 bytes) 000910 1786102135.032172 fallocate 0x02db04 thru 0x0358e8 (0x7de5 bytes) 000911 1786102135.041971 trunc from 0x038796 to 0x012af3 (0x25ca2 bytes) 000912 1786102135.077092 MIRROR_RESYNC 10 mirrors 000913 1786102137.520443 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) 000914 1786102137.880160 fallocate 0x00e7e2 thru 0x00fc90 (0x14af bytes) 000915 1786102137.928617 read 0x00e84a thru 0x012af3 (0x42aa bytes) 000916 1786102137.938934 read 0x00e095 thru 0x012af3 (0x4a5f bytes) 000917 1786102137.940088 mapread 0x01241a thru 0x012af3 (0x6da bytes) 000918 1786102137.942072 punch from 0x001d11 to 0x010b80 (0xee70 bytes) 000919 1786102137.948543 punch from 0x00c00f to 0x012af3 (0x6ae5 bytes) 000920 1786102137.963673 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x12af4): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12af4): Mirror 14: 0x30ce2438 (stale) Mirror 16: 0x30ce2438 (stale) Mirror 19: 0x30ce2438 (stale) Mirror 21: 0x30ce2438 (stale) Mirror 22: 0x30ce2438 (stale) Mirror 23: 0x30ce2438 (stale) Mirror 26: 0x30ce2438 (stale) Mirror 27: 0x30ce2438 (stale) CRC-32 checksum value for chunk [0, 0x12af4): Mirror 1: 0xe9c023f2 Mirror 28: 0xe9c023f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12af4: skipped 000921 1786102139.238892 punch from 0x00a44e to 0x010b7b (0x672e bytes) 000922 1786102139.311724 mapwrite 0x03dd56 thru 0x03ffff (0x22aa bytes) 000923 1786102139.372402 fallocate 0x004a9d thru 0x005abd (0x1021 bytes) 000924 1786102139.378640 write 0x01806f thru 0x019ef7 (0x1e89 bytes) 000925 1786102139.389369 mapread 0x0311e2 thru 0x03199d (0x7bc bytes) 000926 1786102139.408890 MIRROR_RESYNC 10 mirrors 000927 1786102142.642890 punch from 0x02ffdb to 0x0335ea (0x3610 bytes) 000928 1786102142.724163 punch from 0x003b89 to 0x00e32e (0xa7a6 bytes) 000929 1786102142.750571 trunc from 0x017070 to 0x01bc03 (0x4b94 bytes) 000930 1786102142.785873 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xb06dea44 (stale) Mirror 16: 0xb06dea44 (stale) Mirror 19: 0xb06dea44 (stale) Mirror 21: 0xb06dea44 (stale) Mirror 22: 0xb06dea44 (stale) Mirror 23: 0xb06dea44 (stale) Mirror 26: 0xb06dea44 (stale) Mirror 27: 0xb06dea44 (stale) Mirror 28: 0xb06dea44 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd94c3b51 Mirror 29: 0xd94c3b51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000931 1786102144.747941 write 0x02c422 thru 0x031361 (0x4f40 bytes) 000932 1786102144.870250 trunc from 0x00ea63 to 0x018174 (0x9712 bytes) 000933 1786102144.898299 read 0x02ddde thru 0x036e9b (0x90be bytes) 000934 1786102144.920929 mapread 0x029062 thru 0x02b975 (0x2914 bytes) 000935 1786102144.963844 trunc from 0x040000 to 0x02630b (0x19cf4 bytes) 000936 1786102145.015841 trunc from 0x02630c to 0x033d43 (0xda38 bytes) 000937 1786102145.052947 read 0x02ab58 thru 0x02ac0e (0xb7 bytes) 000939 1786102145.083907 mapwrite 0x014950 thru 0x021fec (0xd69d bytes) 000940 1786102145.121595 trunc from 0x033d44 to 0x030fc7 (0x2d7c bytes) 000941 1786102145.144949 mapwrite 0x00a62e thru 0x013e10 (0x97e3 bytes) 000943 1786102145.183801 mapread 0x01464a thru 0x021033 (0xc9ea bytes) 000944 1786102145.188121 fallocate 0x012cde thru 0x020b50 (0xde73 bytes) 000945 1786102145.194227 trunc from 0x030fc8 to 0x03eb17 (0xdb50 bytes) 000946 1786102145.216408 MIRROR_RESYNC 11 mirrors 000947 1786102147.729870 trunc from 0x03eb18 to 0x00b1e5 (0x33932 bytes) 000948 1786102147.776150 fallocate 0x0036be thru 0x004185 (0xac8 bytes) 000949 1786102147.782131 trunc from 0x00b1e6 to 0x02aa57 (0x1f872 bytes) 000950 1786102147.800274 mapread 0x01a05f thru 0x01a09f (0x41 bytes) 000951 1786102147.807027 fallocate 0x02b3bb thru 0x02da10 (0x2656 bytes) 000952 1786102147.812782 mapwrite 0x02477b thru 0x02de35 (0x96bb bytes) 000954 1786102147.862223 mapread 0x010ea1 thru 0x01dc0d (0xcd6d bytes) 000955 1786102147.878794 write 0x00bb27 thru 0x01b858 (0xfd32 bytes) 000956 1786102147.881368 mapwrite 0x00d5e4 thru 0x0115c0 (0x3fdd bytes) 000957 1786102147.901719 trunc from 0x006a43 to 0x00e045 (0x7603 bytes) 000958 1786102147.906887 mapread 0x02bc0e thru 0x02de35 (0x2228 bytes) 000959 1786102147.909282 mapwrite 0x03e7e8 thru 0x03ffff (0x1818 bytes) 000960 1786102147.995606 write 0x00bd37 thru 0x01bd09 (0xffd3 bytes) 000961 1786102148.007714 mapread 0x02e8ea thru 0x037e5e (0x9575 bytes) 000962 1786102148.020617 fallocate 0x01b5e0 thru 0x026b74 (0xb595 bytes) 000963 1786102148.030337 fallocate 0x0163ca thru 0x01a1f5 (0x3e2c bytes) 000964 1786102148.048506 mapread 0x00a710 thru 0x00e8a2 (0x4193 bytes) 000965 1786102148.077170 write 0x0172ad thru 0x019b79 (0x28cd bytes) 000966 1786102148.079238 mapwrite 0x00609c thru 0x006964 (0x8c9 bytes) 000968 1786102148.112221 fallocate 0x01f8f7 thru 0x02ec61 (0xf36b bytes) 000969 1786102148.118698 fallocate 0x01d7eb thru 0x01e5d1 (0xde7 bytes) 000970 1786102148.143738 mapread 0x02235f thru 0x026e0a (0x4aac bytes) 000971 1786102148.150431 punch from 0x002570 to 0x00876c (0x61fd bytes) 000972 1786102148.166254 fallocate 0x02aaaf thru 0x02f5ba (0x4b0c bytes) 000973 1786102148.184661 read 0x012217 thru 0x0135ad (0x1397 bytes) 000974 1786102148.185256 mapwrite 0x013e0c thru 0x01535b (0x1550 bytes) 000975 1786102148.212304 mapwrite 0x03cc93 thru 0x03ffff (0x336d bytes) 000976 1786102148.239756 read 0x019481 thru 0x01a112 (0xc92 bytes) 000977 1786102148.266752 MIRROR_RESYNC 11 mirrors 000978 1786102151.958702 read 0x027183 thru 0x02fa95 (0x8913 bytes) 000979 1786102151.972665 MIRROR_RESYNC 11 mirrors 000980 1786102152.670068 fallocate 0x006400 thru 0x010e29 (0xaa2a bytes) 000981 1786102152.702722 write 0x0076e3 thru 0x0114db (0x9df9 bytes) 000982 1786102152.715794 fallocate 0x00baf1 thru 0x01ab74 (0xf084 bytes) 000983 1786102152.722045 mapread 0x015fce thru 0x0221ad (0xc1e0 bytes) 000984 1786102152.760528 mapread 0x019e0e thru 0x0288f7 (0xeaea bytes) 000986 1786102152.763619 trunc from 0x000a03 to 0x00e9ce (0xdfcc bytes) 000987 1786102152.785054 mapwrite 0x0051e1 thru 0x007756 (0x2576 bytes) 000988 1786102152.825696 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2cc6a6ab (stale) Mirror 16: 0x2cc6a6ab (stale) Mirror 19: 0x2cc6a6ab (stale) Mirror 21: 0x2cc6a6ab (stale) Mirror 22: 0x2cc6a6ab (stale) Mirror 26: 0x2cc6a6ab (stale) Mirror 27: 0x2cc6a6ab (stale) Mirror 28: 0x2cc6a6ab (stale) Mirror 29: 0x2cc6a6ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2cc6a6ab (stale) Mirror 16: 0x2cc6a6ab (stale) Mirror 19: 0x2cc6a6ab (stale) Mirror 21: 0x2cc6a6ab (stale) Mirror 22: 0x2cc6a6ab (stale) Mirror 26: 0x2cc6a6ab (stale) Mirror 27: 0x2cc6a6ab (stale) Mirror 28: 0x2cc6a6ab (stale) Mirror 29: 0x2cc6a6ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1786102153.658242 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2cc6a6ab (stale) Mirror 16: 0x2cc6a6ab (stale) Mirror 19: 0x2cc6a6ab (stale) Mirror 21: 0x2cc6a6ab (stale) Mirror 22: 0x2cc6a6ab (stale) Mirror 26: 0x2cc6a6ab (stale) Mirror 27: 0x2cc6a6ab (stale) Mirror 29: 0x2cc6a6ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2cc6a6ab (stale) Mirror 16: 0x2cc6a6ab (stale) Mirror 19: 0x2cc6a6ab (stale) Mirror 21: 0x2cc6a6ab (stale) Mirror 22: 0x2cc6a6ab (stale) Mirror 26: 0x2cc6a6ab (stale) Mirror 27: 0x2cc6a6ab (stale) Mirror 29: 0x2cc6a6ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000990 1786102154.222518 trunc from 0x040000 to 0x0115cb (0x2ea34 bytes) 000991 1786102154.273632 MIRROR_RESYNC 9 mirrors 000992 1786102155.992199 punch from 0x00caff to 0x010c41 (0x4143 bytes) 000993 1786102156.036979 read 0x0015f8 thru 0x005171 (0x3b7a bytes) 000994 1786102156.046713 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x44:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x115cc): Mirror 14: 0xf8840214 (stale) Mirror 16: 0xf8840214 (stale) Mirror 21: 0xf8840214 (stale) Mirror 22: 0xf8840214 (stale) Mirror 26: 0xf8840214 (stale) Mirror 27: 0xf8840214 (stale) Mirror 29: 0xf8840214 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x115cc): Mirror 14: 0xf8840214 (stale) Mirror 16: 0xf8840214 (stale) Mirror 21: 0xf8840214 (stale) Mirror 22: 0xf8840214 (stale) Mirror 26: 0xf8840214 (stale) Mirror 27: 0xf8840214 (stale) Mirror 29: 0xf8840214 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x115cc: skipped 000995 1786102156.509782 trunc from 0x007a30 to 0x0115cb (0x9b9c bytes) 000996 1786102156.519102 mapwrite 0x034283 thru 0x03e1bf (0x9f3d bytes) 000997 1786102156.635156 trunc from 0x03df3f to 0x03e1bf (0x281 bytes) 000998 1786102156.642948 write 0x0311b3 thru 0x03caaf (0xb8fd bytes) 000999 1786102156.646893 write 0x0316e4 thru 0x03f1d0 (0xdaed bytes) 001000 1786102156.649193 trunc from 0x03167d to 0x03b36e (0x9cf2 bytes) All operations completed A-OK! PASS 70a (152s) == sanity-flr test 71: check mirror extend parent fid ==== 07:29:21 (1786102161) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1ed:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 07:29:31 (1786102171) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218041 s, 9.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130694 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x360:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x360:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x35f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 07:29:38 (1786102178) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3177472 .. Extending file size to 1016320 .. failed Extending file size to 7025376 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7219904 .. Extending file size to 8303744 .. Extending file size to 6637920 .. Extending file size to 7584704 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8352608 .. done Extending file size to 9124288 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8741056 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61516 61517 61518 61520 61523 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) 192.168.206.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:30:59 (1786102259) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 7229248 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1749216 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4396928 .. failed Extending file size to 5314944 .. Extending file size to 8065568 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6435936 .. Extending file size to 8364864 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8080928 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8584480 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9242912 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8837984 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9025984 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63077 63079 63081 63083 63084 done 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) 192.168.206.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:32:21 (1786102341) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2540 1268764 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1269188 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24556 3582464 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18836 3588184 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43392 7170648 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2540 1268764 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2116 1269188 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24556 3582464 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18836 3588184 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43392 7170648 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129504 s, 16.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161883 s, 13.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf5c3c952 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:32:39 (1786102359) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148522 s, 14.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 07:32:46 (1786102366) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1fc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x376:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x376:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x377:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x374:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:32:52 (1786102372) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106557 s, 19.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:33:01 (1786102381) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15833 s, 13.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5f:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:33:11 (1786102391) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161328 s, 13.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x60:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x60:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x61:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:33:46 (1786102426) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg654-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588859 s, 17.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x378:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x376:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x375:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x378:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x376:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x375:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x379:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:34:39 (1786102479) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg654-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x377:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x378:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161433 s, 13.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x377:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x378:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x377:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x378:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:35:25 (1786102525) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg654-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg654-server: Pool lustre.test_204f created Adding targets to pool oleg654-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg654-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11114 s, 18.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x379:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x379:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg654-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg654-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg654-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:36:38 (1786102598) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:36:46 (1786102606) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:36:53 (1786102613) oleg654-server: Pool lustre.test_206 created Adding targets to pool oleg654-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg654-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg654-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg654-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg654-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:37:21 (1786102641) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586223 s, 17.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407638 s, 25.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1d22552c Mirror 3: 0x1d22552c PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:37:32 (1786102652) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:37:35 (1786102655) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:37:38 (1786102658) Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0549897 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1d60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1630000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1720000 Read this (4 bytes): BBBB 192.168.206.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg654-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:38:14 (1786102694) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0246908 s, 332 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:38:21 (1786102701) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464305 s, 22.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x21b lmm_fid: [0x200000405:0x21b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x21b lmm_fid: [0x200000405:0x21b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x385:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611675 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-07 07:38:22.000000000 -0400 Modify: 2026-08-07 07:38:22.000000000 -0400 Change: 2026-08-07 07:38:23.000000000 -0400 Birth: 2026-08-07 07:38:22.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:38:28 (1786102708) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379182 s, 27.7 MB/s oleg654-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg654-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg654-server: O/280000401/d4/384: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:38:49 (1786102729) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.177612 s, 231 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0331913 s, 123 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 7 07:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x220:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 7 07:38 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:38:56 (1786102736) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:39:03 (1786102743) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1705 sec =========== 07:39:11 (1786102751) === sanity-flr: start cleanup 07:39:12 (1786102752) === === sanity-flr: finish cleanup 07:39:20 (1786102760) ===