-----============= acceptance-small: sanity-flr ============----- Sat Aug 15 20:02:10 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg203-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:02:17 (1786838537) === oleg203-client.virtnet: executing check_config_client /mnt/lustre oleg203-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg203-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ed4d0a92000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ed4d0a92000.idle_timeout=debug disable quota as required oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:02:32 (1786838552) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8245] == sanity-flr test 0a: lfs mirror create with -N option == 20:02:33 (1786838553) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:02:39 (1786838559) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:02:41 (1786838561) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:02:42 (1786838562) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:02:50 (1786838570) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:02:52 (1786838572) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 20:02:53 (1786838573) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ed4d0a92000.stats=clear osc.lustre-OST0001-osc-ffff9ed4d0a92000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:02:56 (1786838576) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:03:00 (1786838580) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:03:03 (1786838583) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:03:04 (1786838584) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:03:07 (1786838587) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 20:03:11 (1786838591) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:03:13 (1786838593) PASS 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:03:16 (1786838596) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:03:19 (1786838599) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:03:23 (1786838603) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:03:25 (1786838605) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.439115 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313603 s, 33.4 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:03:29 (1786838609) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.0889524 s, 82.5 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:03:32 (1786838612) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:03:37 (1786838617) Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (30s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:04:07 (1786838647) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.120482 s, 52.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (17s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:04:24 (1786838664) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (39s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:05:03 (1786838703) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.52204 s, 34.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 15 20:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.383s user 0m0.001s sys 0m0.160s Stopping /mnt/lustre-ost1 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ed4d0a92000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.415s user 0m0.000s sys 0m0.173s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg203-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg203-server oleg203-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.466s user 0m0.000s sys 0m0.152s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg203-server: oleg203-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg203-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg203-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ed4d0a92000.ost_server_uuid in FULL state after 0 sec PASS 33b (40s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:05:43 (1786838743) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:05:45 (1786838745) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:05:46 (1786838746) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 20:05:48 (1786838748) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 20:05:50 (1786838750) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.252001 s, 49.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:05:55 (1786838755) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129238 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124287 s, 84.4 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:05:58 (1786838758) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:06:02 (1786838762) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111246 s, 920 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000816664 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000748011 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000632125 s, 1.6 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 20:06:05 (1786838765) create test files with size 15744184 .. 34bde689422336d04c9da461776764a9 - 9baf36055befe2f41eb1a7a305a32d2f - 9595efcbeb53e1f65fe86a6f69400669 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 20:06:12 (1786838772) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:06:17 (1786838777) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:06:19 (1786838779) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037237 s, 28.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188279 s, 55.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 20:06:22 (1786838782) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.033477 s, 62.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.111821 s, 37.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0406299 s, 51.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 20:06:27 (1786838787) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:06:29 (1786838789) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:06:30 (1786838790) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165129 s, 63.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 20:06:33 (1786838793) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0906383 s, 34.7 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:06:36 (1786838796) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:06:40 (1786838800) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.107278 s, 97.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:06:57 (1786838817) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9ed4c997f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.109843 s, 95.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg203-client.virtnet /mnt/lustre (opts:) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9ed4d24ce800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:07:02 (1786838822) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:07:16 (1786838836) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:07:28 (1786838848) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 20:07:29 (1786838849) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0936648 s, 33.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0607181 s, 51.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x559c0c78 (nosync,stale) Mirror 2: 0x559c0c78 (nosync,stale) ** verify mirror 2 doesn't change original checksum: c21f33f64459da145f0cd77c6c8a050d - mirror 1 checksum: b781f677c81d4413d95a29ef3d2f0229 - mirror 2 checksum: c21f33f64459da145f0cd77c6c8a050d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T20:07:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-15T20:07:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c21f33f64459da145f0cd77c6c8a050d - mirror 1 checksum: b781f677c81d4413d95a29ef3d2f0229 - mirror 2 checksum: b781f677c81d4413d95a29ef3d2f0229 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T20:07:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-15T20:07:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (4s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:07:33 (1786838853) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143878 s, 72.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:07:35 (1786838855) Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 20:07:42 (1786838862) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 20:07:57 (1786838877) targets are mounted 20:07:57 (1786838877) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126399 s, 83.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:08:05 (1786838885) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00786942 s, 2.6 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00396397 s, 5.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:08:08 (1786838888) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0179567 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00392453 s, 5.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:08:11 (1786838891) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00262369 s, 6.2 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:08:14 (1786838894) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.018727 s, 1.1 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:08:16 (1786838896) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00268874 s, 1.5 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00924802 s, 443 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:08:19 (1786838899) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:08:24 (1786838904) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:08:43 (1786838923) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:09:01 (1786838941) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:09:19 (1786838959) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 20:09:21 (1786838961) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57492 mirror split pid 57493 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57492 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57493 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 20:10:23 (1786839023) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 9024 skipping zero length punch hole fallocating to largest ever: 0x40000 000002 1786839024.318578 fallocate 0x036180 thru 0x03ffff (0x9e80 bytes) 000003 1786839024.321365 read 0x03889a thru 0x03e215 (0x597c bytes) 000004 1786839024.325060 punch from 0x0164b2 to 0x01730c (0xe5b bytes) 000005 1786839024.326997 trunc from 0x040000 to 0x03c61e (0x39e1 bytes) 000006 1786839024.332081 fallocate 0x0382fe thru 0x03ffff (0x7d02 bytes) 000007 1786839024.334534 trunc from 0x004630 to 0x00cf2d (0x88fe bytes) 000008 1786839024.336292 read 0x021a30 thru 0x03077a (0xed4b bytes) 000009 1786839024.342984 MIRROR_EXTEND to 2 mirrors 000010 1786839024.523385 MIRROR_EXTEND to 3 mirrors 000011 1786839024.687627 mapwrite 0x0297e0 thru 0x0297ee (0xf bytes) 000012 1786839024.717509 read 0x02fc84 thru 0x0342d5 (0x4652 bytes) 000013 1786839024.720497 trunc from 0x03c61f to 0x014d8c (0x27892 bytes) 000014 1786839024.734077 trunc from 0x014d8d to 0x0303ff (0x1b673 bytes) 000015 1786839024.739215 trunc from 0x00781b to 0x00e682 (0x6e68 bytes) 000016 1786839024.742553 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30400): Mirror 3: 0x8c95da6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30400): Mirror 3: 0x8c95da6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30400: skipped 000017 1786839024.820158 write 0x02ef23 thru 0x035988 (0x6a66 bytes) 000018 1786839024.823725 mapread 0x01d3fe thru 0x02a953 (0xd556 bytes) 000019 1786839024.831648 mapread 0x01c8f9 thru 0x02bee9 (0xf5f1 bytes) 000020 1786839024.832574 write 0x033297 thru 0x033728 (0x492 bytes) 000021 1786839024.832958 fallocate 0x005dc5 thru 0x015c93 (0xfecf bytes) 000022 1786839024.835225 mapread 0x0220f9 thru 0x02c4b1 (0xa3b9 bytes) 000023 1786839024.836258 punch from 0x01e823 to 0x0208b7 (0x2095 bytes) 000024 1786839024.837884 trunc from 0x007177 to 0x010c60 (0x9aea bytes) 000026 1786839024.840246 write 0x039e72 thru 0x03ffff (0x618e bytes) 000027 1786839024.841181 read 0x00d1e5 thru 0x018d57 (0xbb73 bytes) 000028 1786839024.844298 fallocate 0x02eafe thru 0x0334d7 (0x49da bytes) 000029 1786839024.846408 punch from 0x03f611 to 0x03ffff (0x9ef bytes) 000030 1786839024.850964 punch from 0x01c4ea to 0x023b2c (0x7643 bytes) 000031 1786839024.852911 trunc from 0x040000 to 0x002f16 (0x3d0e9 bytes) 000032 1786839024.858179 read 0x001122 thru 0x002f16 (0x1df5 bytes) 000033 1786839024.861824 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000034 1786839024.897334 write 0x0229f9 thru 0x02ec3a (0xc242 bytes) 000035 1786839024.902039 MIRROR_EXTEND to 2 mirrors 000036 1786839025.091218 mapwrite 0x006388 thru 0x00976e (0x33e7 bytes) 000037 1786839025.125221 mapwrite 0x0091ff thru 0x00afe2 (0x1de4 bytes) 000038 1786839025.130019 write 0x0277f9 thru 0x028a7b (0x1283 bytes) 000039 1786839025.136167 MIRROR_RESYNC 2 mirrors 000040 1786839025.315548 read 0x01e4fe thru 0x02888b (0xa38e bytes) 000041 1786839025.318308 trunc from 0x02ec3b to 0x02ad95 (0x3ea5 bytes) 000042 1786839025.335184 MIRROR_RESYNC 2 mirrors 000043 1786839025.516808 MIRROR_RESYNC 2 mirrors 000044 1786839025.627067 punch from 0x01ebcc to 0x02ad95 (0xc1ca bytes) 000045 1786839025.635989 trunc from 0x02ad96 to 0x0151bf (0x15bd6 bytes) 000047 1786839025.646927 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000049 1786839025.683114 punch from 0x001dc8 to 0x0086d0 (0x6909 bytes) 000050 1786839025.685154 trunc from 0x014b9c to 0x0151bf (0x624 bytes) 000051 1786839025.686834 mapread 0x002045 thru 0x004594 (0x2550 bytes) 000052 1786839025.693457 read 0x0108b0 thru 0x0151bf (0x4910 bytes) 000054 1786839025.699194 MIRROR_EXTEND to 2 mirrors 000055 1786839025.852787 mapwrite 0x03dfa6 thru 0x03ffff (0x205a bytes) 000056 1786839025.876298 mapread 0x038f0e thru 0x03f6c7 (0x67ba bytes) 000057 1786839025.877317 write 0x03dd0f thru 0x03ffff (0x22f1 bytes) 000058 1786839025.879896 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000059 1786839025.918291 punch from 0x02254e to 0x0289a5 (0x6458 bytes) 000061 1786839025.921081 trunc from 0x0272ff to 0x03198d (0xa68f bytes) 000062 1786839025.923162 fallocate 0x03bdb5 thru 0x03ffff (0x424b bytes) 000063 1786839025.924968 punch from 0x035fe6 to 0x03931a (0x3335 bytes) 000064 1786839025.926381 trunc from 0x038258 to 0x03ffff (0x7da8 bytes) 000066 1786839025.928535 trunc from 0x040000 to 0x03b3f4 (0x4c0b bytes) 000067 1786839025.935240 MIRROR_EXTEND to 2 mirrors 000069 1786839026.098956 punch from 0x021916 to 0x02262a (0xd15 bytes) 000070 1786839026.107537 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b3f5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3f5): Mirror 2: 0xc612f736 (stale) CRC-32 checksum value for chunk [0, 0x3b3f5): Mirror 1: 0xc612f736 Mirror 3: 0xc612f736 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3f5: skipped 000071 1786839026.325190 mapwrite 0x0357e6 thru 0x03ffff (0xa81a bytes) 000072 1786839026.348162 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b25e350 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b25e350 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000073 1786839026.424593 read 0x025653 thru 0x02f0a2 (0x9a50 bytes) 000074 1786839026.427574 mapwrite 0x032f12 thru 0x03ffff (0xd0ee bytes) 000075 1786839026.441030 trunc from 0x02730a to 0x030b0e (0x9805 bytes) 000076 1786839026.442633 fallocate 0x00fc99 thru 0x011acd (0x1e35 bytes) 000077 1786839026.445666 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000078 1786839026.479144 fallocate 0x01eee9 thru 0x0279b2 (0x8aca bytes) 000079 1786839026.481166 mapwrite 0x006b80 thru 0x00c82d (0x5cae bytes) 000081 1786839026.493563 mapwrite 0x03a578 thru 0x03ffff (0x5a88 bytes) 000082 1786839026.502294 mapread 0x005f2b thru 0x014759 (0xe82f bytes) 000083 1786839026.506052 read 0x00763e thru 0x00dbbe (0x6581 bytes) 000084 1786839026.506515 mapread 0x01f3b6 thru 0x0272fa (0x7f45 bytes) 000085 1786839026.512567 trunc from 0x0160ea to 0x0251f6 (0xf10d bytes) 000086 1786839026.514803 mapwrite 0x03212f thru 0x032dae (0xc80 bytes) 000087 1786839026.521911 trunc from 0x040000 to 0x03c30e (0x3cf1 bytes) 000088 1786839026.528336 write 0x015705 thru 0x01973e (0x403a bytes) 000089 1786839026.531668 read 0x01990e thru 0x01bc09 (0x22fc bytes) 000090 1786839026.532294 trunc from 0x03c30f to 0x0195c8 (0x22d46 bytes) 000091 1786839026.536769 trunc from 0x0195c9 to 0x036f0c (0x1d944 bytes) 000092 1786839026.540888 punch from 0x01ac73 to 0x01d52d (0x28bb bytes) 000093 1786839026.542459 mapwrite 0x01cd95 thru 0x01e1e8 (0x1454 bytes) 000094 1786839026.551649 mapwrite 0x007388 thru 0x010921 (0x959a bytes) 000096 1786839026.558959 trunc from 0x036f0d to 0x02e694 (0x8878 bytes) 000097 1786839026.563031 mapread 0x02e3a8 thru 0x02e694 (0x2ed bytes) 000098 1786839026.563716 write 0x00dfef thru 0x01126e (0x3280 bytes) 000099 1786839026.564407 write 0x014773 thru 0x015226 (0xab4 bytes) 000103 1786839026.566372 punch from 0x00dcd8 to 0x01724b (0x9574 bytes) 000104 1786839026.571054 punch from 0x0164ed to 0x025875 (0xf389 bytes) 000105 1786839026.572743 trunc from 0x02e695 to 0x01b627 (0x1306d bytes) 000106 1786839026.577217 trunc from 0x010341 to 0x01565c (0x531c bytes) 000107 1786839026.578770 punch from 0x0062f0 to 0x016281 (0xff92 bytes) 000109 1786839026.580980 mapread 0x013a80 thru 0x01b627 (0x7ba8 bytes) 000110 1786839026.593684 trunc from 0x01b628 to 0x023a4e (0x8427 bytes) 000111 1786839026.598635 trunc from 0x017c49 to 0x023a4e (0xbe06 bytes) 000112 1786839026.600628 trunc from 0x01ad6b to 0x01c7a2 (0x1a38 bytes) 000113 1786839026.602169 trunc from 0x012491 to 0x013924 (0x1494 bytes) 000114 1786839026.603642 mapread 0x01cc95 thru 0x023a4e (0x6dba bytes) 000115 1786839026.613867 trunc from 0x00adbd to 0x017790 (0xc9d4 bytes) 000116 1786839026.615709 mapwrite 0x03e8ac thru 0x03e9fb (0x150 bytes) 000117 1786839026.626582 mapread 0x028e8b thru 0x02907f (0x1f5 bytes) 000118 1786839026.629840 trunc from 0x02e3d4 to 0x0332db (0x4f08 bytes) 000119 1786839026.631756 read 0x00afb4 thru 0x016f60 (0xbfad bytes) 000121 1786839026.635601 fallocate 0x010c27 thru 0x01d4de (0xc8b8 bytes) 000122 1786839026.637742 mapwrite 0x001ea5 thru 0x00fcae (0xde0a bytes) 000123 1786839026.646064 write 0x02348b thru 0x027b51 (0x46c7 bytes) 000124 1786839026.647216 fallocate 0x01b66a thru 0x01fc0a (0x45a1 bytes) 000125 1786839026.649549 mapread 0x039e56 thru 0x03e9fb (0x4ba6 bytes) 000126 1786839026.650536 punch from 0x00c5db to 0x0126b5 (0x60db bytes) 000127 1786839026.656047 trunc from 0x03e9fc to 0x030d9c (0xdc5f bytes) 000128 1786839026.662054 trunc from 0x0019bd to 0x00e4ec (0xcb30 bytes) 000129 1786839026.664116 mapwrite 0x03d904 thru 0x03ffff (0x26fc bytes) 000132 1786839026.679654 mapread 0x00eabd thru 0x010f96 (0x24da bytes) 000133 1786839026.685019 trunc from 0x030ac3 to 0x034af9 (0x4037 bytes) 000134 1786839026.686991 write 0x004be9 thru 0x008c33 (0x404b bytes) 000135 1786839026.690571 trunc from 0x040000 to 0x009bce (0x36431 bytes) 000136 1786839026.696125 trunc from 0x0086fe to 0x009bce (0x14d1 bytes) 000138 1786839026.701990 fallocate 0x011743 thru 0x013776 (0x2034 bytes) 000139 1786839026.703910 read 0x001896 thru 0x009bce (0x8339 bytes) 000140 1786839026.707074 trunc from 0x009bcf to 0x002cf0 (0x6ede bytes) 000141 1786839026.711641 read 0x001967 thru 0x002cf0 (0x138a bytes) 000142 1786839026.712269 read 0x0005e1 thru 0x002cf0 (0x2710 bytes) 000143 1786839026.712890 mapread 0x001303 thru 0x002cf0 (0x19ee bytes) 000144 1786839026.713780 punch from 0x0009db to 0x002cf0 (0x2316 bytes) 000146 1786839026.716150 fallocate 0x00dd16 thru 0x0107f3 (0x2ade bytes) 000147 1786839026.717973 read 0x00174b thru 0x002cf0 (0x15a6 bytes) 000148 1786839026.720197 punch from 0x002bac to 0x002cf0 (0x145 bytes) 000149 1786839026.721948 trunc from 0x001c51 to 0x002cf0 (0x10a0 bytes) 000150 1786839026.723629 trunc from 0x002cf1 to 0x0266ba (0x239ca bytes) 000153 1786839026.728775 read 0x024d38 thru 0x0264c0 (0x1789 bytes) 000154 1786839026.734215 MIRROR_EXTEND to 2 mirrors 000155 1786839026.895075 punch from 0x01548d to 0x01fbf4 (0xa768 bytes) 000157 1786839026.907157 punch from 0x01c394 to 0x021add (0x574a bytes) 000159 1786839026.909875 mapread 0x009d66 thru 0x00fed7 (0x6172 bytes) 000160 1786839026.916231 fallocate 0x03a6c8 thru 0x03ffff (0x5938 bytes) 000161 1786839026.918479 fallocate 0x02cbdf thru 0x030694 (0x3ab6 bytes) 000162 1786839026.920519 punch from 0x027375 to 0x030694 (0x9320 bytes) 000163 1786839026.922116 trunc from 0x003a93 to 0x00c003 (0x8571 bytes) 000164 1786839026.923841 trunc from 0x030695 to 0x02513a (0xb55a bytes) 000165 1786839026.935502 trunc from 0x0031df to 0x01104b (0xde6d bytes) 000166 1786839026.937232 fallocate 0x0109c8 thru 0x01774c (0x6d85 bytes) 000167 1786839026.938967 trunc from 0x02513b to 0x017797 (0xd9a3 bytes) 000168 1786839026.943258 read 0x0120b4 thru 0x017797 (0x56e4 bytes) 000171 1786839026.946619 mapwrite 0x03fe4f thru 0x03ffff (0x1b1 bytes) 000172 1786839026.957673 write 0x0304e4 thru 0x03490f (0x442c bytes) 000173 1786839026.961149 fallocate 0x023484 thru 0x02ce4e (0x99cb bytes) 000174 1786839026.963035 trunc from 0x0227b1 to 0x02cc8e (0xa4de bytes) 000175 1786839026.964722 write 0x0095a8 thru 0x009881 (0x2da bytes) 000176 1786839026.966720 trunc from 0x040000 to 0x0288cf (0x17730 bytes) 000177 1786839026.970951 read 0x007804 thru 0x0126ad (0xaeaa bytes) 000178 1786839026.974351 read 0x00b3e8 thru 0x00ec5b (0x3874 bytes) 000179 1786839026.974966 trunc from 0x0026c1 to 0x009a36 (0x7376 bytes) 000180 1786839026.978748 read 0x0281f1 thru 0x0288cf (0x6df bytes) 000182 1786839026.979740 mapread 0x025c98 thru 0x0288cf (0x2c38 bytes) 000183 1786839026.980742 write 0x02992e thru 0x0369f0 (0xd0c3 bytes) 000184 1786839026.982138 read 0x02a10b thru 0x02e18f (0x4085 bytes) 000185 1786839026.982540 punch from 0x02e0a3 to 0x0369f0 (0x894e bytes) 000187 1786839026.987746 trunc from 0x021604 to 0x02b88a (0xa287 bytes) 000189 1786839026.991244 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x369f1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369f1): Mirror 2: 0xca1777e6 (stale) CRC-32 checksum value for chunk [0, 0x369f1): Mirror 1: 0x361c1277 Mirror 3: 0x361c1277 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369f1: skipped 000190 1786839027.253059 write 0x0042de thru 0x013781 (0xf4a4 bytes) 000192 1786839027.266476 fallocate 0x03c895 thru 0x03ffff (0x376b bytes) 000193 1786839027.268714 trunc from 0x0369f1 to 0x00e45a (0x28596 bytes) 000194 1786839027.274342 punch from 0x006c7e to 0x00e45a (0x77dd bytes) 000195 1786839027.278498 write 0x00ad0c thru 0x00cb8f (0x1e84 bytes) 000196 1786839027.281196 fallocate 0x02f560 thru 0x03764e (0x80ef bytes) 000197 1786839027.282841 trunc from 0x000eff to 0x00e45a (0xd55c bytes) 000198 1786839027.285930 punch from 0x005eb1 to 0x00e45a (0x85aa bytes) 000199 1786839027.287411 punch from 0x000b02 to 0x007f17 (0x7416 bytes) 000200 1786839027.288767 trunc from 0x006a5e to 0x00e45a (0x79fd bytes) 000201 1786839027.291384 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xe45b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe45b): Mirror 2: 0xa7ea74bf (stale) Mirror 3: 0xa7ea74bf (stale) CRC-32 checksum value for chunk [0, 0xe45b): Mirror 1: 0xa7ea74bf Mirror 4: 0xa7ea74bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe45b: skipped 000202 1786839027.523170 trunc from 0x00be5a to 0x00e45a (0x2601 bytes) 000203 1786839027.532923 read 0x003293 thru 0x00e45a (0xb1c8 bytes) 000204 1786839027.535747 trunc from 0x00e45b to 0x0000a6 (0xe3b4 bytes) 000205 1786839027.541932 read 0x000075 thru 0x0000a6 (0x32 bytes) 000206 1786839027.543737 trunc from 0x000077 to 0x0000a6 (0x30 bytes) 000207 1786839027.545269 mapread 0x00006b thru 0x0000a6 (0x3c bytes) 000209 1786839027.547794 mapwrite 0x039c78 thru 0x03c4ce (0x2857 bytes) 000210 1786839027.561327 write 0x00d1d0 thru 0x01c2d4 (0xf105 bytes) 000211 1786839027.562523 write 0x023d86 thru 0x024508 (0x783 bytes) 000212 1786839027.563220 read 0x02a05d thru 0x03542e (0xb3d2 bytes) 000213 1786839027.565337 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c4cf): Mirror 2: 0x8ee3ba3 (stale) Mirror 4: 0x8ee3ba3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4cf): Mirror 2: 0x8ee3ba3 (stale) Mirror 4: 0x8ee3ba3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4cf: skipped 000215 1786839027.670253 fallocate 0x0346bf thru 0x035e09 (0x174b bytes) 000216 1786839027.672456 mapread 0x000f39 thru 0x005cac (0x4d74 bytes) 000217 1786839027.679584 fallocate 0x02728f thru 0x02a354 (0x30c6 bytes) 000218 1786839027.681792 write 0x009cc6 thru 0x016e1f (0xd15a bytes) 000219 1786839027.685891 read 0x023377 thru 0x031a77 (0xe701 bytes) 000220 1786839027.688989 trunc from 0x03c4cf to 0x028008 (0x144c6 bytes) 000221 1786839027.693983 fallocate 0x03aa89 thru 0x03ffff (0x5577 bytes) 000222 1786839027.695954 punch from 0x014f32 to 0x016642 (0x1711 bytes) 000223 1786839027.700226 mapwrite 0x018131 thru 0x024c22 (0xcaf2 bytes) 000224 1786839027.711353 write 0x01e5ab thru 0x022992 (0x43e8 bytes) 000225 1786839027.712158 trunc from 0x028009 to 0x039df2 (0x11dea bytes) 000226 1786839027.716271 read 0x01d14b thru 0x022754 (0x560a bytes) 000227 1786839027.716791 fallocate 0x02fc50 thru 0x02fdef (0x1a0 bytes) 000228 1786839027.720057 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39df3): Mirror 2: 0xc41b64a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39df3): Mirror 2: 0xc41b64a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39df3: skipped 000229 1786839027.802790 read 0x03824a thru 0x039df2 (0x1ba9 bytes) 000230 1786839027.804961 trunc from 0x039984 to 0x039df2 (0x46f bytes) 000231 1786839027.806642 fallocate 0x03c0ce thru 0x03ffff (0x3f32 bytes) 000232 1786839027.808492 mapwrite 0x01a3f9 thru 0x01df65 (0x3b6d bytes) 000233 1786839027.817930 mapwrite 0x03445e thru 0x03fe1d (0xb9c0 bytes) 000234 1786839027.833759 trunc from 0x03fe1e to 0x027e2a (0x17ff3 bytes) 000236 1786839027.838180 trunc from 0x01942d to 0x022ebf (0x9a93 bytes) 000237 1786839027.839663 write 0x01c1ba thru 0x0228e1 (0x6728 bytes) 000238 1786839027.842483 fallocate 0x0251d8 thru 0x02c4e0 (0x7309 bytes) 000239 1786839027.844115 mapwrite 0x007538 thru 0x011099 (0x9b62 bytes) 000240 1786839027.854016 mapread 0x0186eb thru 0x027e24 (0xf73a bytes) 000241 1786839027.858075 fallocate 0x016242 thru 0x016fa7 (0xd66 bytes) 000242 1786839027.860046 punch from 0x00c74a to 0x01b411 (0xecc8 bytes) 000243 1786839027.863267 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000244 1786839027.894539 read 0x003af3 thru 0x005970 (0x1e7e bytes) 000245 1786839027.896949 read 0x01b66b thru 0x01c078 (0xa0e bytes) 000248 1786839027.900955 read 0x003078 thru 0x00a8d9 (0x7862 bytes) 000250 1786839027.901889 mapwrite 0x022962 thru 0x02f248 (0xc8e7 bytes) 000252 1786839027.916601 mapwrite 0x03d524 thru 0x03ffff (0x2adc bytes) 000253 1786839027.927488 write 0x010d84 thru 0x017461 (0x66de bytes) 000254 1786839027.928732 mapwrite 0x0051a8 thru 0x010572 (0xb3cb bytes) 000256 1786839027.937675 trunc from 0x027136 to 0x03711a (0xffe5 bytes) 000258 1786839027.940351 trunc from 0x040000 to 0x01f194 (0x20e6b bytes) 000259 1786839027.945039 fallocate 0x02106c thru 0x02460a (0x359f bytes) 000260 1786839027.947143 read 0x010f2b thru 0x011769 (0x83f bytes) 000261 1786839027.947638 read 0x00d1b2 thru 0x016424 (0x9273 bytes) 000262 1786839027.948021 fallocate 0x001ade thru 0x00cf87 (0xb4aa bytes) 000264 1786839027.950901 trunc from 0x02460b to 0x021e85 (0x2785 bytes) 000265 1786839027.955317 trunc from 0x021e86 to 0x015540 (0xc945 bytes) 000266 1786839027.959848 trunc from 0x015541 to 0x027dd2 (0x12892 bytes) 000267 1786839027.963978 write 0x017cc3 thru 0x019908 (0x1c46 bytes) 000268 1786839027.967193 punch from 0x0128d7 to 0x0200b0 (0xd7da bytes) 000269 1786839027.970653 punch from 0x00c8e4 to 0x00f55a (0x2c77 bytes) 000271 1786839027.973042 fallocate 0x03c00c thru 0x03ffff (0x3ff4 bytes) 000273 1786839027.975891 mapwrite 0x0233f4 thru 0x027728 (0x4335 bytes) 000274 1786839027.982173 punch from 0x0183a3 to 0x01b7ab (0x3409 bytes) 000276 1786839027.984316 write 0x021004 thru 0x022e1b (0x1e18 bytes) 000277 1786839027.984973 read 0x00ae35 thru 0x00b1cf (0x39b bytes) 000278 1786839027.985376 mapwrite 0x00f16b thru 0x0157c2 (0x6658 bytes) 000279 1786839027.994068 trunc from 0x027dd3 to 0x017ea7 (0xff2b bytes) 000280 1786839027.998987 read 0x0003cf thru 0x001079 (0xcab bytes) 000281 1786839027.999515 trunc from 0x00f80f to 0x017ea7 (0x8699 bytes) 000282 1786839028.001458 read 0x01551e thru 0x017ea7 (0x298a bytes) 000283 1786839028.004856 mapread 0x00b0c0 thru 0x017ea7 (0xcde8 bytes) 000284 1786839028.005776 mapwrite 0x00f7ec thru 0x011007 (0x181c bytes) 000285 1786839028.011947 read 0x00a87c thru 0x00c42e (0x1bb3 bytes) 000287 1786839028.012959 trunc from 0x017ea8 to 0x0153f2 (0x2ab5 bytes) 000288 1786839028.019883 fallocate 0x0238c8 thru 0x026c78 (0x33b1 bytes) 000289 1786839028.022138 mapwrite 0x01ab35 thru 0x01f384 (0x4850 bytes) 000291 1786839028.033083 trunc from 0x026c79 to 0x02fc80 (0x9008 bytes) 000292 1786839028.038340 punch from 0x005b6f to 0x00d9e5 (0x7e77 bytes) 000293 1786839028.040728 write 0x0070d0 thru 0x0132fc (0xc22d bytes) 000295 1786839028.045431 read 0x016949 thru 0x01dd1b (0x73d3 bytes) 000296 1786839028.046138 trunc from 0x02fc81 to 0x003dae (0x2bed2 bytes) 000297 1786839028.051769 punch from 0x003cb5 to 0x003dae (0xfa bytes) 000298 1786839028.056648 trunc from 0x003daf to 0x01d02e (0x19280 bytes) 000299 1786839028.063976 MIRROR_EXTEND to 2 mirrors 000300 1786839028.237159 write 0x00dc36 thru 0x01ce72 (0xf23d bytes) 000302 1786839028.251488 MIRROR_RESYNC 2 mirrors 000303 1786839028.421569 punch from 0x016a3e to 0x0180a4 (0x1667 bytes) 000304 1786839028.432183 mapwrite 0x01cca5 thru 0x0211e4 (0x4540 bytes) 000305 1786839028.453120 MIRROR_RESYNC 2 mirrors 000306 1786839028.619257 trunc from 0x0211e5 to 0x026294 (0x50b0 bytes) 000307 1786839028.635790 mapwrite 0x0261c5 thru 0x02a7f7 (0x4633 bytes) 000308 1786839028.653785 read 0x00b9d8 thru 0x013fa4 (0x85cd bytes) 000310 1786839028.657715 punch from 0x00e5a5 to 0x01a239 (0xbc95 bytes) 000311 1786839028.660125 mapwrite 0x01b470 thru 0x01c1f0 (0xd81 bytes) 000312 1786839028.669987 MIRROR_RESYNC 2 mirrors 000313 1786839028.846786 punch from 0x001152 to 0x0075ca (0x6479 bytes) 000315 1786839028.859523 trunc from 0x007d69 to 0x00e384 (0x661c bytes) 000316 1786839028.862033 fallocate 0x03ffc5 thru 0x03ffff (0x3b bytes) 000317 1786839028.865526 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000318 1786839028.908323 punch from 0x02274b to 0x02a7f7 (0x80ad bytes) 000319 1786839028.910396 read 0x01c6f6 thru 0x01d62e (0xf39 bytes) 000320 1786839028.912704 write 0x03c410 thru 0x03ffff (0x3bf0 bytes) 000321 1786839028.914628 trunc from 0x040000 to 0x027978 (0x18687 bytes) 000322 1786839028.924049 mapread 0x0114cd thru 0x01e2a7 (0xcddb bytes) 000323 1786839028.931847 punch from 0x01a53f to 0x0259e2 (0xb4a4 bytes) 000324 1786839028.933749 write 0x023c7b thru 0x02d554 (0x98da bytes) 000325 1786839028.936679 write 0x029d19 thru 0x0398ca (0xfbb2 bytes) 000327 1786839028.938179 read 0x002423 thru 0x005bf9 (0x37d7 bytes) 000328 1786839028.941513 write 0x015f20 thru 0x0177b7 (0x1898 bytes) 000329 1786839028.942166 write 0x000a70 thru 0x00d5b5 (0xcb46 bytes) 000330 1786839028.943108 mapread 0x02f9c5 thru 0x0398ca (0x9f06 bytes) 000331 1786839028.943892 read 0x0197de thru 0x01e86a (0x508d bytes) 000332 1786839028.944334 read 0x0339ba thru 0x0398ca (0x5f11 bytes) 000333 1786839028.944739 fallocate 0x02d28f thru 0x034e8a (0x7bfc bytes) 000334 1786839028.947030 read 0x01f3fe thru 0x02ce3c (0xda3f bytes) 000335 1786839028.947595 trunc from 0x020e77 to 0x022e9c (0x2026 bytes) 000336 1786839028.949560 trunc from 0x01858e to 0x01e007 (0x5a7a bytes) 000338 1786839028.951863 fallocate 0x026c12 thru 0x030443 (0x9832 bytes) 000339 1786839028.953715 trunc from 0x0398cb to 0x028c55 (0x10c75 bytes) 000340 1786839028.958172 read 0x01d0fd thru 0x028c55 (0xbb59 bytes) 000341 1786839028.961124 read 0x005cb4 thru 0x00d232 (0x757f bytes) 000344 1786839028.962537 mapread 0x019fc7 thru 0x023e47 (0x9e81 bytes) 000346 1786839028.964714 mapwrite 0x03d1b4 thru 0x03ed86 (0x1bd3 bytes) 000347 1786839028.976652 trunc from 0x03ed87 to 0x016fa3 (0x27de3 bytes) 000349 1786839028.981667 trunc from 0x003cf9 to 0x004061 (0x369 bytes) 000350 1786839028.983195 trunc from 0x016fa4 to 0x03e26a (0x272c7 bytes) 000351 1786839028.987127 mapread 0x001b07 thru 0x00e045 (0xc53f bytes) 000352 1786839028.990626 trunc from 0x03e26b to 0x00e325 (0x2ff45 bytes) 000353 1786839028.997471 MIRROR_EXTEND to 2 mirrors 000354 1786839029.151593 fallocate 0x010f44 thru 0x019fa0 (0x905d bytes) 000355 1786839029.160446 mapread 0x010ed0 thru 0x019fa0 (0x90d1 bytes) 000356 1786839029.167727 fallocate 0x026d6b thru 0x028a05 (0x1c9b bytes) 000357 1786839029.172065 MIRROR_RESYNC 2 mirrors 000358 1786839029.338430 trunc from 0x019fa1 to 0x000d1e (0x19282 bytes) 000359 1786839029.347911 punch from 0x000006 to 0x000a08 (0xa03 bytes) 000360 1786839029.349656 write 0x02c320 thru 0x03115f (0x4e40 bytes) 000361 1786839029.350783 punch from 0x02ce6c to 0x02ea42 (0x1bd7 bytes) 000362 1786839029.354949 punch from 0x007f8a to 0x0085f9 (0x670 bytes) 000363 1786839029.356594 read 0x021a49 thru 0x026b12 (0x50ca bytes) 000364 1786839029.359555 fallocate 0x014ae5 thru 0x01cd6f (0x828b bytes) 000365 1786839029.361499 mapread 0x003777 thru 0x00ad4b (0x75d5 bytes) 000366 1786839029.368805 mapread 0x004aab thru 0x011b03 (0xd059 bytes) 000368 1786839029.373082 mapread 0x019e46 thru 0x025239 (0xb3f4 bytes) 000370 1786839029.378460 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31160): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31160): Mirror 2: 0x1a28d052 (stale) CRC-32 checksum value for chunk [0, 0x31160): Mirror 1: 0x9f4a52a9 Mirror 3: 0x9f4a52a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31160: skipped 000371 1786839029.616561 punch from 0x015c0e to 0x02475c (0xeb4f bytes) 000372 1786839029.626517 trunc from 0x031160 to 0x01d21d (0x13f42 bytes) 000373 1786839029.637933 mapread 0x01a005 thru 0x01c4d7 (0x24d3 bytes) 000374 1786839029.643516 fallocate 0x0199c7 thru 0x020dae (0x73e8 bytes) 000375 1786839029.647266 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20daf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20daf): Mirror 2: 0x4e83d2f3 (stale) Mirror 3: 0x41fae005 (stale) CRC-32 checksum value for chunk [0, 0x20daf): Mirror 1: 0x41fae005 Mirror 4: 0x41fae005 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20daf: skipped 000376 1786839029.879392 read 0x00a424 thru 0x012cd0 (0x88ad bytes) 000377 1786839029.886089 mapwrite 0x001e65 thru 0x00b163 (0x92ff bytes) 000378 1786839029.928530 punch from 0x0068b4 to 0x00c05b (0x57a8 bytes) 000379 1786839029.932539 MIRROR_RESYNC 4 mirrors 000380 1786839030.187957 MIRROR_RESYNC 4 mirrors 000381 1786839030.330171 mapwrite 0x01a2d4 thru 0x01d4aa (0x31d7 bytes) 000383 1786839030.370842 mapwrite 0x03cb1d thru 0x03ffff (0x34e3 bytes) 000384 1786839030.384770 punch from 0x03b55e to 0x03ffff (0x4aa2 bytes) 000385 1786839030.388844 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6786b087 (stale) Mirror 3: 0x6786b087 (stale) Mirror 4: 0x6786b087 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x202982b2 Mirror 5: 0x202982b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000386 1786839030.650781 mapread 0x019565 thru 0x01d9e7 (0x4483 bytes) 000387 1786839030.657460 mapwrite 0x006528 thru 0x010ee8 (0xa9c1 bytes) 000388 1786839030.694555 trunc from 0x040000 to 0x038e7c (0x7183 bytes) 000389 1786839030.707203 mapwrite 0x0044d4 thru 0x009f90 (0x5abd bytes) 000390 1786839030.715722 mapwrite 0x01643a thru 0x0186ad (0x2274 bytes) 000391 1786839030.724012 mapwrite 0x032d4e thru 0x03ffff (0xd2b2 bytes) 000392 1786839030.742829 trunc from 0x040000 to 0x00581a (0x3a7e5 bytes) 000393 1786839030.748775 punch from 0x000ab9 to 0x00581a (0x4d62 bytes) 000394 1786839030.751026 read 0x000fec thru 0x00581a (0x482f bytes) 000395 1786839030.754010 read 0x0017de thru 0x00581a (0x403d bytes) 000396 1786839030.756485 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x581b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x581b): Mirror 2: 0x7f17700 (stale) Mirror 3: 0x7f17700 (stale) Mirror 4: 0x7f17700 (stale) Mirror 5: 0x7f17700 (stale) CRC-32 checksum value for chunk [0, 0x581b): Mirror 1: 0xfc606dfe Mirror 6: 0xfc606dfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x581b: skipped 000397 1786839030.989949 write 0x01d5c1 thru 0x023167 (0x5ba7 bytes) 000398 1786839031.001746 trunc from 0x0203be to 0x023167 (0x2daa bytes) 000399 1786839031.006301 read 0x0023f2 thru 0x01208c (0xfc9b bytes) 000400 1786839031.009823 fallocate 0x02b018 thru 0x03abc4 (0xfbad bytes) 000401 1786839031.012431 write 0x00ea93 thru 0x017360 (0x88ce bytes) 000403 1786839031.015450 mapread 0x00363e thru 0x006fa7 (0x396a bytes) 000404 1786839031.016201 fallocate 0x03a36e thru 0x03e77e (0x4411 bytes) 000405 1786839031.018798 trunc from 0x03e77f to 0x008a0b (0x35d73 bytes) 000407 1786839031.031284 trunc from 0x005c00 to 0x008a0b (0x2e0c bytes) 000408 1786839031.035025 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8a0c): Mirror 2: 0x9cbfa007 (stale) Mirror 3: 0x9cbfa007 (stale) Mirror 4: 0x9cbfa007 (stale) Mirror 6: 0xff54ef9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8a0c): Mirror 2: 0x9cbfa007 (stale) Mirror 3: 0x9cbfa007 (stale) Mirror 4: 0x9cbfa007 (stale) Mirror 6: 0xff54ef9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8a0c: skipped 000409 1786839031.139275 read 0x001c60 thru 0x008a0b (0x6dac bytes) 000410 1786839031.142717 read 0x0078df thru 0x008a0b (0x112d bytes) 000411 1786839031.143200 punch from 0x00851a to 0x008a0b (0x4f2 bytes) 000412 1786839031.145380 mapread 0x0032e0 thru 0x008a0b (0x572c bytes) 000413 1786839031.148826 write 0x02f84f thru 0x03dcec (0xe49e bytes) 000414 1786839031.151798 trunc from 0x00dcb8 to 0x0151c3 (0x750c bytes) 000415 1786839031.154028 fallocate 0x025e92 thru 0x031dfd (0xbf6c bytes) 000416 1786839031.156685 write 0x03e7e7 thru 0x03ffff (0x1819 bytes) 000417 1786839031.157778 mapwrite 0x00b24f thru 0x00f2ec (0x409e bytes) 000418 1786839031.170373 mapwrite 0x027914 thru 0x033be3 (0xc2d0 bytes) 000419 1786839031.180342 trunc from 0x027664 to 0x02778d (0x12a bytes) 000421 1786839031.183790 mapwrite 0x02448f thru 0x031b07 (0xd679 bytes) 000422 1786839031.194942 trunc from 0x03448e to 0x03ffff (0xbb72 bytes) 000423 1786839031.197244 trunc from 0x040000 to 0x00f0ce (0x30f31 bytes) 000424 1786839031.204193 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf0cf): Mirror 2: 0x48e27a09 (stale) Mirror 3: 0x48e27a09 (stale) Mirror 6: 0xb889f509 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf0cf): Mirror 2: 0x48e27a09 (stale) Mirror 3: 0x48e27a09 (stale) Mirror 6: 0xb889f509 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf0cf: skipped 000425 1786839031.320219 write 0x002183 thru 0x00c2df (0xa15d bytes) 000428 1786839031.327574 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf0cf): Mirror 2: 0x48e27a09 (stale) Mirror 3: 0x48e27a09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf0cf): Mirror 2: 0x48e27a09 (stale) Mirror 3: 0x48e27a09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf0cf: skipped 000429 1786839031.428022 trunc from 0x00ecba to 0x00f0ce (0x415 bytes) 000430 1786839031.430169 read 0x00caa5 thru 0x00f0ce (0x262a bytes) 000431 1786839031.434591 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf0cf): Mirror 3: 0x48e27a09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf0cf): Mirror 3: 0x48e27a09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf0cf: skipped 000432 1786839031.514937 MIRROR_RESYNC 2 mirrors 000433 1786839031.717743 punch from 0x009898 to 0x00f0ce (0x5837 bytes) 000434 1786839031.729276 trunc from 0x00f0cf to 0x02b62e (0x1c560 bytes) 000435 1786839031.741166 read 0x020e6f thru 0x02240b (0x159d bytes) 000436 1786839031.743441 mapread 0x0076b0 thru 0x00b7f0 (0x4141 bytes) 000437 1786839031.753461 read 0x00061f thru 0x007007 (0x69e9 bytes) 000438 1786839031.757078 trunc from 0x01186b to 0x01df3d (0xc6d3 bytes) 000439 1786839031.761064 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b62f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b62f): Mirror 3: 0x144457b (stale) CRC-32 checksum value for chunk [0, 0x2b62f): Mirror 1: 0x7138405a Mirror 4: 0x7138405a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b62f: skipped 000440 1786839032.054297 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b62f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b62f): Mirror 3: 0x144457b (stale) CRC-32 checksum value for chunk [0, 0x2b62f): Mirror 1: 0x7138405a Mirror 4: 0x7138405a Mirror 5: 0x7138405a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b62f: skipped 000441 1786839032.328191 write 0x025b40 thru 0x02bdd3 (0x6294 bytes) 000442 1786839032.341855 mapread 0x014b79 thru 0x01b227 (0x66af bytes) 000443 1786839032.348555 trunc from 0x01ad5b to 0x023c56 (0x8efc bytes) 000444 1786839032.350749 mapread 0x024ffb thru 0x029643 (0x4649 bytes) 000445 1786839032.354346 mapwrite 0x005d84 thru 0x00c3ff (0x667c bytes) 000446 1786839032.365264 read 0x00daf6 thru 0x01aa85 (0xcf90 bytes) 000447 1786839032.367369 read 0x01f249 thru 0x0216a6 (0x245e bytes) 000448 1786839032.367970 write 0x02ab3d thru 0x02f1d1 (0x4695 bytes) 000449 1786839032.368920 punch from 0x003a3d to 0x00ac95 (0x7259 bytes) 000450 1786839032.371418 trunc from 0x010fe9 to 0x018109 (0x7121 bytes) 000451 1786839032.373428 mapwrite 0x00a778 thru 0x011d29 (0x75b2 bytes) 000452 1786839032.388200 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f1d2): Mirror 3: 0xa616bcc3 (stale) Mirror 5: 0x7a4c74ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f1d2): Mirror 3: 0xa616bcc3 (stale) Mirror 5: 0x7a4c74ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f1d2: skipped 000453 1786839032.485526 punch from 0x02620f to 0x02e9fc (0x87ee bytes) 000454 1786839032.487625 trunc from 0x02f1d2 to 0x03b116 (0xbf45 bytes) 000455 1786839032.497804 mapread 0x03a523 thru 0x03b116 (0xbf4 bytes) 000456 1786839032.501305 trunc from 0x021542 to 0x02a191 (0x8c50 bytes) 000457 1786839032.503636 mapread 0x037b40 thru 0x03919c (0x165d bytes) 000458 1786839032.507197 fallocate 0x03f828 thru 0x03ffff (0x7d8 bytes) 000459 1786839032.509426 mapwrite 0x002d81 thru 0x00fe6f (0xd0ef bytes) 000460 1786839032.523288 MIRROR_RESYNC 3 mirrors 000461 1786839032.773902 mapwrite 0x032b9a thru 0x03e543 (0xb9aa bytes) 000462 1786839032.816415 mapwrite 0x030183 thru 0x03ff2b (0xfda9 bytes) 000464 1786839032.827009 trunc from 0x00ff18 to 0x017f6c (0x8055 bytes) 000465 1786839032.829800 fallocate 0x03651b thru 0x03ba7d (0x5563 bytes) 000466 1786839032.832610 punch from 0x01fea4 to 0x0273ca (0x7527 bytes) 000467 1786839032.837304 MIRROR_RESYNC 3 mirrors 000468 1786839033.110212 read 0x026ab0 thru 0x02dfa9 (0x74fa bytes) 000469 1786839033.113860 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000471 1786839033.207199 mapwrite 0x014126 thru 0x019275 (0x5150 bytes) 000472 1786839033.242614 trunc from 0x040000 to 0x03e867 (0x1798 bytes) 000473 1786839033.250010 mapwrite 0x01b2c0 thru 0x01f126 (0x3e67 bytes) 000474 1786839033.265661 fallocate 0x02e690 thru 0x033b50 (0x54c1 bytes) 000475 1786839033.268082 trunc from 0x0034d9 to 0x00811e (0x4c46 bytes) 000476 1786839033.270319 fallocate 0x027711 thru 0x02b148 (0x3a38 bytes) 000477 1786839033.272781 trunc from 0x02752d to 0x033c6e (0xc742 bytes) 000478 1786839033.275115 fallocate 0x029228 thru 0x033f34 (0xad0d bytes) 000479 1786839033.277477 mapread 0x02041e thru 0x02fc17 (0xf7fa bytes) 000481 1786839033.282393 write 0x03cbf2 thru 0x03ffff (0x340e bytes) 000482 1786839033.285040 fallocate 0x03e822 thru 0x03f361 (0xb40 bytes) 000483 1786839033.287477 write 0x02d98b thru 0x0388d6 (0xaf4c bytes) 000484 1786839033.290086 mapwrite 0x03e1d8 thru 0x03ffff (0x1e28 bytes) 000485 1786839033.297357 mapwrite 0x03860d thru 0x03af53 (0x2947 bytes) 000486 1786839033.305217 write 0x02ec0d thru 0x036659 (0x7a4d bytes) 000487 1786839033.306464 mapread 0x0244d1 thru 0x031a49 (0xd579 bytes) 000488 1786839033.307388 trunc from 0x040000 to 0x03bce0 (0x431f bytes) 000489 1786839033.313176 mapwrite 0x02e140 thru 0x039bc3 (0xba84 bytes) 000490 1786839033.321324 mapwrite 0x028c87 thru 0x02e0c7 (0x5441 bytes) 000491 1786839033.329464 write 0x039de1 thru 0x03e377 (0x4597 bytes) 000492 1786839033.330511 punch from 0x02483c to 0x02b2ef (0x6ab4 bytes) 000493 1786839033.336745 punch from 0x005783 to 0x01074b (0xafc9 bytes) 000494 1786839033.341442 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e378): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e378): Mirror 5: 0x48e5755c (stale) CRC-32 checksum value for chunk [0, 0x3e378): Mirror 1: 0xd1ee3144 Mirror 6: 0xd1ee3144 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e378: skipped 000495 1786839033.619598 mapread 0x0298cc thru 0x02cb40 (0x3275 bytes) 000496 1786839033.628884 trunc from 0x004426 to 0x0062b6 (0x1e91 bytes) 000497 1786839033.639434 trunc from 0x000ae8 to 0x00170a (0xc23 bytes) 000498 1786839033.641429 mapwrite 0x023a8d thru 0x0243d0 (0x944 bytes) 000499 1786839033.649081 trunc from 0x03e378 to 0x014e72 (0x29505 bytes) 000500 1786839033.663171 mapread 0x00594f thru 0x0073bc (0x1a6e bytes) 000501 1786839033.667131 trunc from 0x0099b0 to 0x013c4f (0xa2a0 bytes) 000502 1786839033.669927 trunc from 0x013368 to 0x014e72 (0x1b0b bytes) 000503 1786839033.673651 MIRROR_RESYNC 3 mirrors 000504 1786839033.945334 read 0x00b77b thru 0x014e72 (0x96f8 bytes) 000505 1786839033.948854 mapwrite 0x03c9db thru 0x03ffff (0x3625 bytes) 000506 1786839033.979533 write 0x00af99 thru 0x0164b9 (0xb521 bytes) 000507 1786839033.983151 punch from 0x01c650 to 0x028281 (0xbc32 bytes) 000508 1786839033.984907 fallocate 0x031b70 thru 0x03ffff (0xe490 bytes) 000509 1786839033.986937 fallocate 0x03c72e thru 0x03ffff (0x38d2 bytes) 000510 1786839033.988982 read 0x01542d thru 0x021f44 (0xcb18 bytes) 000511 1786839033.992594 trunc from 0x02e208 to 0x0317f1 (0x35ea bytes) 000512 1786839033.994309 mapread 0x028e64 thru 0x035da3 (0xcf40 bytes) 000513 1786839033.999928 trunc from 0x040000 to 0x02fd2e (0x102d1 bytes) 000515 1786839034.006057 fallocate 0x013121 thru 0x014398 (0x1278 bytes) 000516 1786839034.008151 write 0x03b9c5 thru 0x03ffff (0x463b bytes) 000517 1786839034.011571 MIRROR_RESYNC 3 mirrors 000518 1786839034.315390 punch from 0x015d11 to 0x01a286 (0x4576 bytes) 000519 1786839034.330560 punch from 0x0126a2 to 0x01d53e (0xae9d bytes) 000520 1786839034.332939 trunc from 0x001b6a to 0x007727 (0x5bbe bytes) 000521 1786839034.336212 mapread 0x008154 thru 0x012fed (0xae9a bytes) 000522 1786839034.349954 read 0x03748a thru 0x03ffff (0x8b76 bytes) 000523 1786839034.354459 fallocate 0x0238e9 thru 0x02c4dc (0x8bf4 bytes) 000524 1786839034.357477 write 0x03ef38 thru 0x03ffff (0x10c8 bytes) 000526 1786839034.359048 fallocate 0x016595 thru 0x0176c9 (0x1135 bytes) 000527 1786839034.361607 punch from 0x032d4d to 0x03ffff (0xd2b3 bytes) 000528 1786839034.370073 punch from 0x03703a to 0x03ffff (0x8fc6 bytes) 000530 1786839034.377399 trunc from 0x02700b to 0x02786c (0x862 bytes) 000531 1786839034.379870 mapread 0x0225a2 thru 0x02cc53 (0xa6b2 bytes) 000532 1786839034.389007 punch from 0x032e96 to 0x0343c8 (0x1533 bytes) 000533 1786839034.391920 write 0x0055dd thru 0x015042 (0xfa66 bytes) 000534 1786839034.395240 write 0x0329e7 thru 0x0373fa (0x4a14 bytes) 000535 1786839034.405852 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x17edaebf (stale) Mirror 6: 0x17edaebf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x218065a0 Mirror 7: 0x218065a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000536 1786839034.721560 fallocate 0x026a05 thru 0x036311 (0xf90d bytes) 000537 1786839034.735732 write 0x02d51a thru 0x03420a (0x6cf1 bytes) 000538 1786839034.744306 write 0x020989 thru 0x02a5b0 (0x9c28 bytes) 000539 1786839034.753233 fallocate 0x00bee2 thru 0x00d8d0 (0x19ef bytes) 000540 1786839034.755591 punch from 0x00bdfa to 0x012f0d (0x7114 bytes) 000541 1786839034.757878 mapwrite 0x02c39d thru 0x034706 (0x836a bytes) 000542 1786839034.772845 mapread 0x033c90 thru 0x03ffff (0xc370 bytes) 000543 1786839034.783391 mapwrite 0x008a01 thru 0x008e85 (0x485 bytes) 000544 1786839034.792927 write 0x039d0f thru 0x03ffff (0x62f1 bytes) 000545 1786839034.794029 mapread 0x00eccf thru 0x012518 (0x384a bytes) 000546 1786839034.802306 read 0x015b75 thru 0x025a32 (0xfebe bytes) 000547 1786839034.805752 read 0x0298d5 thru 0x036103 (0xc82f bytes) 000550 1786839034.809726 fallocate 0x035990 thru 0x03ffff (0xa670 bytes) 000551 1786839034.812375 fallocate 0x01bfdb thru 0x023be0 (0x7c06 bytes) 000552 1786839034.814998 read 0x00e99c thru 0x01e93c (0xffa1 bytes) 000553 1786839034.815633 fallocate 0x028d23 thru 0x02f00b (0x62e9 bytes) 000554 1786839034.818660 mapread 0x025da5 thru 0x02c409 (0x6665 bytes) 000555 1786839034.819684 punch from 0x030545 to 0x036be2 (0x669e bytes) 000557 1786839034.822991 read 0x011931 thru 0x01c2bc (0xa98c bytes) 000558 1786839034.823519 mapread 0x031da9 thru 0x03602d (0x4285 bytes) 000560 1786839034.827588 punch from 0x031daa to 0x03ffff (0xe256 bytes) 000561 1786839034.832315 fallocate 0x00ff92 thru 0x01cb25 (0xcb94 bytes) 000562 1786839034.834437 trunc from 0x040000 to 0x018c81 (0x2737e bytes) 000563 1786839034.844807 write 0x031dc6 thru 0x03a380 (0x85bb bytes) 000564 1786839034.847330 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a381): Mirror 6: 0x34e68e40 (stale) Mirror 7: 0x39920faf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a381): Mirror 6: 0x34e68e40 (stale) Mirror 7: 0x39920faf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a381: skipped 000565 1786839034.957838 read 0x008d17 thru 0x01055f (0x7849 bytes) 000566 1786839034.960786 read 0x021e16 thru 0x0283cc (0x65b7 bytes) 000567 1786839034.964790 read 0x012d17 thru 0x018ee2 (0x61cc bytes) 000568 1786839034.965519 trunc from 0x03a381 to 0x01c13a (0x1e246 bytes) 000569 1786839034.974719 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c13b): Mirror 7: 0x72345ae2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c13b): Mirror 7: 0x72345ae2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c13b: skipped 000570 1786839035.105264 read 0x00254b thru 0x004560 (0x2016 bytes) 000572 1786839035.111250 mapwrite 0x00c1a0 thru 0x015941 (0x97a2 bytes) 000574 1786839035.138958 trunc from 0x011f7d to 0x01c13a (0xa1be bytes) 000575 1786839035.141697 read 0x015f2d thru 0x01695e (0xa32 bytes) 000576 1786839035.145761 mapwrite 0x023aea thru 0x0338fc (0xfe13 bytes) 000577 1786839035.167283 trunc from 0x013503 to 0x0190f3 (0x5bf1 bytes) 000578 1786839035.169936 trunc from 0x012f8d to 0x01fe29 (0xce9d bytes) 000579 1786839035.172402 trunc from 0x0338fd to 0x03e6f8 (0xadfc bytes) 000580 1786839035.178430 fallocate 0x031e32 thru 0x03966d (0x783c bytes) 000581 1786839035.180670 punch from 0x033ba1 to 0x03989b (0x5cfb bytes) 000582 1786839035.182825 mapwrite 0x032968 thru 0x039f4f (0x75e8 bytes) 000583 1786839035.196672 trunc from 0x03e6f9 to 0x0258ed (0x18e0b bytes) 000584 1786839035.201712 punch from 0x0036db to 0x00fba1 (0xc4c7 bytes) 000585 1786839035.203801 trunc from 0x01f157 to 0x0258ed (0x6797 bytes) 000586 1786839035.205717 trunc from 0x0258ee to 0x033224 (0xd937 bytes) 000587 1786839035.210367 trunc from 0x02e432 to 0x031413 (0x2fe2 bytes) 000588 1786839035.212741 trunc from 0x02bfa7 to 0x02c8b6 (0x910 bytes) 000589 1786839035.214875 mapread 0x022035 thru 0x024fac (0x2f78 bytes) 000591 1786839035.226427 punch from 0x01a77e to 0x028c90 (0xe513 bytes) 000592 1786839035.228691 trunc from 0x033225 to 0x03264e (0xbd6 bytes) 000593 1786839035.234201 trunc from 0x03264f to 0x02efab (0x36a3 bytes) 000594 1786839035.238957 mapwrite 0x0161f3 thru 0x024605 (0xe413 bytes) 000595 1786839035.269822 punch from 0x0161d1 to 0x01e2a6 (0x80d6 bytes) 000596 1786839035.272530 trunc from 0x02efac to 0x00efde (0x1ffcd bytes) 000597 1786839035.277511 mapread 0x00d1a2 thru 0x00efde (0x1e3d bytes) 000598 1786839035.281073 fallocate 0x03ddd5 thru 0x03ffff (0x222b bytes) 000599 1786839035.282968 fallocate 0x01903a thru 0x027b3e (0xeb05 bytes) 000600 1786839035.284867 read 0x0252d0 thru 0x027b3e (0x286f bytes) 000601 1786839035.288202 trunc from 0x0226d1 to 0x027b3e (0x546e bytes) 000602 1786839035.289851 mapread 0x023ce8 thru 0x027b3e (0x3e57 bytes) 000603 1786839035.296520 fallocate 0x019ea4 thru 0x01f278 (0x53d5 bytes) 000604 1786839035.298083 mapwrite 0x034ff0 thru 0x03688a (0x189b bytes) 000605 1786839035.309724 mapwrite 0x0089b6 thru 0x017015 (0xe660 bytes) 000607 1786839035.319002 read 0x02dbae thru 0x02e498 (0x8eb bytes) 000608 1786839035.321434 MIRROR_RESYNC 2 mirrors 000609 1786839035.534905 read 0x0231ff thru 0x02d275 (0xa077 bytes) 000611 1786839035.538182 punch from 0x02fc35 to 0x03688a (0x6c56 bytes) 000612 1786839035.550598 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000613 1786839035.597480 trunc from 0x03688b to 0x01863b (0x1e24f bytes) 000614 1786839035.609646 mapread 0x002607 thru 0x00b08a (0x8a84 bytes) 000615 1786839035.618181 mapread 0x0107ec thru 0x01863b (0x7e50 bytes) 000616 1786839035.625673 trunc from 0x002ef7 to 0x00b071 (0x817b bytes) 000617 1786839035.628523 fallocate 0x024bee thru 0x02aa66 (0x5e79 bytes) 000618 1786839035.630781 trunc from 0x01863c to 0x009544 (0xf0f7 bytes) 000619 1786839035.635866 write 0x0265ad thru 0x0282ee (0x1d42 bytes) 000620 1786839035.637033 punch from 0x017fb0 to 0x022167 (0xa1b8 bytes) 000621 1786839035.638756 punch from 0x0093fc to 0x00cb7e (0x3783 bytes) 000622 1786839035.640839 punch from 0x00a20b to 0x00ae6f (0xc65 bytes) 000623 1786839035.642715 mapread 0x00225b thru 0x002f61 (0xd07 bytes) 000624 1786839035.646855 read 0x001075 thru 0x00fd59 (0xece5 bytes) 000625 1786839035.647457 read 0x01bade thru 0x020592 (0x4ab5 bytes) 000626 1786839035.651627 write 0x022017 thru 0x024a92 (0x2a7c bytes) 000627 1786839035.652705 trunc from 0x00eafb to 0x0179fd (0x8f03 bytes) 000628 1786839035.655058 write 0x008ff1 thru 0x00d92a (0x493a bytes) 000629 1786839035.655885 write 0x010837 thru 0x011675 (0xe3f bytes) 000630 1786839035.659014 mapwrite 0x01c3b2 thru 0x0260fa (0x9d49 bytes) 000631 1786839035.667953 trunc from 0x0109b9 to 0x01d2c6 (0xc90e bytes) 000632 1786839035.671055 mapwrite 0x006e95 thru 0x013d03 (0xce6f bytes) 000633 1786839035.698798 mapread 0x01f1bc thru 0x0282ee (0x9133 bytes) 000635 1786839035.700346 fallocate 0x03d17d thru 0x03ffff (0x2e83 bytes) 000636 1786839035.705256 punch from 0x024aa5 to 0x02923b (0x4797 bytes) 000637 1786839035.709616 read 0x014e81 thru 0x023b22 (0xeca2 bytes) 000638 1786839035.714045 mapread 0x002ee0 thru 0x00aed4 (0x7ff5 bytes) 000639 1786839035.715144 mapread 0x03f2e0 thru 0x03ffff (0xd20 bytes) 000641 1786839035.716603 punch from 0x010f12 to 0x02015e (0xf24d bytes) 000642 1786839035.719562 fallocate 0x00a96e thru 0x014ff8 (0xa68b bytes) 000643 1786839035.721993 mapwrite 0x0039e1 thru 0x0043be (0x9de bytes) 000644 1786839035.728249 read 0x02747c thru 0x032d9f (0xb924 bytes) 000645 1786839035.728889 punch from 0x02ee33 to 0x033039 (0x4207 bytes) 000646 1786839035.731258 fallocate 0x0061ab thru 0x01529b (0xf0f1 bytes) 000647 1786839035.734285 write 0x03a3f5 thru 0x03a773 (0x37f bytes) 000648 1786839035.735123 trunc from 0x037418 to 0x03d528 (0x6111 bytes) 000649 1786839035.739474 trunc from 0x040000 to 0x0102ae (0x2fd51 bytes) 000650 1786839035.745598 write 0x0022bf thru 0x005982 (0x36c4 bytes) 000652 1786839035.747299 mapread 0x000f57 thru 0x002236 (0x12e0 bytes) 000653 1786839035.748279 mapread 0x00d8ec thru 0x0102ae (0x29c3 bytes) 000654 1786839035.751949 trunc from 0x0102af to 0x029eb9 (0x19c0b bytes) 000655 1786839035.759884 trunc from 0x00a8a1 to 0x012dff (0x855f bytes) 000657 1786839035.765204 mapwrite 0x03203d thru 0x039094 (0x7058 bytes) 000658 1786839035.783619 mapwrite 0x016475 thru 0x01fc02 (0x978e bytes) 000659 1786839035.793330 mapread 0x01893a thru 0x021fbb (0x9682 bytes) 000660 1786839035.794568 mapread 0x006328 thru 0x00a9a6 (0x467f bytes) 000661 1786839035.797778 trunc from 0x00ba12 to 0x00bee4 (0x4d3 bytes) 000662 1786839035.799867 mapwrite 0x0114b4 thru 0x0206b1 (0xf1fe bytes) 000663 1786839035.810434 trunc from 0x039095 to 0x022871 (0x16823 bytes) 000664 1786839035.817317 mapwrite 0x00c80b thru 0x00fcc4 (0x34ba bytes) 000665 1786839035.832977 mapwrite 0x03007b thru 0x03cc51 (0xcbd7 bytes) 000666 1786839035.866618 punch from 0x01df35 to 0x02bd74 (0xde40 bytes) 000667 1786839035.869014 trunc from 0x0152e4 to 0x01e9cc (0x96e9 bytes) 000668 1786839035.874758 MIRROR_EXTEND to 2 mirrors 000669 1786839036.086921 trunc from 0x03cc52 to 0x021af7 (0x1b15a bytes) 000670 1786839036.105951 trunc from 0x006f3f to 0x009b81 (0x2c43 bytes) 000671 1786839036.108082 trunc from 0x021af8 to 0x035810 (0x13d19 bytes) 000672 1786839036.113890 mapwrite 0x03190a thru 0x0326ce (0xdc5 bytes) 000673 1786839036.124528 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35811): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35811): Mirror 2: 0xcfc998c5 (stale) CRC-32 checksum value for chunk [0, 0x35811): Mirror 1: 0xa55b4172 Mirror 3: 0xa55b4172 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35811: skipped 000674 1786839036.445428 mapwrite 0x037350 thru 0x03fd7d (0x8a2e bytes) 000675 1786839036.478229 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fd7e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd7e): Mirror 2: 0x24ed8f34 (stale) Mirror 3: 0x59978715 (stale) CRC-32 checksum value for chunk [0, 0x3fd7e): Mirror 1: 0x9f915f22 Mirror 4: 0x9f915f22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd7e: skipped 000676 1786839036.837704 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fd7e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd7e): Mirror 2: 0x24ed8f34 (stale) Mirror 3: 0x59978715 (stale) CRC-32 checksum value for chunk [0, 0x3fd7e): Mirror 1: 0x9f915f22 Mirror 4: 0x9f915f22 Mirror 5: 0x9f915f22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd7e: skipped 000677 1786839037.168943 mapwrite 0x019c9e thru 0x02754c (0xd8af bytes) 000678 1786839037.211246 trunc from 0x03fd7e to 0x03e6e7 (0x1696 bytes) 000679 1786839037.227923 punch from 0x038a82 to 0x03e6e7 (0x5c66 bytes) 000680 1786839037.230242 read 0x025958 thru 0x02fe0d (0xa4b6 bytes) 000681 1786839037.233608 mapread 0x02d484 thru 0x03a4e2 (0xd05f bytes) 000682 1786839037.242596 mapwrite 0x01d436 thru 0x02c89c (0xf467 bytes) 000683 1786839037.256867 mapwrite 0x02696c thru 0x02911f (0x27b4 bytes) 000684 1786839037.264606 trunc from 0x03d965 to 0x03e6e7 (0xd83 bytes) 000685 1786839037.267316 trunc from 0x03e6e8 to 0x002b64 (0x3bb83 bytes) 000686 1786839037.273609 mapread 0x00178a thru 0x002b64 (0x13db bytes) 000687 1786839037.276514 trunc from 0x001cf5 to 0x0020f9 (0x405 bytes) 000688 1786839037.278948 trunc from 0x00113d to 0x002b64 (0x1a28 bytes) 000689 1786839037.280837 write 0x0174fe thru 0x021842 (0xa345 bytes) 000690 1786839037.282035 trunc from 0x021843 to 0x038104 (0x168c2 bytes) 000691 1786839037.287331 read 0x02f797 thru 0x036e9c (0x7706 bytes) 000692 1786839037.292193 read 0x0002ff thru 0x00bddb (0xbadd bytes) 000693 1786839037.293077 trunc from 0x034d5c to 0x0374bf (0x2764 bytes) 000694 1786839037.295272 mapread 0x02ce6b thru 0x02e545 (0x16db bytes) 000695 1786839037.297765 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38105): Mirror 3: 0x9737dfa9 (stale) Mirror 4: 0xaf7d5633 (stale) Mirror 5: 0xaf7d5633 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38105): Mirror 3: 0x9737dfa9 (stale) Mirror 4: 0xaf7d5633 (stale) Mirror 5: 0xaf7d5633 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38105: skipped 000696 1786839037.432857 punch from 0x006fd4 to 0x00d23b (0x6268 bytes) 000697 1786839037.434797 write 0x00aeef thru 0x0144d5 (0x95e7 bytes) 000698 1786839037.440203 trunc from 0x01daa1 to 0x02a14f (0xc6af bytes) 000699 1786839037.444414 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38105): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38105): Mirror 3: 0x9737dfa9 (stale) Mirror 4: 0xaf7d5633 (stale) Mirror 5: 0xaf7d5633 (stale) CRC-32 checksum value for chunk [0, 0x38105): Mirror 1: 0x2ec64826 Mirror 6: 0x2ec64826 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38105: skipped 000700 1786839037.734849 read 0x01418a thru 0x021dc5 (0xdc3c bytes) 000701 1786839037.738145 write 0x0223fa thru 0x02a5f3 (0x81fa bytes) 000702 1786839037.753875 trunc from 0x038105 to 0x016073 (0x22091 bytes) 000703 1786839037.766268 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16074): Mirror 4: 0xf4f49eda (stale) Mirror 5: 0xf4f49eda (stale) Mirror 6: 0xb3d49502 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16074): Mirror 4: 0xf4f49eda (stale) Mirror 5: 0xf4f49eda (stale) Mirror 6: 0xb3d49502 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16074: skipped 000704 1786839037.864548 trunc from 0x002b94 to 0x010501 (0xd96e bytes) 000705 1786839037.867032 mapread 0x00e9e7 thru 0x016073 (0x768d bytes) 000706 1786839037.874536 trunc from 0x007613 to 0x00f261 (0x7c4f bytes) 000707 1786839037.876991 trunc from 0x016074 to 0x023af5 (0xda82 bytes) 000708 1786839037.884004 fallocate 0x001c9b thru 0x00f782 (0xdae8 bytes) 000709 1786839037.886318 write 0x024877 thru 0x034728 (0xfeb2 bytes) 000710 1786839037.888071 punch from 0x013798 to 0x01c120 (0x8989 bytes) 000711 1786839037.891007 fallocate 0x03b1c7 thru 0x03ffff (0x4e39 bytes) 000712 1786839037.893487 trunc from 0x009f0f to 0x016c84 (0xcd76 bytes) 000714 1786839037.896376 mapwrite 0x02f30a thru 0x034ceb (0x59e2 bytes) 000715 1786839037.904368 mapwrite 0x03b50b thru 0x03ffff (0x4af5 bytes) 000717 1786839037.915935 fallocate 0x034622 thru 0x03ffff (0xb9de bytes) 000718 1786839037.918807 punch from 0x031909 to 0x035803 (0x3efb bytes) 000721 1786839037.922666 write 0x028d2c thru 0x02f5c5 (0x689a bytes) 000722 1786839037.923898 fallocate 0x02c003 thru 0x031212 (0x5210 bytes) 000723 1786839037.926456 punch from 0x0047a0 to 0x00ff00 (0xb761 bytes) 000724 1786839037.928436 mapwrite 0x01af79 thru 0x01d320 (0x23a8 bytes) 000725 1786839037.937259 write 0x03f842 thru 0x03ffff (0x7be bytes) 000726 1786839037.937979 trunc from 0x040000 to 0x01d40a (0x22bf5 bytes) 000728 1786839037.943646 mapwrite 0x025f45 thru 0x02d490 (0x754c bytes) 000731 1786839037.957801 punch from 0x0248a0 to 0x025acb (0x122c bytes) 000732 1786839037.959752 mapread 0x021894 thru 0x02d265 (0xb9d2 bytes) 000733 1786839037.962238 trunc from 0x02d491 to 0x005630 (0x27e60 bytes) 000734 1786839037.967073 mapwrite 0x02eb35 thru 0x033caa (0x5176 bytes) 000735 1786839037.982624 trunc from 0x01274c to 0x01ce76 (0xa72b bytes) 000736 1786839037.984971 trunc from 0x033cab to 0x03d3cc (0x9722 bytes) 000737 1786839037.990583 trunc from 0x02abc0 to 0x0370ed (0xc52e bytes) 000738 1786839037.992702 read 0x02a933 thru 0x030c8a (0x6358 bytes) 000739 1786839037.996338 mapwrite 0x00ddf5 thru 0x017504 (0x9710 bytes) 000740 1786839038.011612 mapwrite 0x004ae3 thru 0x00ccef (0x820d bytes) 000741 1786839038.022785 mapread 0x033be4 thru 0x03d3cc (0x97e9 bytes) 000742 1786839038.028591 read 0x00dc36 thru 0x00ea77 (0xe42 bytes) 000743 1786839038.029156 mapread 0x005732 thru 0x008c49 (0x3518 bytes) 000744 1786839038.032899 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d3cd): Mirror 4: 0x2b2c7950 (stale) Mirror 5: 0x2b2c7950 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3cd): Mirror 4: 0x2b2c7950 (stale) Mirror 5: 0x2b2c7950 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3cd: skipped 000745 1786839038.122985 trunc from 0x03d3cd to 0x019723 (0x23ca9 bytes) 000746 1786839038.128547 punch from 0x0057aa to 0x00b71d (0x5f74 bytes) 000747 1786839038.130599 punch from 0x000fb0 to 0x0109e3 (0xfa34 bytes) 000748 1786839038.132467 fallocate 0x03d79b thru 0x03ffff (0x2865 bytes) 000749 1786839038.134346 fallocate 0x01821f thru 0x01b922 (0x3704 bytes) 000750 1786839038.136278 mapwrite 0x00750f thru 0x009f6a (0x2a5c bytes) 000751 1786839038.146316 mapread 0x0117d6 thru 0x017d5f (0x658a bytes) 000752 1786839038.150250 trunc from 0x01b923 to 0x00bdf6 (0xfb2c bytes) 000753 1786839038.155497 fallocate 0x023a93 thru 0x032d2e (0xf29c bytes) 000755 1786839038.157787 trunc from 0x007e3e to 0x00bbf3 (0x3db6 bytes) 000756 1786839038.159902 trunc from 0x00bdf7 to 0x0321ab (0x263b5 bytes) 000757 1786839038.164269 write 0x012322 thru 0x0219c7 (0xf6a6 bytes) 000758 1786839038.168316 write 0x01249a thru 0x0211db (0xed42 bytes) 000759 1786839038.169441 punch from 0x002390 to 0x00a133 (0x7da4 bytes) 000760 1786839038.171568 read 0x003769 thru 0x004ce4 (0x157c bytes) 000761 1786839038.173853 mapwrite 0x00eec8 thru 0x01dc5c (0xed95 bytes) 000762 1786839038.184965 trunc from 0x0321ac to 0x01e4e5 (0x13cc6 bytes) 000763 1786839038.190876 write 0x027aa0 thru 0x030228 (0x8789 bytes) 000764 1786839038.192562 read 0x01c333 thru 0x02238f (0x605d bytes) 000765 1786839038.196650 punch from 0x02b7fc to 0x030228 (0x4a2d bytes) 000766 1786839038.202018 fallocate 0x0371e6 thru 0x03ffff (0x8e1a bytes) 000767 1786839038.204531 punch from 0x005f19 to 0x00f7c7 (0x98af bytes) 000768 1786839038.206924 mapwrite 0x02e50a thru 0x03339d (0x4e94 bytes) 000769 1786839038.221458 punch from 0x02dc17 to 0x02fbb2 (0x1f9c bytes) 000770 1786839038.223841 read 0x0318c5 thru 0x037fcc (0x6708 bytes) 000772 1786839038.224778 trunc from 0x00089c to 0x00dd99 (0xd4fe bytes) 000773 1786839038.227295 mapread 0x037ac6 thru 0x038b6b (0x10a6 bytes) 000775 1786839038.228803 punch from 0x00c352 to 0x015d07 (0x99b6 bytes) 000776 1786839038.231350 read 0x01f68f thru 0x01f8e9 (0x25b bytes) 000777 1786839038.231838 write 0x02808c thru 0x030736 (0x86ab bytes) 000778 1786839038.232726 mapwrite 0x039add thru 0x03cbbf (0x30e3 bytes) 000779 1786839038.239665 trunc from 0x040000 to 0x016445 (0x29bba bytes) 000781 1786839038.245509 read 0x003381 thru 0x00c614 (0x9294 bytes) 000782 1786839038.248936 punch from 0x00d3dd to 0x016445 (0x9069 bytes) 000783 1786839038.251062 read 0x000c7a thru 0x00dd79 (0xd100 bytes) 000785 1786839038.254338 trunc from 0x008f5d to 0x01534a (0xc3ee bytes) 000786 1786839038.256226 trunc from 0x00b9d8 to 0x016445 (0xaa6e bytes) 000787 1786839038.257841 trunc from 0x016446 to 0x03d344 (0x26eff bytes) 000789 1786839038.262614 mapwrite 0x0139ca thru 0x014be8 (0x121f bytes) 000790 1786839038.271674 fallocate 0x01e591 thru 0x02a765 (0xc1d5 bytes) 000791 1786839038.273733 trunc from 0x03d345 to 0x0040f9 (0x3924b bytes) 000793 1786839038.279247 read 0x0013ae thru 0x0040f9 (0x2d4c bytes) 000794 1786839038.281514 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40fa): Mirror 4: 0xadc619a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40fa): Mirror 4: 0xadc619a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40fa: skipped 000795 1786839038.354051 trunc from 0x0040fa to 0x02aa3f (0x26946 bytes) 000796 1786839038.359875 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000797 1786839038.396692 fallocate 0x01c344 thru 0x01e354 (0x2011 bytes) 000798 1786839038.398906 write 0x02b505 thru 0x02f143 (0x3c3f bytes) 000799 1786839038.401083 punch from 0x00e152 to 0x017a88 (0x9937 bytes) 000800 1786839038.403364 mapread 0x02055f thru 0x022be4 (0x2686 bytes) 000801 1786839038.409433 write 0x00621d thru 0x00d6c2 (0x74a6 bytes) 000802 1786839038.412454 mapwrite 0x0370e2 thru 0x03a7b0 (0x36cf bytes) 000803 1786839038.427998 mapread 0x0236a1 thru 0x0253f5 (0x1d55 bytes) 000804 1786839038.428710 mapread 0x012924 thru 0x01de37 (0xb514 bytes) 000805 1786839038.431800 trunc from 0x00c45b to 0x0103c3 (0x3f69 bytes) 000806 1786839038.433951 fallocate 0x00eaa9 thru 0x017f0b (0x9463 bytes) 000807 1786839038.436038 read 0x037522 thru 0x03a7b0 (0x328f bytes) 000808 1786839038.436526 mapwrite 0x014d27 thru 0x01d576 (0x8850 bytes) 000810 1786839038.444105 fallocate 0x03713f thru 0x03ffff (0x8ec1 bytes) 000811 1786839038.446088 mapread 0x02e28e thru 0x0378fb (0x966e bytes) 000812 1786839038.446839 mapwrite 0x03cd26 thru 0x03ffff (0x32da bytes) 000813 1786839038.453511 punch from 0x03355a to 0x03b752 (0x81f9 bytes) 000814 1786839038.455069 trunc from 0x040000 to 0x035658 (0xa9a7 bytes) 000815 1786839038.459063 mapwrite 0x020456 thru 0x022571 (0x211c bytes) 000816 1786839038.463747 mapwrite 0x02a965 thru 0x039128 (0xe7c4 bytes) 000818 1786839038.481085 read 0x00e843 thru 0x014569 (0x5d27 bytes) 000819 1786839038.484580 trunc from 0x039129 to 0x029ee8 (0xf240 bytes) 000820 1786839038.489626 trunc from 0x0136c4 to 0x021082 (0xd9bf bytes) 000821 1786839038.491777 punch from 0x0063b8 to 0x00bd22 (0x596b bytes) 000822 1786839038.493699 write 0x0153d5 thru 0x01e6dd (0x9309 bytes) 000823 1786839038.497050 write 0x009c63 thru 0x00a813 (0xbb1 bytes) 000824 1786839038.500529 read 0x0282f9 thru 0x029ee8 (0x1bf0 bytes) 000825 1786839038.500951 punch from 0x029c34 to 0x029ee8 (0x2b5 bytes) 000826 1786839038.502940 mapread 0x021e87 thru 0x026f9c (0x5116 bytes) 000827 1786839038.506274 trunc from 0x018514 to 0x01be1c (0x3909 bytes) 000828 1786839038.510911 trunc from 0x029ee9 to 0x012922 (0x175c6 bytes) 000829 1786839038.515534 trunc from 0x00729c to 0x0112b0 (0xa015 bytes) 000830 1786839038.517544 trunc from 0x012923 to 0x02fe8c (0x1d56a bytes) 000833 1786839038.523075 trunc from 0x02db54 to 0x02fe8c (0x2339 bytes) 000834 1786839038.524858 punch from 0x00bb65 to 0x017d52 (0xc1ee bytes) 000835 1786839038.526740 fallocate 0x00d46c thru 0x0118b5 (0x444a bytes) 000836 1786839038.528533 mapwrite 0x009da7 thru 0x010a1e (0x6c78 bytes) 000837 1786839038.546443 fallocate 0x00f017 thru 0x00fc59 (0xc43 bytes) 000838 1786839038.548765 punch from 0x00a3d9 to 0x0155d0 (0xb1f8 bytes) 000839 1786839038.550888 trunc from 0x02fe8d to 0x034aa0 (0x4c14 bytes) 000840 1786839038.556559 MIRROR_EXTEND to 2 mirrors 000843 1786839038.740568 mapread 0x01c02f thru 0x01cd85 (0xd57 bytes) 000844 1786839038.746556 mapwrite 0x0101cf thru 0x0166be (0x64f0 bytes) 000845 1786839038.783650 mapwrite 0x020024 thru 0x025ff2 (0x5fcf bytes) 000846 1786839038.794396 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34aa1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34aa1): Mirror 2: 0x70df1659 (stale) CRC-32 checksum value for chunk [0, 0x34aa1): Mirror 1: 0xacbacb92 Mirror 3: 0xacbacb92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34aa1: skipped 000847 1786839039.052430 fallocate 0x0323c6 thru 0x03e2f5 (0xbf30 bytes) 000848 1786839039.065186 write 0x007b11 thru 0x00b901 (0x3df1 bytes) 000849 1786839039.070716 trunc from 0x034aa1 to 0x02d63c (0x7464 bytes) 000851 1786839039.082745 punch from 0x00c059 to 0x01a963 (0xe90b bytes) 000852 1786839039.084842 write 0x02eb92 thru 0x032fce (0x443d bytes) 000853 1786839039.086167 mapread 0x01eb6c thru 0x02dfec (0xf481 bytes) 000854 1786839039.094879 write 0x01c74e thru 0x0219fc (0x52af bytes) 000855 1786839039.095898 mapread 0x004c72 thru 0x00957f (0x490e bytes) 000856 1786839039.099259 trunc from 0x030843 to 0x032fce (0x278c bytes) 000857 1786839039.104043 write 0x00209a thru 0x00b970 (0x98d7 bytes) 000858 1786839039.105423 punch from 0x00edc6 to 0x01dd01 (0xef3c bytes) 000859 1786839039.107662 trunc from 0x032fcf to 0x00d5b5 (0x25a19 bytes) 000860 1786839039.112603 fallocate 0x03b349 thru 0x03da94 (0x274c bytes) 000861 1786839039.114860 fallocate 0x03a3cc thru 0x03ffff (0x5c34 bytes) 000863 1786839039.117926 mapread 0x0123e7 thru 0x01ca80 (0xa69a bytes) 000864 1786839039.126212 read 0x00717a thru 0x00bfcd (0x4e54 bytes) 000865 1786839039.126832 write 0x025cd9 thru 0x034a05 (0xed2d bytes) 000866 1786839039.131869 mapread 0x03f9d7 thru 0x03ffff (0x629 bytes) 000867 1786839039.135780 write 0x01eea8 thru 0x0209f6 (0x1b4f bytes) 000868 1786839039.136696 trunc from 0x02187b to 0x023a2c (0x21b2 bytes) 000869 1786839039.138785 trunc from 0x03f18a to 0x03ffff (0xe76 bytes) 000870 1786839039.140499 mapwrite 0x0251f5 thru 0x02bf39 (0x6d45 bytes) 000872 1786839039.148557 trunc from 0x040000 to 0x022101 (0x1defe bytes) 000873 1786839039.153951 mapwrite 0x006850 thru 0x015c3b (0xf3ec bytes) 000874 1786839039.163817 mapread 0x01a1b0 thru 0x01cde9 (0x2c3a bytes) 000875 1786839039.164532 read 0x01c3b1 thru 0x022101 (0x5d51 bytes) 000876 1786839039.167636 trunc from 0x022102 to 0x027a43 (0x5942 bytes) 000878 1786839039.173231 fallocate 0x0355b8 thru 0x03ffff (0xaa48 bytes) 000879 1786839039.175568 punch from 0x0025a0 to 0x00b404 (0x8e65 bytes) 000881 1786839039.179400 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27a44): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a44): Mirror 2: 0x40c825bc (stale) Mirror 3: 0x60d051f8 (stale) CRC-32 checksum value for chunk [0, 0x27a44): Mirror 1: 0xc1a55174 Mirror 4: 0xc1a55174 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a44: skipped 000882 1786839039.477528 punch from 0x0021e9 to 0x00a951 (0x8769 bytes) 000883 1786839039.489367 mapread 0x009d95 thru 0x016ddf (0xd04b bytes) 000884 1786839039.497738 fallocate 0x0329d6 thru 0x03f7ad (0xcdd8 bytes) 000885 1786839039.500958 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f7ae): Mirror 2: 0xa561758d (stale) Mirror 3: 0x8f8f95cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f7ae): Mirror 2: 0xa561758d (stale) Mirror 3: 0x8f8f95cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f7ae: skipped 000886 1786839039.585739 mapwrite 0x0089cc thru 0x0115e9 (0x8c1e bytes) 000887 1786839039.599253 mapread 0x011085 thru 0x0171f8 (0x6174 bytes) 000889 1786839039.602833 mapread 0x0223fd thru 0x02c0ea (0x9cee bytes) 000890 1786839039.613225 MIRROR_RESYNC 3 mirrors 000891 1786839039.887488 trunc from 0x0034ce to 0x00eb54 (0xb687 bytes) 000892 1786839039.898873 fallocate 0x01da64 thru 0x029369 (0xb906 bytes) 000893 1786839039.901324 read 0x00e935 thru 0x019d6b (0xb437 bytes) 000894 1786839039.907071 fallocate 0x004084 thru 0x010035 (0xbfb2 bytes) 000895 1786839039.909215 fallocate 0x00a986 thru 0x0132f0 (0x896b bytes) 000896 1786839039.912946 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f7ae): Mirror 3: 0xb3673dc0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f7ae): Mirror 3: 0xb3673dc0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f7ae: skipped 000897 1786839039.990039 fallocate 0x00d759 thru 0x01c906 (0xf1ae bytes) 000899 1786839039.992416 mapwrite 0x02f0f5 thru 0x037692 (0x859e bytes) 000900 1786839040.007970 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000902 1786839040.047707 trunc from 0x03f7ae to 0x039681 (0x612c bytes) 000903 1786839040.060255 write 0x02919f thru 0x02d501 (0x4363 bytes) 000904 1786839040.063915 mapwrite 0x01d1da thru 0x021da6 (0x4bcd bytes) 000905 1786839040.076144 punch from 0x0303bf to 0x039681 (0x92c3 bytes) 000906 1786839040.078547 punch from 0x01e9a0 to 0x02e8b0 (0xff11 bytes) 000907 1786839040.081075 fallocate 0x014595 thru 0x01d05f (0x8acb bytes) 000908 1786839040.083581 read 0x0238d6 thru 0x029723 (0x5e4e bytes) 000909 1786839040.086732 read 0x032fd9 thru 0x039681 (0x66a9 bytes) 000911 1786839040.090415 fallocate 0x0194a2 thru 0x02756b (0xe0ca bytes) 000912 1786839040.092664 punch from 0x02d737 to 0x02f261 (0x1b2b bytes) 000913 1786839040.094221 punch from 0x0277e0 to 0x02a7dc (0x2ffd bytes) 000914 1786839040.095962 punch from 0x01e685 to 0x02c414 (0xdd90 bytes) 000915 1786839040.098997 MIRROR_EXTEND to 2 mirrors 000916 1786839040.293192 punch from 0x019d43 to 0x01b287 (0x1545 bytes) 000917 1786839040.303071 punch from 0x00b753 to 0x01b5a5 (0xfe53 bytes) 000918 1786839040.306321 MIRROR_RESYNC 2 mirrors 000919 1786839040.512410 fallocate 0x012d36 thru 0x01535d (0x2628 bytes) 000920 1786839040.521695 mapread 0x02be63 thru 0x037397 (0xb535 bytes) 000921 1786839040.529020 trunc from 0x009a5f to 0x016a5d (0xcfff bytes) 000922 1786839040.530858 fallocate 0x000fae thru 0x0078e2 (0x6935 bytes) 000923 1786839040.532751 fallocate 0x01fb6b thru 0x02a8f5 (0xad8b bytes) 000924 1786839040.535642 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) 000925 1786839040.568501 read 0x028528 thru 0x031310 (0x8de9 bytes) 000926 1786839040.571675 trunc from 0x039682 to 0x015256 (0x2442b bytes) 000927 1786839040.582289 trunc from 0x002d0c to 0x004e49 (0x213e bytes) 000928 1786839040.584176 punch from 0x000bbe to 0x00297b (0x1dbe bytes) 000929 1786839040.585791 read 0x013721 thru 0x015256 (0x1b36 bytes) 000931 1786839040.589675 write 0x03e632 thru 0x03ffff (0x19ce bytes) 000932 1786839040.592452 MIRROR_EXTEND to 2 mirrors 000933 1786839040.773836 trunc from 0x02c2de to 0x0375a0 (0xb2c3 bytes) 000934 1786839040.783134 trunc from 0x040000 to 0x0332ca (0xcd35 bytes) 000935 1786839040.793617 punch from 0x023ce2 to 0x026a31 (0x2d50 bytes) 000936 1786839040.796739 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x332cb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x332cb): Mirror 2: 0x6099da21 (stale) CRC-32 checksum value for chunk [0, 0x332cb): Mirror 1: 0x6099da21 Mirror 3: 0x6099da21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x332cb: skipped 000937 1786839041.025708 punch from 0x00e5e3 to 0x018df2 (0xa810 bytes) 000938 1786839041.037933 mapread 0x02dbaf thru 0x0332ca (0x571c bytes) 000939 1786839041.043843 mapwrite 0x016ad9 thru 0x023f72 (0xd49a bytes) 000940 1786839041.056311 punch from 0x0000f0 to 0x00198f (0x18a0 bytes) 000941 1786839041.059384 MIRROR_RESYNC 3 mirrors 000942 1786839041.259802 fallocate 0x015934 thru 0x025134 (0xf801 bytes) 000943 1786839041.268566 punch from 0x0206d4 to 0x02b312 (0xac3f bytes) 000944 1786839041.270230 trunc from 0x0332cb to 0x01e788 (0x14b42 bytes) 000946 1786839041.279844 mapread 0x017aef thru 0x01d971 (0x5e83 bytes) 000947 1786839041.284995 write 0x018cb7 thru 0x0204f7 (0x7841 bytes) 000948 1786839041.285866 fallocate 0x02dfff thru 0x037dfa (0x9dfc bytes) 000949 1786839041.287807 mapwrite 0x00191a thru 0x00fb2a (0xe211 bytes) 000950 1786839041.297925 read 0x00d910 thru 0x01878d (0xae7e bytes) 000951 1786839041.299942 read 0x010034 thru 0x0111a0 (0x116d bytes) 000952 1786839041.300348 read 0x01af9c thru 0x01c8c6 (0x192b bytes) 000953 1786839041.300803 trunc from 0x037dfb to 0x029313 (0xeae7 bytes) 000954 1786839041.304916 trunc from 0x022406 to 0x029313 (0x6f0e bytes) 000955 1786839041.307472 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x6ac:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29314): Mirror 3: 0x53b150f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29314): Mirror 3: 0x53b150f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29314: skipped 000956 1786839041.376579 read 0x001dcf thru 0x00cc3e (0xae70 bytes) 000957 1786839041.378901 fallocate 0x02b95e thru 0x038689 (0xcd2c bytes) 000958 1786839041.380518 punch from 0x01c99a to 0x02620a (0x9871 bytes) 000959 1786839041.382045 punch from 0x007a7e to 0x013eae (0xc431 bytes) 000960 1786839041.383708 read 0x013a3f thru 0x016699 (0x2c5b bytes) 000961 1786839041.386443 write 0x02ce9d thru 0x03215a (0x52be bytes) 000962 1786839041.389484 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3215b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3215b): Mirror 3: 0x87a6047d (stale) CRC-32 checksum value for chunk [0, 0x3215b): Mirror 1: 0xad829922 Mirror 4: 0xad829922 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3215b: skipped 000963 1786839041.597573 mapread 0x00c957 thru 0x00fc55 (0x32ff bytes) 000964 1786839041.605701 trunc from 0x01c0f9 to 0x022280 (0x6188 bytes) 000965 1786839041.615862 mapread 0x011302 thru 0x01f5da (0xe2d9 bytes) 000966 1786839041.621036 mapwrite 0x0295e2 thru 0x02eea1 (0x58c0 bytes) 000967 1786839041.628495 trunc from 0x028f7b to 0x02de11 (0x4e97 bytes) 000968 1786839041.629952 punch from 0x026877 to 0x03215a (0xb8e4 bytes) 000969 1786839041.631411 trunc from 0x03215b to 0x01fc49 (0x12511 bytes) 000970 1786839041.640861 mapread 0x0111ed thru 0x01a102 (0x8f16 bytes) 000971 1786839041.641567 read 0x00a5f8 thru 0x014e9d (0xa8a6 bytes) 000972 1786839041.641970 read 0x010a3f thru 0x016ffa (0x65bc bytes) 000973 1786839041.642360 trunc from 0x01fc4a to 0x024ee4 (0x529b bytes) 000975 1786839041.647482 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24ee5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ee5): Mirror 3: 0xbfc92ba9 (stale) Mirror 4: 0x2f188e12 (stale) CRC-32 checksum value for chunk [0, 0x24ee5): Mirror 1: 0x6bccd08f Mirror 5: 0x6bccd08f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ee5: skipped 000977 1786839041.847638 write 0x01d221 thru 0x022c61 (0x5a41 bytes) 000978 1786839041.857900 read 0x011135 thru 0x017396 (0x6262 bytes) 000980 1786839041.860430 read 0x0233e3 thru 0x024ee4 (0x1b02 bytes) 000981 1786839041.863196 fallocate 0x03b182 thru 0x03ffff (0x4e7e bytes) 000982 1786839041.864879 punch from 0x00d199 to 0x01994d (0xc7b5 bytes) 000983 1786839041.866513 mapread 0x00d15f thru 0x01ad6d (0xdc0f bytes) 000984 1786839041.884239 trunc from 0x016b73 to 0x01ec63 (0x80f1 bytes) 000985 1786839041.888443 fallocate 0x012b11 thru 0x01a908 (0x7df8 bytes) 000986 1786839041.890767 punch from 0x006160 to 0x012150 (0xbff1 bytes) 000987 1786839041.892898 mapread 0x003796 thru 0x00cb63 (0x93ce bytes) 000988 1786839041.902360 fallocate 0x031795 thru 0x036593 (0x4dff bytes) 000989 1786839041.906018 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1b4babe4 (stale) Mirror 4: 0xaa908592 (stale) Mirror 5: 0xba936837 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe3cc88a2 Mirror 6: 0xe3cc88a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1786839042.117017 fallocate 0x018468 thru 0x02489e (0xc437 bytes) 000994 1786839042.126932 mapread 0x022750 thru 0x02384d (0x10fe bytes) 000995 1786839042.129821 fallocate 0x00ab8d thru 0x011d44 (0x71b8 bytes) 000996 1786839042.131518 trunc from 0x027a45 to 0x0374b6 (0xfa72 bytes) 000998 1786839042.133374 write 0x01bfc8 thru 0x01e47e (0x24b7 bytes) 000999 1786839042.137318 punch from 0x0246c7 to 0x02fca3 (0xb5dd bytes) All operations completed A-OK! PASS 70a (20s) == sanity-flr test 71: check mirror extend parent fid ==== 20:10:43 (1786839043) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x6ee:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:10:47 (1786839047) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0591274 s, 35.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356089 s, 29.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (2s) == sanity-flr test 200a: stress test ===================== 20:10:49 (1786839049) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5141216 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8017888 .. done Extending file size to 5518848 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7285248 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7300032 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8660832 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8720000 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7397472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7447200 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8450752 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8746848 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8580128 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9040928 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9342176 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 64774 64775 64777 64780 64781 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:11:55 (1786839115) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3240768 .. Extending file size to 2613760 .. done Extending file size to 5953536 .. Extending file size to 3807776 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6024832 .. Extending file size to 8517376 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 6058816 .. Extending file size to 8282112 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8626976 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8708864 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8818944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9072768 .. Extending file size to 8523744 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9223104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9105504 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9151648 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 67448 67449 67451 67453 67455 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) 192.168.202.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:13:01 (1786839181) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2568 1268736 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2864 1268440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25848 3581172 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17944 3589076 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43792 7170248 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2568 1268736 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2864 1268440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25848 3581172 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17944 3589076 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43792 7170248 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0441991 s, 47.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0470315 s, 44.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc118c73a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:13:07 (1786839187) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0328168 s, 63.9 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 20:13:10 (1786839190) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:13:12 (1786839192) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0251716 s, 83.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x998:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x998:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x999:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:13:15 (1786839195) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0191416 s, 110 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:13:17 (1786839197) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.022051 s, 95.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x999:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x99a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x999:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x99a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x99b:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:13:41 (1786839221) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169649 s, 61.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x99a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x99d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x99c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x99a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x99d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x99c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x99b:0x0] } PASS 204d (34s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:14:15 (1786839255) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0322694 s, 65.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (36s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:14:51 (1786839291) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg203-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg203-server: Pool lustre.test_204f created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0305508 s, 68.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg203-server: Pool lustre.test_204f destroyed PASS 204f (51s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:15:42 (1786839342) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:15:45 (1786839345) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:15:47 (1786839347) oleg203-server: Pool lustre.test_206 created Adding targets to pool oleg203-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg203-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg203-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg203-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:16:03 (1786839363) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027179 s, 38.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012881 s, 81.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf141cb61 Mirror 3: 0xf141cb61 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:16:06 (1786839366) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:16:07 (1786839367) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:16:08 (1786839368) Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0161735 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xac0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x710000 Read this (4 bytes): BBBB 192.168.202.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg203-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:16:25 (1786839385) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00371178 s, 2.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:16:27 (1786839387) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168232 s, 62.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-15 20:16:28.000000000 -0400 Modify: 2026-08-15 20:16:28.000000000 -0400 Change: 2026-08-15 20:16:28.000000000 -0400 Birth: 2026-08-15 20:16:28.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:16:30 (1786839390) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132056 s, 79.4 MB/s oleg203-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg203-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg203-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (11s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:16:41 (1786839401) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0420937 s, 973 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00583783 s, 702 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 15 20:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 15 20:16 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:16:43 (1786839403) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:16:46 (1786839406) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (2s) == sanity-flr test complete, duration 878 sec ============ 20:16:48 (1786839408) === sanity-flr: start cleanup 20:16:49 (1786839409) === === sanity-flr: finish cleanup 20:16:51 (1786839411) ===