-----============= acceptance-small: sanity-flr ============----- Mon Mar 16 09:37:32 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg327-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:37:39 (1773668259) === oleg327-client.virtnet: executing check_config_client /mnt/lustre oleg327-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg327-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89c802b5a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff89c802b5a000.idle_timeout=debug disable quota as required oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:37:52 (1773668272) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8176] == sanity-flr test 0a: lfs mirror create with -N option == 09:37:52 (1773668272) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:37:58 (1773668278) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:37:59 (1773668279) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:38:00 (1773668280) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:38:07 (1773668287) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:38:08 (1773668288) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:38:09 (1773668289) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89c802b5a000.stats=clear osc.lustre-OST0001-osc-ffff89c802b5a000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:38:11 (1773668291) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:38:14 (1773668294) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:38:17 (1773668297) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:38:18 (1773668298) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 09:38:21 (1773668301) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:38:23 (1773668303) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:38:25 (1773668305) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:38:28 (1773668308) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:38:31 (1773668311) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:38:33 (1773668313) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.431033 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281049 s, 37.3 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:38:36 (1773668316) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0406972 s, 103 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:38:39 (1773668319) Stopping /mnt/lustre-ost1 (opts:) on oleg327-server oleg327-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg327-server: oleg327-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg327-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg327-server oleg327-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg327-server: oleg327-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg327-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (26s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:39:05 (1773668345) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.144302 s, 50.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg327-server oleg327-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg327-server: oleg327-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg327-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (13s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:39:18 (1773668358) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg327-server oleg327-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg327-server: oleg327-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg327-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg327-server oleg327-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg327-server: oleg327-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg327-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:39:44 (1773668384) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18631 s, 44.2 MB/s -rw-r--r-- 1 root root 52428800 Mar 16 09:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.329s user 0m0.000s sys 0m0.144s Stopping /mnt/lustre-ost1 (opts:) on oleg327-server oleg327-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89c802b5a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.281s user 0m0.002s sys 0m0.136s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg327-server: oleg327-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg327-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg327-server oleg327-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.303s user 0m0.000s sys 0m0.132s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg327-server: oleg327-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg327-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg327-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89c802b5a000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:40:11 (1773668411) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:40:12 (1773668412) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:40:14 (1773668414) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 09:40:15 (1773668415) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 09:40:18 (1773668418) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.174021 s, 72.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:40:22 (1773668422) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138904 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014722 s, 71.2 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:40:24 (1773668424) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:40:28 (1773668428) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000644283 s, 1.6 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00246263 s, 416 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000693827 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000688 s, 1.5 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 09:40:31 (1773668431) create test files with size 15751789 .. 6f1f786d66334ac2d130a4b633d8234d - 0509e48d4946a6838fe916edfca4483d - 38b9e31e07775c2d4a27499ccf3b7326 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg327-client.virtnet /mnt/lustre (opts:) Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 09:40:37 (1773668437) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:40:41 (1773668441) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:40:43 (1773668443) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268337 s, 39.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128711 s, 81.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x42:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 09:40:46 (1773668446) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0275682 s, 76.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.104169 s, 40.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0619743 s, 33.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 09:40:51 (1773668451) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:40:52 (1773668452) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:40:53 (1773668453) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011958 s, 87.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 09:40:56 (1773668456) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.103168 s, 30.5 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:40:59 (1773668459) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:41:02 (1773668462) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.109289 s, 95.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg327-client.virtnet /mnt/lustre (opts:) Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:41:20 (1773668480) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff89c80dc99000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0871003 s, 120 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg327-client.virtnet /mnt/lustre (opts:) Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff89c810c74800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:41:24 (1773668484) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:41:36 (1773668496) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:41:50 (1773668510) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 09:41:51 (1773668511) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0881903 s, 35.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0611076 s, 51.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x92f728cd (stale) Mirror 2: 0x92f728cd (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: a07a2e63a09bff8282c3ccce9fd8596a - mirror 1 checksum: ce592283b9142a5871a8d06e286c1205 - mirror 2 checksum: a07a2e63a09bff8282c3ccce9fd8596a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 16 09:41:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 16 09:41:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a07a2e63a09bff8282c3ccce9fd8596a - mirror 1 checksum: ce592283b9142a5871a8d06e286c1205 - mirror 2 checksum: ce592283b9142a5871a8d06e286c1205 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 16 09:41:52 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x53:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 16 09:41:52 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:41:55 (1773668515) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130343 s, 80.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 316416.. 317439: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:41:57 (1773668517) Failing mds1 on oleg327-server Stopping /mnt/lustre-mds1 (opts:) on oleg327-server 09:42:05 (1773668525) shut down facet: mds1 facet_host: oleg327-server facet_failover_host: oleg327-server Failover mds1 to oleg327-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 09:42:21 (1773668541) targets are mounted 09:42:21 (1773668541) facet_failover done oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282242 s, 37.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x56:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:42:37 (1773668557) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0240854 s, 850 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0111702 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:42:46 (1773668566) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.033559 s, 610 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0138591 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (7s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:42:53 (1773668573) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0088007 s, 1.9 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:42:59 (1773668579) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0310243 s, 660 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:43:05 (1773668585) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00680039 s, 602 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0398507 s, 103 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:43:13 (1773668593) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x10:0x0]: Invalid argument (22) PASS 60b (33s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:43:46 (1773668626) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:44:11 (1773668651) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:44:32 (1773668672) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:44:54 (1773668694) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 09:44:59 (1773668699) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55802 mirror split pid 55803 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55802 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 55803 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 09:46:04 (1773668764) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 8765 truncating to largest ever: 0x2e552 000001 1773668765.263678 trunc from 00000000 to 0x02e551 (0x2e552 bytes) truncating to largest ever: 0x3e7f6 000002 1773668765.280675 trunc from 0x02e552 to 0x03e7f5 (0x102a4 bytes) 000003 1773668765.306252 mapwrite 0x02757a thru 0x031981 (0xa408 bytes) 000005 1773668765.329007 write 0x012314 thru 0x020154 (0xde41 bytes) 000006 1773668765.347288 write 0x005be5 thru 0x00bc43 (0x605f bytes) 000008 1773668765.360348 fallocate 0x00c539 thru 0x011243 (0x4d0b bytes) 000009 1773668765.363762 punch from 0x022f85 to 0x0291cf (0x624b bytes) 000011 1773668765.368197 mapwrite 0x01d457 thru 0x01e7dd (0x1387 bytes) 000013 1773668765.383913 write 0x0173a5 thru 0x01d73b (0x6397 bytes) 000014 1773668765.384902 mapwrite 0x03caa4 thru 0x03ffff (0x355c bytes) 000015 1773668765.408689 mapread 0x02cde0 thru 0x038d12 (0xbf33 bytes) 000016 1773668765.413805 punch from 0x03f539 to 0x03ffff (0xac7 bytes) 000017 1773668765.417961 mapread 0x00116c thru 0x004ebc (0x3d51 bytes) 000018 1773668765.418927 mapread 0x0035ce thru 0x005a09 (0x243c bytes) 000019 1773668765.420167 trunc from 0x021512 to 0x02fcac (0xe79b bytes) 000022 1773668765.430323 mapread 0x03a2ad thru 0x03ffff (0x5d53 bytes) 000023 1773668765.436096 read 0x0356e4 thru 0x03ffff (0xa91c bytes) 000024 1773668765.436914 mapwrite 0x01a6dc thru 0x024105 (0x9a2a bytes) 000025 1773668765.457113 trunc from 0x040000 to 0x00423d (0x3bdc2 bytes) 000026 1773668765.467822 write 0x0343fc thru 0x037a75 (0x367a bytes) 000027 1773668765.469154 trunc from 0x037a76 to 0x022952 (0x15123 bytes) 000029 1773668765.485711 read 0x01b8e1 thru 0x022952 (0x7072 bytes) 000030 1773668765.494109 trunc from 0x022953 to 0x022035 (0x91d bytes) 000031 1773668765.517592 MIRROR_EXTEND to 2 mirrors 000032 1773668765.890592 punch from 0x0217be to 0x022035 (0x878 bytes) 000033 1773668765.934570 trunc from 0x00fe08 to 0x01b6e8 (0xb8e1 bytes) 000034 1773668765.940761 mapread 0x016906 thru 0x02023f (0x993a bytes) 000035 1773668765.974106 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x22036): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22036): Mirror 2: 0xb461e4b7 (stale) CRC-32 checksum value for chunk [0, 0x22036): Mirror 1: 0xb461e4b7 Mirror 3: 0xb461e4b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22036: skipped 000036 1773668766.621469 punch from 0x00e034 to 0x014d4c (0x6d19 bytes) 000037 1773668766.660378 fallocate 0x01ba01 thru 0x02271a (0x6d1a bytes) 000038 1773668766.667520 mapread 0x020cfd thru 0x022035 (0x1339 bytes) 000039 1773668766.688074 read 0x004397 thru 0x0080b2 (0x3d1c bytes) 000040 1773668766.695685 mapwrite 0x01dfb0 thru 0x02a64b (0xc69c bytes) 000041 1773668766.743086 trunc from 0x016d12 to 0x01bd97 (0x5086 bytes) 000042 1773668766.747698 write 0x02b84a thru 0x02e104 (0x28bb bytes) 000043 1773668766.749095 trunc from 0x01f1e0 to 0x02c7a9 (0xd5ca bytes) 000045 1773668766.759402 read 0x029a26 thru 0x02b941 (0x1f1c bytes) 000046 1773668766.771413 read 0x02bd39 thru 0x02e104 (0x23cc bytes) 000048 1773668766.772732 mapwrite 0x02b876 thru 0x0365b7 (0xad42 bytes) 000049 1773668766.812347 mapwrite 0x015630 thru 0x01ae6e (0x583f bytes) 000050 1773668766.826162 mapwrite 0x00f97f thru 0x013984 (0x4006 bytes) 000051 1773668766.845167 read 0x02d384 thru 0x031228 (0x3ea5 bytes) 000052 1773668766.845826 trunc from 0x0095ca to 0x018218 (0xec4f bytes) 000053 1773668766.851157 write 0x026775 thru 0x029d0c (0x3598 bytes) 000054 1773668766.852350 trunc from 0x028df8 to 0x02f2e0 (0x64e9 bytes) 000055 1773668766.860563 mapread 0x007970 thru 0x00c625 (0x4cb6 bytes) 000057 1773668766.869714 mapwrite 0x02b1d8 thru 0x0395cd (0xe3f6 bytes) 000058 1773668766.904268 read 0x02c5ce thru 0x02c6a6 (0xd9 bytes) 000059 1773668766.905045 read 0x02c98a thru 0x02e3f1 (0x1a68 bytes) 000060 1773668766.905548 read 0x03841e thru 0x0395cd (0x11b0 bytes) 000062 1773668766.906609 trunc from 0x0395ce to 0x02a478 (0xf155 bytes) 000063 1773668766.918362 fallocate 0x0243b8 thru 0x02c642 (0x828b bytes) 000064 1773668766.923358 mapread 0x022d14 thru 0x02aa89 (0x7d76 bytes) 000065 1773668766.926927 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c643): Mirror 2: 0x1793bb8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c643): Mirror 2: 0x1793bb8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c643: skipped 000066 1773668767.142584 read 0x005c2d thru 0x00bb04 (0x5ed8 bytes) 000067 1773668767.160610 fallocate 0x00e1dd thru 0x012dbe (0x4be2 bytes) 000068 1773668767.164301 fallocate 0x030848 thru 0x033e1f (0x35d8 bytes) 000069 1773668767.170932 trunc from 0x013e39 to 0x0188fc (0x4ac4 bytes) 000070 1773668767.178326 trunc from 0x033e20 to 0x00660e (0x2d811 bytes) 000071 1773668767.204737 punch from 0x006543 to 0x00660e (0xcc bytes) 000072 1773668767.210655 write 0x01c750 thru 0x020ee6 (0x4797 bytes) 000073 1773668767.211562 mapwrite 0x017245 thru 0x01ccc5 (0x5a81 bytes) 000074 1773668767.257504 write 0x00f071 thru 0x016a3f (0x79cf bytes) 000075 1773668767.269207 trunc from 0x009f0b to 0x0154af (0xb5a5 bytes) 000076 1773668767.282721 trunc from 0x020ee7 to 0x0093fe (0x17ae8 bytes) 000077 1773668767.301846 mapwrite 0x027c09 thru 0x02e2d6 (0x66ce bytes) 000078 1773668767.354002 write 0x00861f thru 0x0161a7 (0xdb89 bytes) 000079 1773668767.360718 write 0x027846 thru 0x031306 (0x9ac1 bytes) 000080 1773668767.363333 punch from 0x024d72 to 0x02f25b (0xa4ea bytes) 000082 1773668767.388316 mapwrite 0x02b131 thru 0x0335a8 (0x8478 bytes) 000084 1773668767.436201 trunc from 0x0335a9 to 0x025f76 (0xd632 bytes) 000085 1773668767.460096 trunc from 0x01a8d6 to 0x01ed2d (0x4458 bytes) 000087 1773668767.464285 fallocate 0x02006e thru 0x022c14 (0x2ba7 bytes) 000089 1773668767.472406 read 0x01a08b thru 0x0228db (0x8851 bytes) 000090 1773668767.487376 punch from 0x001d56 to 0x00bf2d (0xa1d8 bytes) 000091 1773668767.495169 read 0x0180ba thru 0x018d02 (0xc49 bytes) 000093 1773668767.498789 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) 000094 1773668767.560059 read 0x00c923 thru 0x0134d0 (0x6bae bytes) 000095 1773668767.569786 mapread 0x000989 thru 0x00e888 (0xdf00 bytes) 000096 1773668767.595498 mapread 0x001c8c thru 0x0079e6 (0x5d5b bytes) 000097 1773668767.596628 trunc from 0x025f77 to 0x007de6 (0x1e190 bytes) 000098 1773668767.623642 read 0x0035c1 thru 0x007de6 (0x4826 bytes) 000099 1773668767.624431 trunc from 0x001f2c to 0x007de6 (0x5ebb bytes) 000101 1773668767.630537 mapwrite 0x01f875 thru 0x0248f4 (0x5080 bytes) 000102 1773668767.668358 MIRROR_EXTEND to 2 mirrors 000103 1773668768.048502 trunc from 0x0248f5 to 0x02e028 (0x9734 bytes) 000104 1773668768.097538 write 0x00660f thru 0x015194 (0xeb86 bytes) 000105 1773668768.105591 read 0x00ada6 thru 0x00c89f (0x1afa bytes) 000106 1773668768.106100 fallocate 0x00dd2a thru 0x01bb1d (0xddf4 bytes) 000107 1773668768.110398 trunc from 0x02e029 to 0x00e629 (0x1f9ff bytes) 000108 1773668768.140554 trunc from 0x00c421 to 0x00e629 (0x2209 bytes) 000109 1773668768.153980 punch from 0x007d6c to 0x00e629 (0x68be bytes) 000110 1773668768.164105 mapread 0x005858 thru 0x00bf58 (0x6701 bytes) 000112 1773668768.175672 trunc from 0x00e62a to 0x00fedb (0x18b2 bytes) 000113 1773668768.191678 mapwrite 0x035dfb thru 0x037a4b (0x1c51 bytes) 000114 1773668768.226011 write 0x00977a thru 0x00fdfb (0x6682 bytes) 000115 1773668768.231888 trunc from 0x017d91 to 0x01baa3 (0x3d13 bytes) 000116 1773668768.235846 trunc from 0x037a4c to 0x004223 (0x33828 bytes) 000117 1773668768.253597 mapread 0x0001c7 thru 0x004223 (0x405d bytes) 000118 1773668768.260426 trunc from 0x004224 to 0x00d827 (0x9604 bytes) 000119 1773668768.287341 mapwrite 0x03d2a6 thru 0x03ffff (0x2d5a bytes) 000120 1773668768.326187 punch from 0x00a826 to 0x00ec95 (0x4470 bytes) 000121 1773668768.332201 punch from 0x032fac to 0x034789 (0x17de bytes) 000122 1773668768.336725 punch from 0x03126f to 0x033ac1 (0x2853 bytes) 000123 1773668768.342614 read 0x00e1bf thru 0x019c47 (0xba89 bytes) 000124 1773668768.351483 write 0x019125 thru 0x025273 (0xc14f bytes) 000125 1773668768.353079 read 0x03ff8f thru 0x03ffff (0x71 bytes) 000126 1773668768.354285 punch from 0x00ca7f to 0x01af63 (0xe4e5 bytes) 000127 1773668768.367599 fallocate 0x01bad6 thru 0x02b2a9 (0xf7d4 bytes) 000128 1773668768.381078 mapread 0x008499 thru 0x00b470 (0x2fd8 bytes) 000129 1773668768.382833 mapread 0x0293ca thru 0x033c63 (0xa89a bytes) 000130 1773668768.389231 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa63b8f20 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x617bd4c Mirror 3: 0x617bd4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000131 1773668769.021264 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa63b8f20 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x617bd4c Mirror 3: 0x617bd4c Mirror 4: 0x617bd4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000132 1773668769.607768 mapwrite 0x019190 thru 0x024433 (0xb2a4 bytes) 000133 1773668769.705411 trunc from 0x040000 to 0x032548 (0xdab7 bytes) 000135 1773668769.752033 mapread 0x014733 thru 0x014c70 (0x53e bytes) 000136 1773668769.753036 punch from 0x00663a to 0x0152b4 (0xec7b bytes) 000137 1773668769.759035 mapread 0x02fe1d thru 0x032548 (0x272c bytes) fallocating to largest ever: 0x40000 000138 1773668769.769375 fallocate 0x03b5f9 thru 0x03ffff (0x4a07 bytes) 000139 1773668769.773425 fallocate 0x026a8c thru 0x0350cf (0xe644 bytes) 000140 1773668769.777490 read 0x0138b2 thru 0x01e01d (0xa76c bytes) 000141 1773668769.781477 mapread 0x03f821 thru 0x03ffff (0x7df bytes) 000142 1773668769.787170 trunc from 0x040000 to 0x01f697 (0x20968 bytes) 000143 1773668769.800439 write 0x03a290 thru 0x03f0c3 (0x4e34 bytes) 000144 1773668769.805379 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f0c4): Mirror 2: 0x84584aa3 (stale) Mirror 4: 0x38f68f70 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0c4): Mirror 2: 0x84584aa3 (stale) Mirror 4: 0x38f68f70 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0c4: skipped 000145 1773668770.112013 write 0x00488f thru 0x00a718 (0x5e8a bytes) 000146 1773668770.149617 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f0c4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f0c4): Mirror 2: 0x84584aa3 (stale) Mirror 4: 0x38f68f70 (stale) CRC-32 checksum value for chunk [0, 0x3f0c4): Mirror 1: 0xbbcfaf40 Mirror 5: 0xbbcfaf40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f0c4: skipped 000147 1773668771.071760 trunc from 0x03f0c4 to 0x00e115 (0x30fae bytes) 000148 1773668771.106394 mapread 0x008179 thru 0x00e115 (0x5f9d bytes) 000149 1773668771.129847 trunc from 0x006458 to 0x00e115 (0x7cbe bytes) 000151 1773668771.135638 trunc from 0x00646f to 0x00821e (0x1db0 bytes) 000152 1773668771.143081 punch from 0x0083d3 to 0x00e115 (0x5d43 bytes) 000153 1773668771.149758 write 0x021b20 thru 0x030e50 (0xf331 bytes) 000154 1773668771.156481 write 0x015113 thru 0x0192ae (0x419c bytes) 000155 1773668771.171734 punch from 0x01b11e to 0x024590 (0x9473 bytes) 000156 1773668771.189041 read 0x00dae6 thru 0x01a038 (0xc553 bytes) 000157 1773668771.201704 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30e51): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e51): Mirror 2: 0xb8db16a4 (stale) Mirror 4: 0xb40969cf (stale) Mirror 5: 0xcb95b56c (stale) CRC-32 checksum value for chunk [0, 0x30e51): Mirror 1: 0x20771ecf Mirror 6: 0x20771ecf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e51: skipped 000158 1773668772.069999 punch from 0x0294e5 to 0x029891 (0x3ad bytes) 000159 1773668772.103456 trunc from 0x030e51 to 0x02d715 (0x373b bytes) 000160 1773668772.161247 write 0x0187aa thru 0x01abda (0x2431 bytes) 000161 1773668772.184154 MIRROR_RESYNC 5 mirrors 000162 1773668773.572143 write 0x016448 thru 0x018ed5 (0x2a8e bytes) 000163 1773668773.630766 trunc from 0x017e5d to 0x027cb8 (0xfe5c bytes) 000164 1773668773.646273 fallocate 0x02ce10 thru 0x038452 (0xb643 bytes) 000165 1773668773.654245 read 0x00100a thru 0x00eea3 (0xde9a bytes) 000166 1773668773.666087 fallocate 0x038cda thru 0x03ffff (0x7326 bytes) 000167 1773668773.671641 trunc from 0x038453 to 0x01622d (0x22225 bytes) 000168 1773668773.696648 trunc from 0x005b16 to 0x007490 (0x197b bytes) 000169 1773668773.702365 read 0x006747 thru 0x00d1fe (0x6ab8 bytes) 000170 1773668773.711608 mapwrite 0x0368bc thru 0x03ffff (0x9744 bytes) 000171 1773668773.763995 punch from 0x0286d7 to 0x02baa2 (0x33cc bytes) 000172 1773668773.770088 mapread 0x0398d5 thru 0x03ffff (0x672b bytes) 000174 1773668773.774091 mapread 0x030c04 thru 0x0390ca (0x84c7 bytes) 000175 1773668773.776479 trunc from 0x040000 to 0x01fc21 (0x203de bytes) 000176 1773668773.805372 mapread 0x008ae0 thru 0x00a53d (0x1a5e bytes) 000177 1773668773.813419 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1fc22): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fc22): Mirror 2: 0xaaf06362 (stale) Mirror 4: 0xaaf06362 (stale) Mirror 5: 0xaaf06362 (stale) Mirror 6: 0xaaf06362 (stale) CRC-32 checksum value for chunk [0, 0x1fc22): Mirror 1: 0x30097eec Mirror 7: 0x30097eec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fc22: skipped 000178 1773668774.852058 read 0x01f377 thru 0x01fc21 (0x8ab bytes) 000179 1773668774.869274 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1fc22): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fc22): Mirror 2: 0xaaf06362 (stale) Mirror 4: 0xaaf06362 (stale) Mirror 5: 0xaaf06362 (stale) Mirror 6: 0xaaf06362 (stale) CRC-32 checksum value for chunk [0, 0x1fc22): Mirror 1: 0x30097eec Mirror 7: 0x30097eec Mirror 8: 0x30097eec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fc22: skipped 000182 1773668775.929294 read 0x007496 thru 0x012b57 (0xb6c2 bytes) 000183 1773668775.948398 write 0x000b6d thru 0x00bffa (0xb48e bytes) 000184 1773668776.017967 trunc from 0x01d4b5 to 0x01fc21 (0x276d bytes) 000185 1773668776.022409 write 0x031ebf thru 0x036987 (0x4ac9 bytes) 000186 1773668776.024094 punch from 0x0323fc to 0x033950 (0x1555 bytes) 000187 1773668776.051435 mapread 0x018f01 thru 0x01e997 (0x5a97 bytes) 000189 1773668776.066404 read 0x020c2f thru 0x02a352 (0x9724 bytes) 000190 1773668776.080604 write 0x03da58 thru 0x03fc59 (0x2202 bytes) 000191 1773668776.082624 fallocate 0x034f7d thru 0x03ffff (0xb083 bytes) 000192 1773668776.088450 read 0x00d6e1 thru 0x0112dd (0x3bfd bytes) 000193 1773668776.089527 mapread 0x01da52 thru 0x022437 (0x49e6 bytes) 000194 1773668776.091384 fallocate 0x01400d thru 0x01a565 (0x6559 bytes) 000195 1773668776.097402 read 0x036d98 thru 0x03b169 (0x43d2 bytes) 000196 1773668776.105028 mapwrite 0x03ca12 thru 0x03ffff (0x35ee bytes) 000199 1773668776.157821 trunc from 0x040000 to 0x01df0b (0x220f4 bytes) 000200 1773668776.181652 MIRROR_RESYNC 7 mirrors 000203 1773668777.552648 fallocate 0x005c68 thru 0x006791 (0xb2a bytes) 000204 1773668777.586320 punch from 0x012e4d to 0x01df0b (0xb0bf bytes) 000205 1773668777.594071 punch from 0x009027 to 0x00e92f (0x5909 bytes) 000206 1773668777.603464 mapwrite 0x029061 thru 0x02d841 (0x47e1 bytes) 000207 1773668777.701830 read 0x011e24 thru 0x0216a2 (0xf87f bytes) 000208 1773668777.724568 mapread 0x00e447 thru 0x01d831 (0xf3eb bytes) 000209 1773668777.742614 write 0x031aa0 thru 0x03ffff (0xe560 bytes) 000210 1773668777.744844 trunc from 0x040000 to 0x014f66 (0x2b099 bytes) 000211 1773668777.777844 write 0x01d877 thru 0x027434 (0x9bbe bytes) 000212 1773668777.779920 punch from 0x01dcbe to 0x020b45 (0x2e88 bytes) 000213 1773668777.804798 trunc from 0x0014d6 to 0x0074b3 (0x5fde bytes) 000214 1773668777.813947 write 0x002497 thru 0x00e5ad (0xc117 bytes) 000215 1773668777.828635 trunc from 0x027435 to 0x01a1f1 (0xd243 bytes) 000217 1773668777.845007 mapwrite 0x0119d3 thru 0x011e61 (0x48f bytes) 000218 1773668777.861594 trunc from 0x011851 to 0x01a1f1 (0x89a1 bytes) 000220 1773668777.867169 read 0x014772 thru 0x017884 (0x3113 bytes) 000222 1773668777.874835 trunc from 0x01a1f2 to 0x019197 (0x105a bytes) 000223 1773668777.900104 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19198): Mirror 2: 0x63fe9704 (stale) Mirror 4: 0x63fe9704 (stale) Mirror 5: 0x63fe9704 (stale) Mirror 6: 0x63fe9704 (stale) Mirror 7: 0x63fe9704 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19198): Mirror 2: 0x63fe9704 (stale) Mirror 4: 0x63fe9704 (stale) Mirror 5: 0x63fe9704 (stale) Mirror 6: 0x63fe9704 (stale) Mirror 7: 0x63fe9704 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19198: skipped 000224 1773668778.274942 fallocate 0x00a300 thru 0x00de24 (0x3b25 bytes) 000225 1773668778.281141 mapwrite 0x0136fd thru 0x01d801 (0xa105 bytes) 000226 1773668778.336947 trunc from 0x00aae3 to 0x00f90f (0x4e2d bytes) 000227 1773668778.340751 read 0x01a98b thru 0x01bd01 (0x1377 bytes) 000229 1773668778.341824 mapwrite 0x02853c thru 0x02da72 (0x5537 bytes) 000230 1773668778.376110 read 0x0036d1 thru 0x00ac02 (0x7532 bytes) 000231 1773668778.385360 mapwrite 0x00797b thru 0x015cbb (0xe341 bytes) 000233 1773668778.405117 write 0x009adb thru 0x0148b9 (0xaddf bytes) 000234 1773668778.407395 mapwrite 0x004fd4 thru 0x00e4cc (0x94f9 bytes) 000235 1773668778.429071 fallocate 0x013acb thru 0x01ae88 (0x73be bytes) 000236 1773668778.435782 mapread 0x0275b8 thru 0x02da72 (0x64bb bytes) 000237 1773668778.437069 write 0x0034e8 thru 0x00fde6 (0xc8ff bytes) 000238 1773668778.438760 read 0x005d6d thru 0x0119eb (0xbc7f bytes) 000239 1773668778.439728 trunc from 0x02da73 to 0x013c5e (0x19e14 bytes) 000240 1773668778.469331 trunc from 0x009ed9 to 0x01292e (0x8a56 bytes) 000241 1773668778.488805 read 0x011bd2 thru 0x013c5e (0x208d bytes) 000242 1773668778.509517 punch from 0x00a21b to 0x013c5e (0x9a44 bytes) 000243 1773668778.526432 write 0x007a0a thru 0x017074 (0xf66b bytes) 000244 1773668778.530562 trunc from 0x017075 to 0x008f5a (0xe11a bytes) 000245 1773668778.550808 punch from 0x000301 to 0x008f5a (0x8c5a bytes) 000246 1773668778.578297 write 0x01e244 thru 0x02a29f (0xc05c bytes) 000248 1773668778.587817 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 2: 0x1aef0037 (stale) Mirror 4: 0x1aef0037 (stale) Mirror 6: 0x1aef0037 (stale) Mirror 7: 0x1aef0037 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 2: 0x1aef0037 (stale) Mirror 4: 0x1aef0037 (stale) Mirror 6: 0x1aef0037 (stale) Mirror 7: 0x1aef0037 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2a0: skipped 000249 1773668779.008369 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 2: 0x1aef0037 (stale) Mirror 4: 0x1aef0037 (stale) Mirror 6: 0x1aef0037 (stale) Mirror 7: 0x1aef0037 (stale) CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 1: 0x551a091f Mirror 8: 0x551a091f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2a0: skipped 000250 1773668779.837157 mapread 0x00154c thru 0x00fc61 (0xe716 bytes) 000251 1773668779.872343 punch from 0x010799 to 0x01233e (0x1ba6 bytes) 000252 1773668779.905019 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 4: 0x1aef0037 (stale) Mirror 6: 0x1aef0037 (stale) Mirror 7: 0x1aef0037 (stale) Mirror 8: 0x551a091f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2a0): Mirror 4: 0x1aef0037 (stale) Mirror 6: 0x1aef0037 (stale) Mirror 7: 0x1aef0037 (stale) Mirror 8: 0x551a091f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2a0: skipped 000253 1773668780.194955 mapread 0x0190af thru 0x019e5d (0xdaf bytes) 000254 1773668780.199231 trunc from 0x005d16 to 0x00b32f (0x561a bytes) 000255 1773668780.207471 read 0x0072cd thru 0x00b6bc (0x43f0 bytes) 000256 1773668780.214631 trunc from 0x02a2a0 to 0x0298c0 (0x9df bytes) 000257 1773668780.228547 trunc from 0x00c431 to 0x0100be (0x3c8e bytes) 000258 1773668780.233390 punch from 0x0157ce to 0x023cc1 (0xe4f4 bytes) 000259 1773668780.239437 trunc from 0x0298c1 to 0x036754 (0xce94 bytes) 000260 1773668780.256761 mapwrite 0x03d740 thru 0x03ffff (0x28c0 bytes) 000261 1773668780.307436 mapwrite 0x03bfca thru 0x03de3f (0x1e76 bytes) 000262 1773668780.330837 read 0x03488e thru 0x03b69f (0x6e12 bytes) 000263 1773668780.366658 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf33f1d84 (stale) Mirror 6: 0xf33f1d84 (stale) Mirror 7: 0xf33f1d84 (stale) Mirror 8: 0xcc27156c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x78e22b59 Mirror 9: 0x78e22b59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000264 1773668781.137792 trunc from 0x040000 to 0x00c67f (0x33980 bytes) 000267 1773668781.255880 trunc from 0x0088ed to 0x00c67f (0x3d93 bytes) 000268 1773668781.267200 MIRROR_RESYNC 6 mirrors 000269 1773668782.232804 punch from 0x006dba to 0x00c67f (0x58c6 bytes) 000270 1773668782.265277 mapwrite 0x02a5cb thru 0x0369e0 (0xc416 bytes) 000271 1773668782.350061 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x369e1): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369e1): Mirror 4: 0x43726f19 (stale) Mirror 6: 0x43726f19 (stale) Mirror 7: 0x43726f19 (stale) Mirror 8: 0x43726f19 (stale) Mirror 9: 0x43726f19 (stale) CRC-32 checksum value for chunk [0, 0x369e1): Mirror 1: 0xe19723d Mirror 10: 0xe19723d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369e1: skipped 000272 1773668783.123100 trunc from 0x0369e1 to 0x026513 (0x104cd bytes) 000273 1773668783.208644 read 0x01d08d thru 0x026513 (0x9487 bytes) 000274 1773668783.219421 mapread 0x01514a thru 0x01c35f (0x7216 bytes) 000275 1773668783.245765 trunc from 0x026514 to 0x00e0e9 (0x1842a bytes) 000276 1773668783.269362 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe0ea): Mirror 4: 0xb6ca18ee (stale) Mirror 6: 0xb6ca18ee (stale) Mirror 7: 0xb6ca18ee (stale) Mirror 8: 0xb6ca18ee (stale) Mirror 10: 0xb6ca18ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe0ea): Mirror 4: 0xb6ca18ee (stale) Mirror 6: 0xb6ca18ee (stale) Mirror 7: 0xb6ca18ee (stale) Mirror 8: 0xb6ca18ee (stale) Mirror 10: 0xb6ca18ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe0ea: skipped 000277 1773668783.602554 punch from 0x00460b to 0x00e0e9 (0x9adf bytes) 000278 1773668783.612036 mapwrite 0x003e3d thru 0x007220 (0x33e4 bytes) 000280 1773668783.653590 write 0x03565f thru 0x03ffff (0xa9a1 bytes) 000281 1773668783.661804 mapread 0x0126d5 thru 0x01beea (0x9816 bytes) 000282 1773668783.694045 punch from 0x0164ee to 0x025609 (0xf11c bytes) 000283 1773668783.708325 trunc from 0x040000 to 0x01fc09 (0x203f6 bytes) 000284 1773668783.731281 mapwrite 0x02c275 thru 0x035128 (0x8eb4 bytes) 000286 1773668783.790393 trunc from 0x020a6b to 0x02ae3e (0xa3d4 bytes) 000287 1773668783.796028 mapwrite 0x03f988 thru 0x03ffff (0x678 bytes) 000288 1773668783.845741 trunc from 0x040000 to 0x00f42c (0x30bd3 bytes) 000289 1773668783.868036 fallocate 0x017f7c thru 0x0272ab (0xf330 bytes) 000290 1773668783.872949 punch from 0x004ba7 to 0x00c805 (0x7c5f bytes) 000291 1773668783.882757 trunc from 0x00e6a3 to 0x00f42c (0xd8a bytes) 000292 1773668783.888637 mapwrite 0x001ee5 thru 0x002c16 (0xd32 bytes) 000293 1773668783.907613 fallocate 0x03cbf3 thru 0x03dcbd (0x10cb bytes) 000294 1773668783.913768 punch from 0x003df3 to 0x00859f (0x47ad bytes) 000295 1773668783.921423 MIRROR_RESYNC 6 mirrors 000296 1773668784.882785 trunc from 0x01bba5 to 0x01f980 (0x3ddc bytes) 000297 1773668784.914327 mapwrite 0x03461d thru 0x03e08e (0x9a72 bytes) 000298 1773668784.986935 MIRROR_RESYNC 6 mirrors 000299 1773668785.955183 punch from 0x000857 to 0x001856 (0x1000 bytes) 000300 1773668785.991095 trunc from 0x009019 to 0x00e6d3 (0x56bb bytes) 000301 1773668786.002330 punch from 0x021827 to 0x02ec39 (0xd413 bytes) 000302 1773668786.016845 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e08f): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e08f): Mirror 4: 0xe71a12ba (stale) Mirror 6: 0xe71a12ba (stale) Mirror 7: 0xe71a12ba (stale) Mirror 8: 0xe71a12ba (stale) Mirror 10: 0xe71a12ba (stale) CRC-32 checksum value for chunk [0, 0x3e08f): Mirror 1: 0xe71a12ba Mirror 11: 0xe71a12ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e08f: skipped 000303 1773668786.948270 read 0x036381 thru 0x03e08e (0x7d0e bytes) 000304 1773668786.967626 read 0x02b034 thru 0x03458f (0x955c bytes) 000305 1773668786.985612 trunc from 0x03e08f to 0x024580 (0x19b0e bytes) 000306 1773668787.045311 fallocate 0x005bf5 thru 0x0136ce (0xdada bytes) 000307 1773668787.054119 fallocate 0x02e79a thru 0x03b2ed (0xcb54 bytes) 000308 1773668787.063452 MIRROR_RESYNC 7 mirrors 000309 1773668788.199578 write 0x02bd3a thru 0x02c3c8 (0x68f bytes) 000310 1773668788.243625 punch from 0x022f65 to 0x0253b1 (0x244d bytes) 000312 1773668788.256096 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c3c9): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3c9): Mirror 4: 0xd56485a3 (stale) Mirror 6: 0xd56485a3 (stale) Mirror 7: 0xd56485a3 (stale) Mirror 8: 0xd56485a3 (stale) Mirror 10: 0xd56485a3 (stale) Mirror 11: 0xd56485a3 (stale) CRC-32 checksum value for chunk [0, 0x2c3c9): Mirror 1: 0xfde0f86 Mirror 12: 0xfde0f86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3c9: skipped 000313 1773668789.260631 write 0x039eaf thru 0x03ffff (0x6151 bytes) 000314 1773668789.327817 write 0x0318e9 thru 0x0383ba (0x6ad2 bytes) 000315 1773668789.349010 punch from 0x00c207 to 0x01adaf (0xeba9 bytes) 000316 1773668789.356919 trunc from 0x040000 to 0x001170 (0x3ee8f bytes) 000317 1773668789.384734 mapread 0x0001b1 thru 0x001170 (0xfc0 bytes) 000318 1773668789.390405 punch from 0x000ff8 to 0x001170 (0x179 bytes) 000319 1773668789.400527 trunc from 0x001171 to 0x016580 (0x15410 bytes) 000320 1773668789.427197 trunc from 0x010b77 to 0x014fea (0x4474 bytes) 000321 1773668789.432194 trunc from 0x016581 to 0x02f9cb (0x1944b bytes) 000323 1773668789.454564 write 0x00baea thru 0x01152c (0x5a43 bytes) 000324 1773668789.464010 write 0x01c629 thru 0x029989 (0xd361 bytes) 000327 1773668789.478915 mapread 0x014e54 thru 0x0188f5 (0x3aa2 bytes) 000328 1773668789.485536 trunc from 0x02f9cc to 0x037693 (0x7cc8 bytes) 000330 1773668789.502329 mapwrite 0x002374 thru 0x004eb6 (0x2b43 bytes) 000331 1773668789.540890 read 0x013cc9 thru 0x015767 (0x1a9f bytes) 000332 1773668789.542444 trunc from 0x037694 to 0x0314f2 (0x61a1 bytes) 000334 1773668789.563945 punch from 0x02a52c to 0x02a6bb (0x190 bytes) 000335 1773668789.567953 read 0x0242a5 thru 0x02ecf4 (0xaa50 bytes) 000336 1773668789.577964 mapread 0x00ec00 thru 0x016916 (0x7d17 bytes) 000337 1773668789.579322 mapread 0x0197b9 thru 0x01aac5 (0x130d bytes) 000338 1773668789.580316 trunc from 0x001fd7 to 0x00a11a (0x8144 bytes) 000339 1773668789.594760 read 0x017dd3 thru 0x0186d7 (0x905 bytes) 000340 1773668789.595554 mapwrite 0x03d4f5 thru 0x03d928 (0x434 bytes) 000341 1773668789.684829 fallocate 0x0218f9 thru 0x02dea6 (0xc5ae bytes) 000342 1773668789.699329 trunc from 0x03d929 to 0x037831 (0x60f7 bytes) 000343 1773668789.721303 mapwrite 0x00a3dc thru 0x00efe6 (0x4c0b bytes) 000344 1773668789.738200 trunc from 0x0027d9 to 0x007b8a (0x53b2 bytes) 000345 1773668789.742072 mapread 0x001808 thru 0x00b23c (0x9a35 bytes) 000346 1773668789.778073 write 0x003c2f thru 0x00aaf0 (0x6ec2 bytes) 000348 1773668789.781595 read 0x00315e thru 0x010d06 (0xdba9 bytes) 000349 1773668789.783536 punch from 0x008e79 to 0x010039 (0x71c1 bytes) 000351 1773668789.826936 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37832): Mirror 4: 0xfc09e02c (stale) Mirror 6: 0xfc09e02c (stale) Mirror 7: 0xfc09e02c (stale) Mirror 8: 0xfc09e02c (stale) Mirror 10: 0xfc09e02c (stale) Mirror 12: 0x33a1e7b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37832): Mirror 4: 0xfc09e02c (stale) Mirror 6: 0xfc09e02c (stale) Mirror 7: 0xfc09e02c (stale) Mirror 8: 0xfc09e02c (stale) Mirror 10: 0xfc09e02c (stale) Mirror 12: 0x33a1e7b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37832: skipped 000352 1773668790.335232 fallocate 0x02684c thru 0x02934c (0x2b01 bytes) 000353 1773668790.349331 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37832): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37832): Mirror 4: 0xfc09e02c (stale) Mirror 6: 0xfc09e02c (stale) Mirror 7: 0xfc09e02c (stale) Mirror 8: 0xfc09e02c (stale) Mirror 10: 0xfc09e02c (stale) Mirror 12: 0x33a1e7b3 (stale) CRC-32 checksum value for chunk [0, 0x37832): Mirror 1: 0x502d6b79 Mirror 13: 0x502d6b79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37832: skipped 000354 1773668791.354185 fallocate 0x013933 thru 0x0147ec (0xeba bytes) 000355 1773668791.409125 trunc from 0x037832 to 0x0129bc (0x24e75 bytes) 000356 1773668791.452323 read 0x008a67 thru 0x00a247 (0x17e1 bytes) 000357 1773668791.470119 mapwrite 0x021c61 thru 0x0232e4 (0x1684 bytes) 000358 1773668791.521386 write 0x01c6f4 thru 0x026e5e (0xa76b bytes) 000359 1773668791.523961 mapread 0x01974f thru 0x026e5e (0xd710 bytes) 000360 1773668791.534605 mapread 0x012587 thru 0x014878 (0x22f2 bytes) 000361 1773668791.536645 mapwrite 0x017c20 thru 0x01d498 (0x5879 bytes) 000362 1773668791.564681 MIRROR_RESYNC 8 mirrors 000363 1773668793.558703 read 0x018633 thru 0x01b92f (0x32fd bytes) 000364 1773668793.566671 fallocate 0x024542 thru 0x026997 (0x2456 bytes) 000365 1773668793.614954 mapwrite 0x0354a1 thru 0x03e925 (0x9485 bytes) 000366 1773668793.671547 punch from 0x00b3b0 to 0x014263 (0x8eb4 bytes) 000367 1773668793.675655 read 0x03e456 thru 0x03e925 (0x4d0 bytes) 000368 1773668793.676422 punch from 0x01916c to 0x01a513 (0x13a8 bytes) 000369 1773668793.680614 write 0x032592 thru 0x03e3e1 (0xbe50 bytes) 000370 1773668793.682891 write 0x03ea21 thru 0x03ffff (0x15df bytes) 000371 1773668793.684492 mapread 0x03f859 thru 0x03ffff (0x7a7 bytes) 000372 1773668793.685606 mapwrite 0x002c20 thru 0x00dd31 (0xb112 bytes) 000373 1773668793.710308 fallocate 0x00dbba thru 0x01d9a0 (0xfde7 bytes) 000374 1773668793.718654 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x813ccf98 (stale) Mirror 7: 0x813ccf98 (stale) Mirror 8: 0x813ccf98 (stale) Mirror 10: 0x813ccf98 (stale) Mirror 12: 0x813ccf98 (stale) Mirror 13: 0x813ccf98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x813ccf98 (stale) Mirror 7: 0x813ccf98 (stale) Mirror 8: 0x813ccf98 (stale) Mirror 10: 0x813ccf98 (stale) Mirror 12: 0x813ccf98 (stale) Mirror 13: 0x813ccf98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000375 1773668794.169681 punch from 0x007012 to 0x014c2c (0xdc1b bytes) 000376 1773668794.176229 punch from 0x0185f3 to 0x01b1bf (0x2bcd bytes) 000377 1773668794.180574 mapread 0x02cdd1 thru 0x031755 (0x4985 bytes) 000378 1773668794.203831 punch from 0x01dca4 to 0x02c395 (0xe6f2 bytes) 000379 1773668794.208917 trunc from 0x028705 to 0x02fd2a (0x7626 bytes) 000380 1773668794.218164 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x813ccf98 (stale) Mirror 7: 0x813ccf98 (stale) Mirror 8: 0x813ccf98 (stale) Mirror 10: 0x813ccf98 (stale) Mirror 12: 0x813ccf98 (stale) Mirror 13: 0x813ccf98 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x306b5458 Mirror 14: 0x306b5458 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000381 1773668795.110047 trunc from 0x033f4f to 0x03ffff (0xc0b1 bytes) 000382 1773668795.165564 mapwrite 0x0174c8 thru 0x02675b (0xf294 bytes) 000383 1773668795.253328 trunc from 0x040000 to 0x022554 (0x1daab bytes) 000384 1773668795.309393 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22555): Mirror 6: 0x7f84e1b7 (stale) Mirror 7: 0x7f84e1b7 (stale) Mirror 10: 0x7f84e1b7 (stale) Mirror 12: 0x7f84e1b7 (stale) Mirror 13: 0x7f84e1b7 (stale) Mirror 14: 0xc0c539e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22555): Mirror 6: 0x7f84e1b7 (stale) Mirror 7: 0x7f84e1b7 (stale) Mirror 10: 0x7f84e1b7 (stale) Mirror 12: 0x7f84e1b7 (stale) Mirror 13: 0x7f84e1b7 (stale) Mirror 14: 0xc0c539e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22555: skipped 000386 1773668795.762846 mapread 0x00a99f thru 0x01854b (0xdbad bytes) 000387 1773668795.789410 trunc from 0x0041e2 to 0x012efe (0xed1d bytes) 000389 1773668795.799599 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x22555): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22555): Mirror 6: 0x7f84e1b7 (stale) Mirror 7: 0x7f84e1b7 (stale) Mirror 10: 0x7f84e1b7 (stale) Mirror 12: 0x7f84e1b7 (stale) Mirror 13: 0x7f84e1b7 (stale) Mirror 14: 0xc0c539e2 (stale) CRC-32 checksum value for chunk [0, 0x22555): Mirror 1: 0xb454a62c Mirror 15: 0xb454a62c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22555: skipped 000390 1773668796.797151 mapread 0x004673 thru 0x012be1 (0xe56f bytes) 000391 1773668796.831332 trunc from 0x022555 to 0x01eaf5 (0x3a5f bytes) 000392 1773668796.885179 mapread 0x01ba06 thru 0x01eaf5 (0x30f0 bytes) 000393 1773668796.897210 trunc from 0x002945 to 0x010104 (0xd7c0 bytes) 000394 1773668796.909547 fallocate 0x0081de thru 0x00da04 (0x5827 bytes) 000396 1773668796.922721 write 0x03f010 thru 0x03f4ea (0x4db bytes) 000397 1773668796.931180 punch from 0x02cde1 to 0x0368a7 (0x9ac7 bytes) 000398 1773668796.938327 mapwrite 0x00c94a thru 0x013dd5 (0x748c bytes) 000399 1773668797.003538 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f4eb): Mirror 7: 0x58f754ab (stale) Mirror 10: 0x58f754ab (stale) Mirror 12: 0x58f754ab (stale) Mirror 13: 0x58f754ab (stale) Mirror 14: 0x24504006 (stale) Mirror 15: 0x53498b23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4eb): Mirror 7: 0x58f754ab (stale) Mirror 10: 0x58f754ab (stale) Mirror 12: 0x58f754ab (stale) Mirror 13: 0x58f754ab (stale) Mirror 14: 0x24504006 (stale) Mirror 15: 0x53498b23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4eb: skipped 000400 1773668797.621607 trunc from 0x022437 to 0x0268a3 (0x446d bytes) 000401 1773668797.630429 read 0x0078a2 thru 0x00b6b9 (0x3e18 bytes) 000404 1773668797.640307 write 0x00d41d thru 0x00ec51 (0x1835 bytes) 000406 1773668797.651409 trunc from 0x03f4eb to 0x0098c1 (0x35c29 bytes) 000407 1773668797.666072 mapwrite 0x0210e0 thru 0x02ceb3 (0xbdd4 bytes) 000408 1773668797.717265 trunc from 0x02ceb4 to 0x03afe5 (0xe132 bytes) 000409 1773668797.740532 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3afe6): Mirror 7: 0x1cccdfb9 (stale) Mirror 12: 0x1cccdfb9 (stale) Mirror 13: 0x1cccdfb9 (stale) Mirror 14: 0x4dcc7c8e (stale) Mirror 15: 0xb4c4b71e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afe6): Mirror 7: 0x1cccdfb9 (stale) Mirror 12: 0x1cccdfb9 (stale) Mirror 13: 0x1cccdfb9 (stale) Mirror 14: 0x4dcc7c8e (stale) Mirror 15: 0xb4c4b71e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afe6: skipped 000410 1773668798.135670 trunc from 0x03afe6 to 0x0112fd (0x29ce8 bytes) 000411 1773668798.170249 mapread 0x00b175 thru 0x0112fd (0x6189 bytes) 000412 1773668798.196787 fallocate 0x034d13 thru 0x03e975 (0x9c63 bytes) 000413 1773668798.204907 trunc from 0x03e976 to 0x00bca4 (0x32cd1 bytes) 000414 1773668798.226591 read 0x001042 thru 0x0021ba (0x1179 bytes) 000416 1773668798.240575 punch from 0x0098b6 to 0x00bca4 (0x23ef bytes) 000417 1773668798.247504 trunc from 0x004c61 to 0x009b6f (0x4f0f bytes) 000418 1773668798.262946 read 0x0002bf thru 0x008069 (0x7dab bytes) 000419 1773668798.285526 mapread 0x0085ab thru 0x00bca4 (0x36fa bytes) 000420 1773668798.294597 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xbca5): Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbca5): Mirror 7: 0x1b582427 (stale) Mirror 12: 0x1b582427 (stale) Mirror 13: 0x1b582427 (stale) Mirror 14: 0x337b26bd (stale) Mirror 15: 0x41c3d15e (stale) CRC-32 checksum value for chunk [0, 0xbca5): Mirror 1: 0x66db8c6 Mirror 16: 0x66db8c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbca5: skipped 000421 1773668799.023603 read 0x004350 thru 0x00bca4 (0x7955 bytes) 000422 1773668799.037135 trunc from 0x00bca5 to 0x0360c8 (0x2a424 bytes) 000423 1773668799.129558 trunc from 0x0360c9 to 0x02061a (0x15aae bytes) 000426 1773668799.149272 read 0x00bc86 thru 0x017de3 (0xc15e bytes) 000427 1773668799.157219 trunc from 0x02061b to 0x03fb03 (0x1f4e9 bytes) 000428 1773668799.169462 trunc from 0x03fb04 to 0x01782c (0x282d7 bytes) 000429 1773668799.190325 mapwrite 0x013df5 thru 0x019dcc (0x5fd8 bytes) 000430 1773668799.239934 punch from 0x009372 to 0x012a48 (0x96d7 bytes) 000431 1773668799.248518 fallocate 0x015505 thru 0x0205e5 (0xb0e1 bytes) 000432 1773668799.254344 fallocate 0x03f95f thru 0x03ffff (0x6a1 bytes) 000433 1773668799.259823 trunc from 0x040000 to 0x027ef0 (0x1810f bytes) 000434 1773668799.283081 mapwrite 0x0033fd thru 0x0050a5 (0x1ca9 bytes) 000435 1773668799.301574 trunc from 0x027ef1 to 0x00624c (0x21ca4 bytes) 000436 1773668799.319468 trunc from 0x005269 to 0x00624c (0xfe4 bytes) 000437 1773668799.331580 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x624d): Mirror 7: 0xf48c9839 (stale) Mirror 12: 0xf48c9839 (stale) Mirror 13: 0xf48c9839 (stale) Mirror 15: 0x57e7843f (stale) Mirror 16: 0xe5086b19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x624d): Mirror 7: 0xf48c9839 (stale) Mirror 12: 0xf48c9839 (stale) Mirror 13: 0xf48c9839 (stale) Mirror 15: 0x57e7843f (stale) Mirror 16: 0xe5086b19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x624d: skipped 000438 1773668799.672262 trunc from 0x003270 to 0x00415f (0xef0 bytes) 000439 1773668799.676634 fallocate 0x03a10d thru 0x03ffff (0x5ef3 bytes) 000440 1773668799.687234 fallocate 0x002580 thru 0x007b3b (0x55bc bytes) 000441 1773668799.697230 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x624d): Mirror 7: 0xf48c9839 (stale) Mirror 13: 0xf48c9839 (stale) Mirror 15: 0x57e7843f (stale) Mirror 16: 0xe5086b19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x624d): Mirror 7: 0xf48c9839 (stale) Mirror 13: 0xf48c9839 (stale) Mirror 15: 0x57e7843f (stale) Mirror 16: 0xe5086b19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x624d: skipped 000442 1773668800.054129 mapread 0x001620 thru 0x00624c (0x4c2d bytes) 000443 1773668800.079597 write 0x01ded7 thru 0x01e1ab (0x2d5 bytes) 000444 1773668800.086561 trunc from 0x00724a to 0x00807b (0xe32 bytes) 000445 1773668800.099377 fallocate 0x01a614 thru 0x029b7b (0xf568 bytes) 000446 1773668800.104172 trunc from 0x01e1ac to 0x0355df (0x17434 bytes) 000447 1773668800.125672 read 0x013007 thru 0x022c4a (0xfc44 bytes) 000448 1773668800.141080 punch from 0x00b1b0 to 0x0184ed (0xd33e bytes) 000449 1773668800.146584 read 0x0164e4 thru 0x01c843 (0x6360 bytes) 000452 1773668800.175291 mapread 0x0227dc thru 0x02ccbc (0xa4e1 bytes) 000454 1773668800.185276 MIRROR_RESYNC 5 mirrors 000455 1773668801.186418 MIRROR_EXTEND to 6 mirrors 000456 1773668802.010296 trunc from 0x0355e0 to 0x02ebbd (0x6a22 bytes) 000457 1773668802.048478 fallocate 0x022759 thru 0x02f5c0 (0xce68 bytes) 000459 1773668802.052028 trunc from 0x00da10 to 0x0134f3 (0x5ae4 bytes) 000460 1773668802.054807 write 0x004a29 thru 0x00a208 (0x57e0 bytes) 000461 1773668802.060637 trunc from 0x02a4e0 to 0x02b788 (0x12a9 bytes) 000463 1773668802.064269 read 0x0204f4 thru 0x02966e (0x917b bytes) 000464 1773668802.073876 punch from 0x02a347 to 0x02c336 (0x1ff0 bytes) 000466 1773668802.083146 punch from 0x01bb0c to 0x01fdf6 (0x42eb bytes) 000467 1773668802.087895 fallocate 0x01becd thru 0x023cc1 (0x7df5 bytes) 000468 1773668802.092129 mapwrite 0x012308 thru 0x0189d0 (0x66c9 bytes) 000469 1773668802.123229 mapwrite 0x00d9c4 thru 0x01669f (0x8cdc bytes) 000470 1773668802.146804 MIRROR_RESYNC 6 mirrors 000471 1773668803.551649 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) 000472 1773668803.874376 mapread 0x01318b thru 0x01b9cb (0x8841 bytes) 000473 1773668803.895331 MIRROR_RESYNC 5 mirrors 000474 1773668804.282904 mapread 0x02796f thru 0x02f5c0 (0x7c52 bytes) 000475 1773668804.307836 punch from 0x02d84c to 0x02f5c0 (0x1d75 bytes) 000476 1773668804.340038 read 0x0242f7 thru 0x02f5c0 (0xb2ca bytes) 000477 1773668804.358754 trunc from 0x02504e to 0x02ecf8 (0x9cab bytes) 000478 1773668804.372720 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f5c1): Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5c1): Mirror 7: 0x21eb0180 (stale) Mirror 13: 0x21eb0180 (stale) Mirror 15: 0x21eb0180 (stale) Mirror 17: 0x21eb0180 (stale) CRC-32 checksum value for chunk [0, 0x2f5c1): Mirror 1: 0x21eb0180 Mirror 18: 0x21eb0180 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5c1: skipped 000479 1773668805.195443 read 0x0249c3 thru 0x02f32d (0xa96b bytes) 000480 1773668805.210328 mapread 0x00d5bb thru 0x014f84 (0x79ca bytes) 000483 1773668805.262727 MIRROR_RESYNC 6 mirrors 000484 1773668806.326806 mapwrite 0x03729b thru 0x03ecc2 (0x7a28 bytes) 000485 1773668806.402258 read 0x01aba7 thru 0x021f36 (0x7390 bytes) 000486 1773668806.419998 trunc from 0x03ecc3 to 0x030706 (0xe5bc bytes) 000487 1773668806.439348 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30707): Mirror 7: 0x41bfbd20 (stale) Mirror 13: 0x41bfbd20 (stale) Mirror 15: 0x41bfbd20 (stale) Mirror 17: 0x41bfbd20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30707): Mirror 7: 0x41bfbd20 (stale) Mirror 13: 0x41bfbd20 (stale) Mirror 15: 0x41bfbd20 (stale) Mirror 17: 0x41bfbd20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30707: skipped 000488 1773668806.729679 write 0x029bc3 thru 0x032ab0 (0x8eee bytes) 000489 1773668806.741319 punch from 0x004559 to 0x00cc59 (0x8701 bytes) 000490 1773668806.760982 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32ab1): Mirror 7: 0xa739392f (stale) Mirror 13: 0xa739392f (stale) Mirror 17: 0xa739392f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ab1): Mirror 7: 0xa739392f (stale) Mirror 13: 0xa739392f (stale) Mirror 17: 0xa739392f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ab1: skipped 000491 1773668807.100674 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32ab1): Mirror 7: 0xa739392f (stale) Mirror 13: 0xa739392f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ab1): Mirror 7: 0xa739392f (stale) Mirror 13: 0xa739392f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ab1: skipped 000492 1773668807.365353 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32ab1): Mirror 13: 0xa739392f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ab1): Mirror 13: 0xa739392f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ab1: skipped 000493 1773668807.582702 mapwrite 0x015f09 thru 0x0202a6 (0xa39e bytes) 000494 1773668807.628320 mapwrite 0x01396c thru 0x021275 (0xd90a bytes) 000495 1773668807.655555 punch from 0x017b6d to 0x01edda (0x726e bytes) 000496 1773668807.664052 mapwrite 0x02e4a7 thru 0x02ec57 (0x7b1 bytes) 000497 1773668807.695347 fallocate 0x0158a0 thru 0x0173d2 (0x1b33 bytes) 000498 1773668807.703572 mapwrite 0x0254bd thru 0x033e04 (0xe948 bytes) 000499 1773668807.772827 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) 000500 1773668807.905222 write 0x02d1ac thru 0x03570c (0x8561 bytes) 000501 1773668807.927591 trunc from 0x0239ca to 0x03250d (0xeb44 bytes) 000502 1773668807.946171 trunc from 0x00dd96 to 0x01b3d8 (0xd643 bytes) 000503 1773668807.953798 punch from 0x004cae to 0x01320d (0xe560 bytes) 000504 1773668807.964709 punch from 0x000604 to 0x001ee2 (0x18df bytes) 000505 1773668807.971296 trunc from 0x03570d to 0x011644 (0x240c8 bytes) 000506 1773668807.990950 mapwrite 0x011422 thru 0x0184af (0x708e bytes) 000507 1773668808.040794 fallocate 0x036756 thru 0x03ae13 (0x46be bytes) 000508 1773668808.053796 mapread 0x005969 thru 0x00c2e2 (0x697a bytes) 000510 1773668808.066042 trunc from 0x03ae14 to 0x03573f (0x56d4 bytes) 000512 1773668808.096245 write 0x03850f thru 0x03ffff (0x7af1 bytes) 000513 1773668808.098312 mapread 0x013d40 thru 0x01c7d4 (0x8a95 bytes) 000514 1773668808.107730 trunc from 0x040000 to 0x029d89 (0x16276 bytes) 000515 1773668808.125711 mapread 0x01d718 thru 0x023e42 (0x672b bytes) 000516 1773668808.140831 mapread 0x0153ab thru 0x01c0ab (0x6d01 bytes) 000517 1773668808.142390 fallocate 0x01f679 thru 0x0285ed (0x8f75 bytes) 000518 1773668808.153128 trunc from 0x029d8a to 0x0075f9 (0x22790 bytes) 000520 1773668808.171478 punch from 0x003587 to 0x0075f9 (0x4073 bytes) 000521 1773668808.179359 trunc from 0x0075fa to 0x038e33 (0x3183a bytes) 000522 1773668808.199514 trunc from 0x015cb2 to 0x024359 (0xe6a8 bytes) 000523 1773668808.208259 read 0x00c851 thru 0x010c65 (0x4415 bytes) 000524 1773668808.221762 read 0x0226cb thru 0x02b493 (0x8dc9 bytes) 000525 1773668808.241327 write 0x028226 thru 0x036599 (0xe374 bytes) 000526 1773668808.242823 fallocate 0x02e929 thru 0x02faea (0x11c2 bytes) 000529 1773668808.247484 trunc from 0x038e34 to 0x0094d2 (0x2f961 bytes) 000530 1773668808.259582 write 0x02102d thru 0x028b03 (0x7ad7 bytes) 000531 1773668808.261349 mapread 0x00f8c7 thru 0x014fc8 (0x5702 bytes) 000532 1773668808.286491 mapread 0x01de4e thru 0x028b03 (0xacb6 bytes) 000533 1773668808.305337 mapwrite 0x02c851 thru 0x033646 (0x6df6 bytes) 000534 1773668808.390527 mapread 0x002261 thru 0x003f9f (0x1d3f bytes) 000535 1773668808.391818 mapwrite 0x02b4cf thru 0x03a0b8 (0xebea bytes) 000536 1773668808.472102 MIRROR_EXTEND to 2 mirrors 000537 1773668809.010085 MIRROR_RESYNC 2 mirrors 000538 1773668809.321789 trunc from 0x03a0b9 to 0x03cc09 (0x2b51 bytes) 000540 1773668809.391249 read 0x0369ae thru 0x03919b (0x27ee bytes) 000541 1773668809.399909 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) 000542 1773668809.526922 MIRROR_EXTEND to 2 mirrors 000543 1773668810.116532 MIRROR_RESYNC 2 mirrors 000544 1773668810.396871 trunc from 0x030b49 to 0x03c091 (0xb549 bytes) 000545 1773668810.426469 fallocate 0x004621 thru 0x007428 (0x2e08 bytes) 000546 1773668810.429731 write 0x01316b thru 0x019844 (0x66da bytes) 000547 1773668810.439327 read 0x020b2f thru 0x0239c1 (0x2e93 bytes) 000548 1773668810.443444 trunc from 0x00e51a to 0x015ea0 (0x7987 bytes) 000549 1773668810.455222 read 0x01982d thru 0x022854 (0x9028 bytes) 000550 1773668810.463758 read 0x0373ac thru 0x03cc09 (0x585e bytes) 000552 1773668810.464765 trunc from 0x03cc0a to 0x0137a9 (0x29460 bytes) 000553 1773668810.490809 mapread 0x00539e thru 0x007e39 (0x2a9c bytes) 000554 1773668810.491776 fallocate 0x01ac8a thru 0x01e78b (0x3b02 bytes) 000555 1773668810.499759 trunc from 0x01e78c to 0x02e097 (0xf90c bytes) 000556 1773668810.518191 read 0x007d5a thru 0x011243 (0x94ea bytes) 000557 1773668810.519179 write 0x03e825 thru 0x03ffff (0x17db bytes) 000558 1773668810.520735 fallocate 0x03042e thru 0x038a99 (0x866c bytes) 000559 1773668810.526289 mapwrite 0x037248 thru 0x03ffff (0x8db8 bytes) 000560 1773668810.557089 trunc from 0x03592d to 0x03c85f (0x6f33 bytes) 000561 1773668810.560927 fallocate 0x02932b thru 0x02ade4 (0x1aba bytes) 000562 1773668810.567498 trunc from 0x040000 to 0x03f486 (0xb79 bytes) 000563 1773668810.586384 fallocate 0x012d90 thru 0x017e4a (0x50bb bytes) 000564 1773668810.601631 MIRROR_RESYNC 2 mirrors 000565 1773668810.984961 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) 000567 1773668811.063493 trunc from 0x00e675 to 0x012e11 (0x479d bytes) 000570 1773668811.079500 mapwrite 0x00d9aa thru 0x011220 (0x3877 bytes) 000572 1773668811.109632 mapread 0x008c5a thru 0x017bd2 (0xef79 bytes) 000573 1773668811.111092 write 0x0019c1 thru 0x003773 (0x1db3 bytes) 000574 1773668811.124820 trunc from 0x02855e to 0x03483a (0xc2dd bytes) 000576 1773668811.129816 write 0x02c719 thru 0x031093 (0x497b bytes) 000577 1773668811.155082 MIRROR_EXTEND to 2 mirrors 000578 1773668811.705480 fallocate 0x033908 thru 0x0366a3 (0x2d9c bytes) 000579 1773668811.745338 mapwrite 0x02e610 thru 0x03cdd1 (0xe7c2 bytes) 000580 1773668811.799981 write 0x010a20 thru 0x01c57a (0xbb5b bytes) 000583 1773668811.841892 mapwrite 0x009e6e thru 0x00f767 (0x58fa bytes) 000584 1773668811.868515 write 0x02dd2b thru 0x033938 (0x5c0e bytes) 000585 1773668811.877372 MIRROR_RESYNC 2 mirrors 000586 1773668812.412588 mapwrite 0x03cf57 thru 0x03ffff (0x30a9 bytes) 000587 1773668812.493794 fallocate 0x03e119 thru 0x03ffff (0x1ee7 bytes) 000588 1773668812.500378 trunc from 0x02d047 to 0x03c0c7 (0xf081 bytes) 000589 1773668812.507632 mapread 0x02bb48 thru 0x03b2d0 (0xf789 bytes) 000590 1773668812.536539 trunc from 0x040000 to 0x034961 (0xb69e bytes) 000591 1773668812.555344 mapwrite 0x005217 thru 0x009489 (0x4273 bytes) 000592 1773668812.586243 mapread 0x03105a thru 0x034961 (0x3908 bytes) 000593 1773668812.591554 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34962): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34962): Mirror 2: 0x95ff9da7 (stale) CRC-32 checksum value for chunk [0, 0x34962): Mirror 1: 0x8bc19e1b Mirror 3: 0x8bc19e1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34962: skipped 000594 1773668813.206666 MIRROR_RESYNC 3 mirrors 000595 1773668813.722214 trunc from 0x00bec0 to 0x0182e3 (0xc424 bytes) 000596 1773668813.754214 mapread 0x025b86 thru 0x0270ed (0x1568 bytes) 000597 1773668813.771764 trunc from 0x034962 to 0x02e36b (0x65f6 bytes) 000598 1773668813.805769 read 0x01d11f thru 0x02abfa (0xdadc bytes) 000599 1773668813.812299 trunc from 0x02366f to 0x02e36b (0xacfd bytes) 000600 1773668813.817298 trunc from 0x02e36c to 0x038380 (0xa015 bytes) 000601 1773668813.845829 MIRROR_RESYNC 3 mirrors 000602 1773668814.480406 trunc from 0x038381 to 0x034dcc (0x35b4 bytes) 000603 1773668814.518415 mapwrite 0x01ae17 thru 0x01b247 (0x431 bytes) 000604 1773668814.544162 read 0x02cfec thru 0x034dcc (0x7de1 bytes) 000605 1773668814.551288 mapread 0x01be97 thru 0x0288e1 (0xca4b bytes) 000606 1773668814.562307 punch from 0x034a63 to 0x034dcc (0x36a bytes) 000607 1773668814.567047 mapread 0x01af88 thru 0x020a0a (0x5a83 bytes) 000608 1773668814.568268 punch from 0x01f4e3 to 0x021527 (0x2045 bytes) 000609 1773668814.573734 read 0x01eb9a thru 0x020cef (0x2156 bytes) 000610 1773668814.581081 mapwrite 0x016617 thru 0x017e0b (0x17f5 bytes) 000612 1773668814.600624 write 0x00e72b thru 0x01aeb9 (0xc78f bytes) 000613 1773668814.605806 write 0x003ff6 thru 0x00d7cb (0x97d6 bytes) 000614 1773668814.610572 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34dcd): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34dcd): Mirror 2: 0xa756d3bd (stale) Mirror 3: 0xa756d3bd (stale) CRC-32 checksum value for chunk [0, 0x34dcd): Mirror 1: 0xe0d7c6a2 Mirror 4: 0xe0d7c6a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34dcd: skipped 000615 1773668815.092757 punch from 0x0286b6 to 0x02c4ec (0x3e37 bytes) 000616 1773668815.128239 trunc from 0x01d800 to 0x01f59a (0x1d9b bytes) 000617 1773668815.132649 trunc from 0x027ea7 to 0x032d9c (0xaef6 bytes) 000619 1773668815.137241 mapwrite 0x00066b thru 0x00b317 (0xacad bytes) 000620 1773668815.168572 write 0x0162af thru 0x02126e (0xafc0 bytes) 000621 1773668815.188136 trunc from 0x034dcd to 0x024fbb (0xfe11 bytes) 000622 1773668815.221221 read 0x001659 thru 0x00b29c (0x9c44 bytes) 000623 1773668815.228349 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x24fbc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fbc): Mirror 2: 0xa03b7adf (stale) Mirror 3: 0xa03b7adf (stale) Mirror 4: 0xb1c1d784 (stale) CRC-32 checksum value for chunk [0, 0x24fbc): Mirror 1: 0x65eda82 Mirror 5: 0x65eda82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fbc: skipped 000625 1773668815.708063 punch from 0x01aca6 to 0x024fbb (0xa316 bytes) 000626 1773668815.723293 punch from 0x01d979 to 0x024fbb (0x7643 bytes) 000627 1773668815.725867 mapread 0x01d16d thru 0x024fbb (0x7e4f bytes) 000628 1773668815.734918 punch from 0x010824 to 0x014ee0 (0x46bd bytes) 000629 1773668815.737758 mapwrite 0x009be0 thru 0x0196c1 (0xfae2 bytes) 000630 1773668815.767848 mapread 0x0041a8 thru 0x0073aa (0x3203 bytes) 000631 1773668815.769065 read 0x005f9f thru 0x012225 (0xc287 bytes) 000632 1773668815.775628 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24fbc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fbc): Mirror 2: 0xa03b7adf (stale) Mirror 3: 0xa03b7adf (stale) Mirror 4: 0xb1c1d784 (stale) Mirror 5: 0x65eda82 (stale) CRC-32 checksum value for chunk [0, 0x24fbc): Mirror 1: 0xa2c6c4e9 Mirror 6: 0xa2c6c4e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fbc: skipped 000633 1773668816.306305 mapread 0x002256 thru 0x00fd42 (0xdaed bytes) 000634 1773668816.343522 fallocate 0x03d493 thru 0x03ffff (0x2b6d bytes) 000635 1773668816.366284 trunc from 0x024fbc to 0x02dfd6 (0x901b bytes) 000636 1773668816.401154 fallocate 0x030d4d thru 0x03b255 (0xa509 bytes) 000637 1773668816.405741 read 0x0078e1 thru 0x009e4f (0x256f bytes) 000638 1773668816.410507 read 0x01b69e thru 0x01eeeb (0x384e bytes) 000639 1773668816.426687 write 0x012c22 thru 0x021266 (0xe645 bytes) 000640 1773668816.428676 punch from 0x00897f to 0x00ba2d (0x30af bytes) 000641 1773668816.438081 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b256): Mirror 3: 0x55c03232 (stale) Mirror 4: 0x1af38d2 (stale) Mirror 5: 0x8b39821d (stale) Mirror 6: 0xbc35a7c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b256): Mirror 3: 0x55c03232 (stale) Mirror 4: 0x1af38d2 (stale) Mirror 5: 0x8b39821d (stale) Mirror 6: 0xbc35a7c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b256: skipped 000643 1773668816.727895 trunc from 0x0231c0 to 0x02c2f3 (0x9134 bytes) 000644 1773668816.741310 trunc from 0x029b31 to 0x036cd8 (0xd1a8 bytes) 000645 1773668816.746321 read 0x00a99d thru 0x01675b (0xbdbf bytes) 000646 1773668816.760755 trunc from 0x03b256 to 0x01937a (0x21edb bytes) 000647 1773668816.779798 mapwrite 0x00f22d thru 0x018de3 (0x9bb7 bytes) 000648 1773668816.805500 write 0x0313c2 thru 0x038737 (0x7376 bytes) 000649 1773668816.806635 punch from 0x00eefd to 0x010817 (0x191b bytes) 000650 1773668816.812738 punch from 0x02ef19 to 0x038737 (0x981f bytes) 000651 1773668816.828628 MIRROR_RESYNC 5 mirrors 000652 1773668817.836735 trunc from 0x038738 to 0x034b3d (0x3bfa bytes) 000653 1773668817.856428 trunc from 0x034b3e to 0x02f034 (0x5b09 bytes) 000654 1773668817.865482 mapread 0x01f4f6 thru 0x02e106 (0xec11 bytes) 000655 1773668817.893895 mapread 0x01148b thru 0x0164ac (0x5022 bytes) 000656 1773668817.913301 fallocate 0x01a120 thru 0x01e087 (0x3f68 bytes) 000657 1773668817.918824 fallocate 0x01eae0 thru 0x02c43a (0xd95b bytes) 000658 1773668817.931348 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f035): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f035): Mirror 3: 0x2d893df5 (stale) Mirror 4: 0x2d893df5 (stale) Mirror 5: 0x2d893df5 (stale) Mirror 6: 0x2d893df5 (stale) CRC-32 checksum value for chunk [0, 0x2f035): Mirror 1: 0x2d893df5 Mirror 7: 0x2d893df5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f035: skipped 000659 1773668818.550801 mapwrite 0x024228 thru 0x02dd50 (0x9b29 bytes) 000660 1773668818.637666 mapwrite 0x019ab0 thru 0x01dda0 (0x42f1 bytes) 000661 1773668818.673339 MIRROR_RESYNC 6 mirrors 000662 1773668819.787317 MIRROR_EXTEND to 7 mirrors 000663 1773668820.333588 trunc from 0x02f035 to 0x0158bf (0x19775 bytes) 000664 1773668820.373046 mapread 0x0030cd thru 0x00c88e (0x97c2 bytes) 000665 1773668820.387417 fallocate 0x005843 thru 0x008187 (0x2945 bytes) 000666 1773668820.391504 trunc from 0x00bbe3 to 0x0158bf (0x9cdd bytes) 000667 1773668820.398156 MIRROR_RESYNC 7 mirrors 000668 1773668821.418888 fallocate 0x008d3a thru 0x00f288 (0x654f bytes) 000669 1773668821.452829 read 0x00915a thru 0x0158bf (0xc766 bytes) 000670 1773668821.462931 punch from 0x008117 to 0x00f96c (0x7856 bytes) 000672 1773668821.468119 read 0x00c90b thru 0x00e99c (0x2092 bytes) 000673 1773668821.474356 mapread 0x00441a thru 0x005248 (0xe2f bytes) 000675 1773668821.475524 mapwrite 0x01238d thru 0x01259f (0x213 bytes) 000676 1773668821.497831 trunc from 0x0158c0 to 0x02b322 (0x15a63 bytes) 000677 1773668821.523583 punch from 0x00c53c to 0x018d1b (0xc7e0 bytes) 000678 1773668821.527422 mapwrite 0x019008 thru 0x028786 (0xf77f bytes) 000679 1773668821.550171 punch from 0x00eb01 to 0x01d66c (0xeb6c bytes) 000680 1773668821.557825 punch from 0x00bb31 to 0x00e6af (0x2b7f bytes) 000681 1773668821.566259 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b323): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b323): Mirror 3: 0xb2f1683f (stale) Mirror 4: 0xb2f1683f (stale) Mirror 5: 0xb2f1683f (stale) Mirror 6: 0xb2f1683f (stale) Mirror 7: 0xb2f1683f (stale) Mirror 8: 0xb2f1683f (stale) CRC-32 checksum value for chunk [0, 0x2b323): Mirror 1: 0xe028c86d Mirror 9: 0xe028c86d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b323: skipped 000682 1773668822.203582 read 0x01f6b0 thru 0x02289d (0x31ee bytes) 000683 1773668822.210296 read 0x01a405 thru 0x029d42 (0xf93e bytes) 000684 1773668822.231563 MIRROR_RESYNC 8 mirrors 000685 1773668823.219979 write 0x030470 thru 0x03d241 (0xcdd2 bytes) 000686 1773668823.243940 mapread 0x00a0a1 thru 0x00e140 (0x40a0 bytes) 000687 1773668823.259354 punch from 0x019c2e to 0x026486 (0xc859 bytes) 000688 1773668823.274329 MIRROR_RESYNC 8 mirrors 000689 1773668824.563773 trunc from 0x03d242 to 0x01de51 (0x1f3f0 bytes) 000690 1773668824.603748 write 0x031f6d thru 0x0389c0 (0x6a54 bytes) 000691 1773668824.605251 trunc from 0x005f73 to 0x00898b (0x2a19 bytes) 000692 1773668824.608576 trunc from 0x0389c1 to 0x01e90e (0x1a0b2 bytes) 000693 1773668824.622816 mapwrite 0x01e7a5 thru 0x02939e (0xabfa bytes) 000695 1773668824.670785 trunc from 0x00edd7 to 0x00ee58 (0x82 bytes) 000696 1773668824.675841 mapwrite 0x01fbef thru 0x028a87 (0x8e99 bytes) 000697 1773668824.691320 trunc from 0x02939f to 0x0346ec (0xb34e bytes) 000698 1773668824.712627 punch from 0x007db6 to 0x008ec5 (0x1110 bytes) 000699 1773668824.728568 MIRROR_RESYNC 8 mirrors 000700 1773668826.222335 trunc from 0x0346ed to 0x017ff3 (0x1c6f9 bytes) 000701 1773668826.259939 fallocate 0x001b76 thru 0x0103a9 (0xe834 bytes) 000702 1773668826.271670 trunc from 0x017ff4 to 0x03432f (0x1c33c bytes) 000703 1773668826.289257 write 0x024a86 thru 0x026b84 (0x20ff bytes) 000704 1773668826.296639 trunc from 0x02bfa5 to 0x02e07f (0x20db bytes) 000706 1773668826.307957 punch from 0x02d0a6 to 0x03432f (0x728a bytes) 000708 1773668826.316608 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34330): Mirror 4: 0x2b1ecad (stale) Mirror 5: 0x2b1ecad (stale) Mirror 6: 0x2b1ecad (stale) Mirror 7: 0x2b1ecad (stale) Mirror 8: 0x2b1ecad (stale) Mirror 9: 0x2b1ecad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34330): Mirror 4: 0x2b1ecad (stale) Mirror 5: 0x2b1ecad (stale) Mirror 6: 0x2b1ecad (stale) Mirror 7: 0x2b1ecad (stale) Mirror 8: 0x2b1ecad (stale) Mirror 9: 0x2b1ecad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34330: skipped 000709 1773668826.770156 trunc from 0x02c099 to 0x03432f (0x8297 bytes) 000710 1773668826.776064 fallocate 0x0012ba thru 0x0084fd (0x7244 bytes) 000711 1773668826.788467 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34330): Mirror 4: 0x2b1ecad (stale) Mirror 6: 0x2b1ecad (stale) Mirror 7: 0x2b1ecad (stale) Mirror 8: 0x2b1ecad (stale) Mirror 9: 0x2b1ecad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34330): Mirror 4: 0x2b1ecad (stale) Mirror 6: 0x2b1ecad (stale) Mirror 7: 0x2b1ecad (stale) Mirror 8: 0x2b1ecad (stale) Mirror 9: 0x2b1ecad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34330: skipped 000712 1773668827.055214 trunc from 0x016344 to 0x021762 (0xb41f bytes) 000713 1773668827.057940 trunc from 0x034330 to 0x02f413 (0x4f1c bytes) 000714 1773668827.067630 mapread 0x029419 thru 0x02f413 (0x5ffb bytes) 000715 1773668827.079181 write 0x02e56d thru 0x032e9d (0x4931 bytes) 000716 1773668827.080164 punch from 0x015221 to 0x02425e (0xf03e bytes) 000717 1773668827.083780 read 0x019888 thru 0x026c50 (0xd3c9 bytes) 000718 1773668827.091607 mapwrite 0x02a048 thru 0x03475c (0xa715 bytes) 000719 1773668827.127978 trunc from 0x00ebe6 to 0x010b07 (0x1f22 bytes) 000720 1773668827.132516 punch from 0x007dd2 to 0x012b07 (0xad36 bytes) 000721 1773668827.136968 punch from 0x00daae to 0x010dc9 (0x331c bytes) 000722 1773668827.144157 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3475d): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3475d): Mirror 4: 0x96b72ad2 (stale) Mirror 6: 0x96b72ad2 (stale) Mirror 7: 0x96b72ad2 (stale) Mirror 8: 0x96b72ad2 (stale) Mirror 9: 0x96b72ad2 (stale) CRC-32 checksum value for chunk [0, 0x3475d): Mirror 1: 0x89db290d Mirror 10: 0x89db290d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3475d: skipped 000723 1773668827.947809 mapwrite 0x03a6af thru 0x03bde2 (0x1734 bytes) 000724 1773668828.057272 mapwrite 0x027b3c thru 0x0318b9 (0x9d7e bytes) 000725 1773668828.091711 read 0x039ae5 thru 0x03bde2 (0x22fe bytes) 000726 1773668828.095735 trunc from 0x00df85 to 0x01b799 (0xd815 bytes) 000727 1773668828.100569 write 0x00ab4c thru 0x00d7e7 (0x2c9c bytes) 000728 1773668828.112792 mapread 0x0243dd thru 0x031dad (0xd9d1 bytes) 000729 1773668828.114401 punch from 0x01bf53 to 0x020896 (0x4944 bytes) 000730 1773668828.120278 mapread 0x025fb4 thru 0x02c408 (0x6455 bytes) 000731 1773668828.123028 read 0x00b456 thru 0x01ab5d (0xf708 bytes) 000732 1773668828.131641 read 0x03b540 thru 0x03bb96 (0x657 bytes) 000733 1773668828.136827 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3bde3): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bde3): Mirror 4: 0x621eef11 (stale) Mirror 6: 0x621eef11 (stale) Mirror 7: 0x621eef11 (stale) Mirror 8: 0x621eef11 (stale) Mirror 9: 0x621eef11 (stale) Mirror 10: 0xab190279 (stale) CRC-32 checksum value for chunk [0, 0x3bde3): Mirror 1: 0x95fdb31a Mirror 11: 0x95fdb31a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bde3: skipped 000734 1773668828.888441 mapwrite 0x02e3d8 thru 0x0348d4 (0x64fd bytes) 000735 1773668828.983199 trunc from 0x00c610 to 0x017583 (0xaf74 bytes) 000736 1773668828.991962 MIRROR_RESYNC 8 mirrors 000737 1773668830.492155 trunc from 0x03bde3 to 0x008374 (0x33a6e bytes) 000738 1773668830.535894 write 0x006e0d thru 0x00d2c1 (0x64b5 bytes) 000739 1773668830.553576 punch from 0x0019d8 to 0x00ae1b (0x9444 bytes) 000741 1773668830.567616 mapread 0x008dcf thru 0x00d2c1 (0x44f3 bytes) 000742 1773668830.586911 mapread 0x004569 thru 0x00be61 (0x78f9 bytes) 000743 1773668830.588226 write 0x02b507 thru 0x036aed (0xb5e7 bytes) 000744 1773668830.591256 punch from 0x033078 to 0x03429c (0x1225 bytes) 000745 1773668830.613162 mapwrite 0x03a32a thru 0x03a69a (0x371 bytes) 000746 1773668830.646262 fallocate 0x012e7d thru 0x018691 (0x5815 bytes) 000747 1773668830.649475 mapwrite 0x01f22e thru 0x02e0ac (0xee7f bytes) 000748 1773668830.670161 mapread 0x012fcd thru 0x01f8df (0xc913 bytes) 000749 1773668830.686294 trunc from 0x03a69b to 0x00d720 (0x2cf7a bytes) 000751 1773668830.714290 write 0x008aec thru 0x015d42 (0xd257 bytes) 000753 1773668830.719599 write 0x0118dc thru 0x01cf2c (0xb651 bytes) 000754 1773668830.723465 read 0x010a5d thru 0x01cf2c (0xc4d0 bytes) 000756 1773668830.725190 trunc from 0x01330c to 0x01cf2c (0x9c21 bytes) 000757 1773668830.741063 fallocate 0x035f99 thru 0x03ffff (0xa067 bytes) 000758 1773668830.750759 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x251dcf9c (stale) Mirror 7: 0x251dcf9c (stale) Mirror 8: 0x251dcf9c (stale) Mirror 9: 0x251dcf9c (stale) Mirror 10: 0x251dcf9c (stale) Mirror 11: 0x251dcf9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x251dcf9c (stale) Mirror 7: 0x251dcf9c (stale) Mirror 8: 0x251dcf9c (stale) Mirror 9: 0x251dcf9c (stale) Mirror 10: 0x251dcf9c (stale) Mirror 11: 0x251dcf9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000759 1773668831.076523 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x251dcf9c (stale) Mirror 7: 0x251dcf9c (stale) Mirror 8: 0x251dcf9c (stale) Mirror 9: 0x251dcf9c (stale) Mirror 11: 0x251dcf9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x251dcf9c (stale) Mirror 7: 0x251dcf9c (stale) Mirror 8: 0x251dcf9c (stale) Mirror 9: 0x251dcf9c (stale) Mirror 11: 0x251dcf9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1773668831.376518 fallocate 0x01713b thru 0x01ff9e (0x8e64 bytes) 000761 1773668831.382767 mapwrite 0x019402 thru 0x022568 (0x9167 bytes) 000762 1773668831.428327 fallocate 0x0278a5 thru 0x02dae4 (0x6240 bytes) 000763 1773668831.431692 trunc from 0x040000 to 0x0284cd (0x17b32 bytes) 000764 1773668831.453203 mapwrite 0x0064dc thru 0x007823 (0x1348 bytes) 000765 1773668831.478165 mapread 0x00ef2c thru 0x0142eb (0x53c0 bytes) 000766 1773668831.482889 write 0x0023b0 thru 0x00e739 (0xc38a bytes) 000767 1773668831.486717 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x284ce): Mirror 7: 0xe1a7cc33 (stale) Mirror 8: 0xe1a7cc33 (stale) Mirror 9: 0xe1a7cc33 (stale) Mirror 11: 0xe1a7cc33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x284ce): Mirror 7: 0xe1a7cc33 (stale) Mirror 8: 0xe1a7cc33 (stale) Mirror 9: 0xe1a7cc33 (stale) Mirror 11: 0xe1a7cc33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x284ce: skipped 000768 1773668831.771295 fallocate 0x03dec3 thru 0x03ffff (0x213d bytes) 000769 1773668831.775465 write 0x02b285 thru 0x037470 (0xc1ec bytes) 000771 1773668831.780324 mapwrite 0x01c663 thru 0x0274fc (0xae9a bytes) 000772 1773668831.811931 write 0x0054cd thru 0x0153a8 (0xfedc bytes) 000773 1773668831.821496 mapread 0x01d7c9 thru 0x023acf (0x6307 bytes) 000774 1773668831.823653 trunc from 0x016ffc to 0x020f50 (0x9f55 bytes) 000775 1773668831.827823 write 0x024036 thru 0x02f301 (0xb2cc bytes) 000777 1773668831.830833 fallocate 0x0161af thru 0x01a7ba (0x460c bytes) 000778 1773668831.834597 trunc from 0x022ddf to 0x030d71 (0xdf93 bytes) 000779 1773668831.845072 read 0x0317b9 thru 0x037470 (0x5cb8 bytes) 000780 1773668831.845804 mapwrite 0x01d53c thru 0x0203e1 (0x2ea6 bytes) 000781 1773668831.872974 mapwrite 0x030408 thru 0x0359ce (0x55c7 bytes) 000782 1773668831.892644 trunc from 0x037471 to 0x03cf96 (0x5b26 bytes) 000783 1773668831.902980 punch from 0x02f87b to 0x03a212 (0xa998 bytes) 000784 1773668831.906204 mapread 0x03365c thru 0x03a746 (0x70eb bytes) 000785 1773668831.919251 mapwrite 0x021212 thru 0x028022 (0x6e11 bytes) 000787 1773668831.945677 fallocate 0x03425d thru 0x03f802 (0xb5a6 bytes) 000788 1773668831.951387 trunc from 0x03cf97 to 0x013adf (0x294b7 bytes) 000789 1773668831.964786 punch from 0x00a5e4 to 0x013adf (0x94fc bytes) 000791 1773668831.971187 write 0x034fb9 thru 0x036975 (0x19bd bytes) 000792 1773668831.972209 read 0x00f975 thru 0x018910 (0x8f9c bytes) 000793 1773668831.984101 mapwrite 0x00b8fc thru 0x00dae0 (0x21e5 bytes) 000794 1773668832.002925 trunc from 0x036976 to 0x0257c3 (0x111b2 bytes) 000796 1773668832.030693 fallocate 0x020694 thru 0x021659 (0xfc6 bytes) 000797 1773668832.035527 mapwrite 0x021425 thru 0x0240f5 (0x2cd1 bytes) 000798 1773668832.049011 read 0x005407 thru 0x0070f6 (0x1cf0 bytes) 000799 1773668832.049591 trunc from 0x0257c4 to 0x01a39a (0xb429 bytes) 000800 1773668832.060899 trunc from 0x007434 to 0x00aeee (0x3abb bytes) 000801 1773668832.070424 read 0x00e3a1 thru 0x015d99 (0x79f9 bytes) 000804 1773668832.072483 trunc from 0x00cfd9 to 0x019dfb (0xce23 bytes) 000806 1773668832.082411 read 0x00a34a thru 0x0115d9 (0x7290 bytes) 000807 1773668832.089540 mapread 0x0104ef thru 0x01a39a (0x9eac bytes) 000808 1773668832.090963 trunc from 0x01a39b to 0x0043c9 (0x15fd1 bytes) 000810 1773668832.117278 fallocate 0x003624 thru 0x00dcbc (0xa699 bytes) 000811 1773668832.129435 trunc from 0x00c63f to 0x00dcbc (0x167e bytes) 000813 1773668832.146916 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdcbd): Mirror 7: 0x7a2cdef5 (stale) Mirror 8: 0x7a2cdef5 (stale) Mirror 9: 0x7a2cdef5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdcbd): Mirror 7: 0x7a2cdef5 (stale) Mirror 8: 0x7a2cdef5 (stale) Mirror 9: 0x7a2cdef5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdcbd: skipped 000814 1773668832.421544 write 0x012beb thru 0x019606 (0x6a1c bytes) 000815 1773668832.429475 mapread 0x00f2af thru 0x019606 (0xa358 bytes) 000818 1773668832.448806 mapread 0x0038b9 thru 0x012f93 (0xf6db bytes) 000819 1773668832.461932 mapread 0x014e33 thru 0x019606 (0x47d4 bytes) 000820 1773668832.463700 mapwrite 0x0068bf thru 0x00870d (0x1e4f bytes) 000821 1773668832.484897 trunc from 0x0150b9 to 0x019606 (0x454e bytes) 000822 1773668832.492338 punch from 0x00648c to 0x013860 (0xd3d5 bytes) 000823 1773668832.497956 fallocate 0x02905a thru 0x030e71 (0x7e18 bytes) 000824 1773668832.505537 mapread 0x00e999 thru 0x00efa3 (0x60b bytes) 000825 1773668832.520442 punch from 0x01d56e to 0x024bf4 (0x7687 bytes) 000826 1773668832.527040 fallocate 0x018fe9 thru 0x01daa3 (0x4abb bytes) 000828 1773668832.535047 mapwrite 0x007160 thru 0x009d71 (0x2c12 bytes) 000829 1773668832.545974 write 0x013ef7 thru 0x023ad8 (0xfbe2 bytes) 000830 1773668832.552365 mapread 0x01f445 thru 0x02850e (0x90ca bytes) 000831 1773668832.562000 mapwrite 0x009fc3 thru 0x0139ae (0x99ec bytes) 000833 1773668832.580671 MIRROR_RESYNC 4 mirrors 000834 1773668833.264600 fallocate 0x024154 thru 0x0287ed (0x469a bytes) 000835 1773668833.305020 mapwrite 0x02c81b thru 0x0311ca (0x49b0 bytes) 000836 1773668833.378527 mapwrite 0x02fdc5 thru 0x03f29a (0xf4d6 bytes) 000837 1773668833.450143 read 0x012db1 thru 0x01b9ae (0x8bfe bytes) 000838 1773668833.462514 trunc from 0x03f29b to 0x034067 (0xb233 bytes) 000839 1773668833.483479 trunc from 0x02e18d to 0x030032 (0x1ea6 bytes) 000840 1773668833.490288 mapwrite 0x039b49 thru 0x03b216 (0x16ce bytes) 000841 1773668833.527127 punch from 0x00e7a8 to 0x012cd6 (0x452f bytes) 000842 1773668833.532593 fallocate 0x03a93b thru 0x03ffff (0x56c5 bytes) 000843 1773668833.538762 trunc from 0x040000 to 0x01bbb7 (0x24448 bytes) 000844 1773668833.562403 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1bbb8): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bbb8): Mirror 7: 0xc8223c85 (stale) Mirror 8: 0xc8223c85 (stale) Mirror 9: 0xc8223c85 (stale) CRC-32 checksum value for chunk [0, 0x1bbb8): Mirror 1: 0xc1a83457 Mirror 10: 0xc1a83457 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bbb8: skipped 000845 1773668834.204464 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bbb8): Mirror 7: 0xc8223c85 (stale) Mirror 8: 0xc8223c85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bbb8): Mirror 7: 0xc8223c85 (stale) Mirror 8: 0xc8223c85 (stale) CRC-32 checksum value for chunk [0, 0x1bbb8): Mirror 1: 0xc1a83457 Mirror 10: 0xc1a83457 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bbb8: skipped 000846 1773668834.584681 punch from 0x012a94 to 0x01a96b (0x7ed8 bytes) 000847 1773668834.625950 MIRROR_RESYNC 4 mirrors 000849 1773668835.378296 mapread 0x0113f4 thru 0x017da6 (0x69b3 bytes) 000850 1773668835.403988 trunc from 0x010e1f to 0x01825d (0x743f bytes) 000851 1773668835.440107 trunc from 0x002ffe to 0x0089c2 (0x59c5 bytes) 000852 1773668835.447481 mapwrite 0x0093ca thru 0x00c438 (0x306f bytes) 000853 1773668835.466263 trunc from 0x0138bd to 0x01bbb7 (0x82fb bytes) 000854 1773668835.470500 trunc from 0x01bbb8 to 0x011ac5 (0xa0f2 bytes) 000855 1773668835.482750 mapread 0x00dac8 thru 0x00ead1 (0x100a bytes) 000856 1773668835.483829 punch from 0x00bf8c to 0x00d989 (0x19fe bytes) 000857 1773668835.490123 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x11ac6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ac6): Mirror 7: 0xb9e6de4d (stale) Mirror 8: 0xb9e6de4d (stale) Mirror 10: 0xb9e6de4d (stale) CRC-32 checksum value for chunk [0, 0x11ac6): Mirror 1: 0x69b6f63d Mirror 11: 0x69b6f63d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ac6: skipped 000858 1773668836.165560 punch from 0x0004f9 to 0x004cee (0x47f6 bytes) 000859 1773668836.198208 write 0x0259fa thru 0x025b4c (0x153 bytes) 000860 1773668836.203834 mapread 0x0023e8 thru 0x0057e7 (0x3400 bytes) 000861 1773668836.222085 fallocate 0x02cad1 thru 0x030cb5 (0x41e5 bytes) 000862 1773668836.225912 read 0x02ba4a thru 0x030cb5 (0x526c bytes) 000863 1773668836.234382 trunc from 0x030cb6 to 0x00eac5 (0x221f0 bytes) 000864 1773668836.268510 mapwrite 0x00222d thru 0x008592 (0x6366 bytes) 000865 1773668836.280604 mapwrite 0x024fc1 thru 0x03010f (0xb14f bytes) 000866 1773668836.323791 mapread 0x000c6d thru 0x001e41 (0x11d5 bytes) 000867 1773668836.332500 punch from 0x00b99c to 0x010a0e (0x5073 bytes) 000869 1773668836.336927 trunc from 0x030110 to 0x021157 (0xefb8 bytes) 000870 1773668836.345472 read 0x01eda8 thru 0x021157 (0x23b0 bytes) 000871 1773668836.346088 fallocate 0x034e8b thru 0x03ffff (0xb175 bytes) 000872 1773668836.349397 read 0x005bb5 thru 0x013c42 (0xe08e bytes) 000873 1773668836.354754 punch from 0x005bf6 to 0x007b5b (0x1f66 bytes) 000875 1773668836.363426 mapwrite 0x013a5c thru 0x015185 (0x172a bytes) 000876 1773668836.375477 write 0x01176c thru 0x01ea25 (0xd2ba bytes) 000877 1773668836.381921 MIRROR_RESYNC 5 mirrors 000878 1773668837.327155 read 0x01437b thru 0x01d97f (0x9605 bytes) 000879 1773668837.336796 fallocate 0x01368b thru 0x013fde (0x954 bytes) 000880 1773668837.368482 punch from 0x0142b9 to 0x021157 (0xce9f bytes) 000881 1773668837.379100 mapwrite 0x0359ff thru 0x038059 (0x265b bytes) 000882 1773668837.431285 mapread 0x0185d2 thru 0x01b358 (0x2d87 bytes) 000883 1773668837.437162 write 0x03eadf thru 0x03ffff (0x1521 bytes) 000884 1773668837.438365 trunc from 0x00c2c6 to 0x016ba5 (0xa8e0 bytes) 000885 1773668837.446410 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3b015402 (stale) Mirror 10: 0x3b015402 (stale) Mirror 11: 0x3b015402 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3b015402 (stale) Mirror 10: 0x3b015402 (stale) Mirror 11: 0x3b015402 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000886 1773668837.656073 write 0x00be50 thru 0x014885 (0x8a36 bytes) 000887 1773668837.673662 write 0x024308 thru 0x02c7a3 (0x849c bytes) 000888 1773668837.680595 read 0x029f35 thru 0x030579 (0x6645 bytes) 000889 1773668837.688583 trunc from 0x040000 to 0x011be9 (0x2e416 bytes) 000890 1773668837.697011 read 0x002cd1 thru 0x00d44a (0xa77a bytes) 000891 1773668837.705907 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x11bea): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11bea): Mirror 8: 0x52e32e8b (stale) Mirror 10: 0x52e32e8b (stale) Mirror 11: 0x52e32e8b (stale) CRC-32 checksum value for chunk [0, 0x11bea): Mirror 1: 0xd8362d92 Mirror 12: 0xd8362d92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11bea: skipped 000892 1773668838.283172 trunc from 0x0065f7 to 0x011be9 (0xb5f3 bytes) 000893 1773668838.310125 trunc from 0x011bea to 0x03f5d4 (0x2d9eb bytes) 000894 1773668838.354470 mapread 0x02e4df thru 0x033574 (0x5096 bytes) 000895 1773668838.370188 punch from 0x01f1de to 0x02c97d (0xd7a0 bytes) 000896 1773668838.375141 fallocate 0x03814d thru 0x03ffff (0x7eb3 bytes) 000897 1773668838.381165 read 0x0139a5 thru 0x01a6f8 (0x6d54 bytes) 000898 1773668838.390229 read 0x01a0d0 thru 0x026f3d (0xce6e bytes) 000899 1773668838.397333 read 0x01a077 thru 0x025044 (0xafce bytes) 000900 1773668838.397907 mapread 0x009792 thru 0x0140d1 (0xa940 bytes) 000901 1773668838.399006 write 0x027f9e thru 0x02b9e0 (0x3a43 bytes) 000902 1773668838.400335 fallocate 0x00e97a thru 0x012035 (0x36bc bytes) 000903 1773668838.405117 mapread 0x02a693 thru 0x0375ea (0xcf58 bytes) 000904 1773668838.408571 mapwrite 0x0347a4 thru 0x03d20b (0x8a68 bytes) 000905 1773668838.422222 write 0x00d033 thru 0x00fceb (0x2cb9 bytes) 000906 1773668838.426842 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3b015402 (stale) Mirror 10: 0x3b015402 (stale) Mirror 11: 0x3b015402 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3b015402 (stale) Mirror 10: 0x3b015402 (stale) Mirror 11: 0x3b015402 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000907 1773668838.685553 fallocate 0x00f6cb thru 0x017286 (0x7bbc bytes) 000908 1773668838.693806 fallocate 0x0374b9 thru 0x03ffff (0x8b47 bytes) 000909 1773668838.699941 punch from 0x00823a to 0x00a225 (0x1fec bytes) 000910 1773668838.705052 mapwrite 0x03a818 thru 0x03ffff (0x57e8 bytes) 000911 1773668838.763975 mapwrite 0x02d28f thru 0x03cab1 (0xf823 bytes) 000912 1773668838.799436 MIRROR_RESYNC 4 mirrors 000913 1773668839.784927 fallocate 0x0354aa thru 0x03fa54 (0xa5ab bytes) 000914 1773668839.817301 read 0x0070fd thru 0x01091b (0x981f bytes) 000915 1773668839.836557 punch from 0x031981 to 0x03e351 (0xc9d1 bytes) 000916 1773668839.845508 mapread 0x020d12 thru 0x02d2dc (0xc5cb bytes) 000917 1773668839.893781 mapwrite 0x01ba54 thru 0x022d4b (0x72f8 bytes) 000918 1773668839.924700 mapwrite 0x03880e thru 0x03ffff (0x77f2 bytes) 000919 1773668839.957176 trunc from 0x040000 to 0x0189ca (0x27635 bytes) 000920 1773668839.980163 mapwrite 0x038d2e thru 0x03a912 (0x1be5 bytes) 000921 1773668840.016949 mapread 0x02f433 thru 0x0348e9 (0x54b7 bytes) 000922 1773668840.037606 punch from 0x017ae3 to 0x025a1a (0xdf38 bytes) 000923 1773668840.045805 read 0x01fdf4 thru 0x02a142 (0xa34f bytes) 000924 1773668840.060491 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a913): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a913): Mirror 8: 0x4205417f (stale) Mirror 10: 0x4205417f (stale) Mirror 11: 0x4205417f (stale) CRC-32 checksum value for chunk [0, 0x3a913): Mirror 1: 0x502b347 Mirror 12: 0x502b347 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a913: skipped 000925 1773668840.838581 trunc from 0x03a913 to 0x03aedc (0x5ca bytes) 000927 1773668840.900115 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3aedd): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aedd): Mirror 8: 0xdbcef997 (stale) Mirror 10: 0xdbcef997 (stale) Mirror 11: 0xdbcef997 (stale) Mirror 12: 0xa999f36e (stale) CRC-32 checksum value for chunk [0, 0x3aedd): Mirror 1: 0xa999f36e Mirror 13: 0xa999f36e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aedd: skipped 000928 1773668841.764444 write 0x0065ec thru 0x00f5a3 (0x8fb8 bytes) 000930 1773668841.815192 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x27d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aedd): Mirror 10: 0xdbcef997 (stale) Mirror 11: 0xdbcef997 (stale) Mirror 12: 0xa999f36e (stale) Mirror 13: 0xa999f36e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aedd): Mirror 10: 0xdbcef997 (stale) Mirror 11: 0xdbcef997 (stale) Mirror 12: 0xa999f36e (stale) Mirror 13: 0xa999f36e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aedd: skipped 000932 1773668842.160527 trunc from 0x03aedd to 0x01ca67 (0x1e475 bytes) 000933 1773668842.179241 write 0x022daf thru 0x02f928 (0xcb7a bytes) 000934 1773668842.184377 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f929): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f929): Mirror 10: 0xef1fbd19 (stale) Mirror 11: 0xef1fbd19 (stale) Mirror 12: 0x5f92d36c (stale) Mirror 13: 0x5f92d36c (stale) CRC-32 checksum value for chunk [0, 0x2f929): Mirror 1: 0xc2bda22 Mirror 14: 0xc2bda22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f929: skipped 000935 1773668842.979563 fallocate 0x030506 thru 0x03b63b (0xb136 bytes) 000937 1773668843.024602 trunc from 0x012ef7 to 0x01d4ad (0xa5b7 bytes) 000939 1773668843.031229 fallocate 0x011b1c thru 0x0165a9 (0x4a8e bytes) 000940 1773668843.038252 read 0x0122a4 thru 0x015d31 (0x3a8e bytes) 000941 1773668843.050601 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f929): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f929): Mirror 10: 0xef1fbd19 (stale) Mirror 11: 0xef1fbd19 (stale) Mirror 12: 0x5f92d36c (stale) Mirror 13: 0x5f92d36c (stale) Mirror 14: 0xc2bda22 (stale) CRC-32 checksum value for chunk [0, 0x2f929): Mirror 1: 0xc2bda22 Mirror 15: 0xc2bda22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f929: skipped 000942 1773668844.051399 trunc from 0x02f929 to 0x01ddbd (0x11b6b bytes) 000943 1773668844.112270 mapread 0x0027b0 thru 0x0104b8 (0xdd09 bytes) 000944 1773668844.140096 trunc from 0x01ddbe to 0x027b48 (0x9d8b bytes) 000945 1773668844.154311 fallocate 0x00f3f9 thru 0x018546 (0x914e bytes) 000946 1773668844.160949 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x27b49): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b49): Mirror 10: 0x4368efe6 (stale) Mirror 11: 0x4368efe6 (stale) Mirror 12: 0x4368efe6 (stale) Mirror 13: 0x4368efe6 (stale) Mirror 14: 0xcfeba7ea (stale) Mirror 15: 0xcfeba7ea (stale) CRC-32 checksum value for chunk [0, 0x27b49): Mirror 1: 0xdf1eafe9 Mirror 16: 0xdf1eafe9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b49: skipped 000947 1773668845.069581 trunc from 0x0125c7 to 0x02185b (0xf295 bytes) 000948 1773668845.108179 punch from 0x01a30e to 0x0212a4 (0x6f97 bytes) 000949 1773668845.112091 trunc from 0x027b49 to 0x00d519 (0x1a62f bytes) 000950 1773668845.133500 fallocate 0x00dae8 thru 0x00de60 (0x379 bytes) 000951 1773668845.144171 write 0x0306ab thru 0x036081 (0x59d7 bytes) 000952 1773668845.148274 trunc from 0x036082 to 0x012bb4 (0x234cd bytes) 000953 1773668845.172274 mapread 0x00451b thru 0x00f0c3 (0xaba9 bytes) 000954 1773668845.188717 read 0x00b693 thru 0x00e29c (0x2c0a bytes) 000955 1773668845.190035 mapwrite 0x009882 thru 0x011df1 (0x8570 bytes) 000956 1773668845.211577 trunc from 0x012bb5 to 0x008d05 (0x9eaf bytes) 000957 1773668845.230949 write 0x008e61 thru 0x018485 (0xf625 bytes) 000958 1773668845.236163 trunc from 0x006c2f to 0x009c63 (0x3035 bytes) 000959 1773668845.253399 punch from 0x002f22 to 0x010ab3 (0xdb92 bytes) 000960 1773668845.266348 write 0x013b60 thru 0x01f62b (0xbacc bytes) 000961 1773668845.271154 punch from 0x008aa0 to 0x018072 (0xf5d3 bytes) 000962 1773668845.286009 mapwrite 0x006ffb thru 0x013f35 (0xcf3b bytes) 000963 1773668845.329153 mapwrite 0x00b4d0 thru 0x015a8f (0xa5c0 bytes) 000964 1773668845.354428 trunc from 0x01f62c to 0x03340f (0x13de4 bytes) 000965 1773668845.374437 fallocate 0x03f7a2 thru 0x03ffff (0x85e bytes) 000966 1773668845.380677 trunc from 0x033410 to 0x0144dd (0x1ef32 bytes) 000967 1773668845.395538 mapread 0x006a27 thru 0x008311 (0x18eb bytes) 000968 1773668845.396907 mapread 0x00fb62 thru 0x0144dd (0x497c bytes) 000969 1773668845.398101 trunc from 0x0144de to 0x00804e (0xc48f bytes) 000970 1773668845.412511 mapread 0x001625 thru 0x00804e (0x6a2a bytes) 000971 1773668845.414388 mapread 0x0000ee thru 0x00804e (0x7f61 bytes) 000972 1773668845.420263 fallocate 0x02dec6 thru 0x0389d2 (0xab0d bytes) 000973 1773668845.429846 MIRROR_RESYNC 8 mirrors 000974 1773668847.014385 fallocate 0x02dfdc thru 0x03b4b2 (0xd4d7 bytes) 000975 1773668847.060338 trunc from 0x001451 to 0x005242 (0x3df2 bytes) 000976 1773668847.069369 punch from 0x0149e9 to 0x0174cd (0x2ae5 bytes) 000977 1773668847.077118 mapwrite 0x01b617 thru 0x02a733 (0xf11d bytes) 000979 1773668847.176714 fallocate 0x029e49 thru 0x0307d1 (0x6989 bytes) 000980 1773668847.185117 punch from 0x006d49 to 0x008b10 (0x1dc8 bytes) 000981 1773668847.192824 read 0x00a303 thru 0x00b032 (0xd30 bytes) 000983 1773668847.205671 punch from 0x01e607 to 0x020659 (0x2053 bytes) 000984 1773668847.218649 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x389d3): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389d3): Mirror 10: 0x675bbbb8 (stale) Mirror 11: 0x675bbbb8 (stale) Mirror 12: 0x675bbbb8 (stale) Mirror 13: 0x675bbbb8 (stale) Mirror 14: 0x675bbbb8 (stale) Mirror 15: 0x675bbbb8 (stale) Mirror 16: 0x675bbbb8 (stale) CRC-32 checksum value for chunk [0, 0x389d3): Mirror 1: 0x2ba5f7bb Mirror 17: 0x2ba5f7bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389d3: skipped 000985 1773668848.131547 write 0x01abcf thru 0x01f27b (0x46ad bytes) 000986 1773668848.205068 trunc from 0x0389d3 to 0x01ac68 (0x1dd6a bytes) 000987 1773668848.241813 punch from 0x003a3d to 0x00cff4 (0x95b8 bytes) 000989 1773668848.247069 trunc from 0x0139b2 to 0x018cc0 (0x530f bytes) 000990 1773668848.252480 mapwrite 0x012b1b thru 0x01df5c (0xb442 bytes) 000991 1773668848.306379 trunc from 0x015691 to 0x01c959 (0x72c9 bytes) 000992 1773668848.312372 trunc from 0x01df5d to 0x035340 (0x173e4 bytes) 000993 1773668848.329187 trunc from 0x01764e to 0x01b4fb (0x3eae bytes) 000994 1773668848.334472 trunc from 0x02489f to 0x02abe8 (0x634a bytes) 000995 1773668848.339377 mapread 0x0307fd thru 0x035340 (0x4b44 bytes) 000996 1773668848.349194 read 0x022750 thru 0x02cfc6 (0xa877 bytes) 000997 1773668848.362209 read 0x00a53d thru 0x00d652 (0x3116 bytes) 000998 1773668848.371236 write 0x01651a thru 0x0228aa (0xc391 bytes) 000999 1773668848.372504 write 0x03c41a thru 0x03ffff (0x3be6 bytes) 001000 1773668848.374122 read 0x008ed2 thru 0x009230 (0x35f bytes) All operations completed A-OK! PASS 70a (88s) == sanity-flr test 71: check mirror extend parent fid ==== 09:47:32 (1773668852) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x2e:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 09:47:39 (1773668859) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189624 s, 11.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104104 s, 10.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x8c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 09:47:46 (1773668866) Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre2 Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4107872 .. Extending file size to 8551872 .. failed Extending file size to 7232384 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8631200 .. Extending file size to 9186592 .. Extending file size to 8820224 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9279424 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9083616 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60056 60057 60058 60061 60062 done 192.168.203.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg327-client.virtnet /mnt/lustre2 (opts:) 192.168.203.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg327-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:49:01 (1773668941) Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre2 Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6063520 .. Extending file size to 8846912 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6518464 .. Extending file size to 6544704 .. Extending file size to 6788736 .. Extending file size to 8018880 .. Extending file size to 8859808 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 61770 61771 61773 61774 61778 192.168.203.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg327-client.virtnet /mnt/lustre2 (opts:) 192.168.203.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg327-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:50:16 (1773669016) Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2280 1269024 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17500 3589520 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25008 3582012 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42508 7171532 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2280 1269024 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17500 3589520 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25008 3582012 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42508 7171532 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0541097 s, 38.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0672064 s, 31.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7ebb4d81 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg327-client.virtnet /mnt/lustre2 (opts:) PASS 200c (9s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:50:26 (1773669026) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0574178 s, 36.5 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 09:50:30 (1773669030) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3d:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa4:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:50:33 (1773669033) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0682983 s, 30.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (4s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:50:37 (1773669037) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.048979 s, 42.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa8:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:50:42 (1773669042) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.052582 s, 39.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaa:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:51:09 (1773669069) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg327-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270311 s, 38.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x336:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x339:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x338:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x336:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x339:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x338:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x337:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:51:51 (1773669111) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg327-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0673302 s, 31.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (42s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:52:33 (1773669153) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg327-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg327-server: Pool lustre.test_204f created Adding targets to pool oleg327-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg327-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22798 s, 9.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg327-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg327-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg327-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:53:42 (1773669222) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:53:51 (1773669231) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:54:00 (1773669240) oleg327-server: Pool lustre.test_206 created Adding targets to pool oleg327-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg327-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg327-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg327-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg327-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:54:26 (1773669266) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408751 s, 25.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171236 s, 61.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbe8619b6 Mirror 3: 0xbe8619b6 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:54:34 (1773669274) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:54:37 (1773669277) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:54:40 (1773669280) Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0918625 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x830000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2070000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2070000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2070000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1390000 Read this (4 bytes): BBBB 192.168.203.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg327-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:55:14 (1773669314) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0343115 s, 239 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:55:21 (1773669321) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708178 s, 14.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x60 lmm_fid: [0x200000405:0x60:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x60 lmm_fid: [0x200000405:0x60:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611232 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-16 09:55:22.000000000 -0400 Modify: 2026-03-16 09:55:22.000000000 -0400 Change: 2026-03-16 09:55:23.000000000 -0400 Birth: 2026-03-16 09:55:22.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:55:28 (1773669328) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379632 s, 27.6 MB/s oleg327-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg327-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg327-server: O/280000401/d21/b5: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:55:47 (1773669347) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.17624 s, 232 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0148903 s, 275 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 16 09:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x65:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 16 09:55 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:55:53 (1773669353) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:55:58 (1773669358) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1110 sec =========== 09:56:03 (1773669363) === sanity-flr: start cleanup 09:56:04 (1773669364) === === sanity-flr: finish cleanup 09:56:09 (1773669369) ===