-----============= acceptance-small: sanity-flr ============----- Mon Nov 17 12:51:16 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg245-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:51:35 (1763401895) === oleg245-client.virtnet: executing check_config_client /mnt/lustre oleg245-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg245-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa024644d8800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa024644d8800.idle_timeout=debug disable quota as required oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:52:10 (1763401930) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8183] == sanity-flr test 0a: lfs mirror create with -N option == 12:52:12 (1763401932) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:52:26 (1763401946) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:52:29 (1763401949) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:52:32 (1763401952) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:52:52 (1763401972) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:52:55 (1763401975) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 12:52:58 (1763401978) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa024644d8800.stats=clear osc.lustre-OST0001-osc-ffffa024644d8800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:53:05 (1763401985) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:53:14 (1763401994) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:53:22 (1763402002) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:53:26 (1763402006) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 12:53:34 (1763402014) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:53:41 (1763402021) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:53:48 (1763402028) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:53:56 (1763402036) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:54:05 (1763402045) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:54:12 (1763402052) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.421407 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506261 s, 20.7 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:54:19 (1763402059) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.32926 s, 41.4 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:54:26 (1763402066) Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (61s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:55:27 (1763402127) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.509948 s, 16.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (33s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:56:00 (1763402160) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (62s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:57:02 (1763402222) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.54588 s, 20.6 MB/s -rw-r--r-- 1 root root 52428800 Nov 17 12:57 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.832s user 0m0.000s sys 0m0.154s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa024644d8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.722s user 0m0.002s sys 0m0.182s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.246s user 0m0.000s sys 0m0.149s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa024644d8800.ost_server_uuid in FULL state after 0 sec PASS 33b (76s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:58:18 (1763402298) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:58:21 (1763402301) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:58:24 (1763402304) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 12:58:27 (1763402307) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 12:58:34 (1763402314) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.331944 s, 37.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:58:43 (1763402323) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293369 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482261 s, 21.7 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:58:49 (1763402329) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:58:56 (1763402336) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00067152 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00238393 s, 430 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00526449 s, 195 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00636646 s, 161 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 12:59:05 (1763402345) create test files with size 15758573 .. cda016004eb669f38fe2897f42d12ef9 - 9e341d92187a855c2ae762024f8847d6 - 6b8bae12d60995d622d4fddc6e387544 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 12:59:25 (1763402365) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (10s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:59:35 (1763402375) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:59:42 (1763402382) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096453 s, 10.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305755 s, 34.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 12:59:50 (1763402390) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0804079 s, 26.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.324842 s, 12.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165949 s, 12.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 13:00:03 (1763402403) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 13:00:07 (1763402407) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:00:11 (1763402411) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746516 s, 14.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 13:00:19 (1763402419) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.302008 s, 10.4 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:00:28 (1763402428) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x44:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x45:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:00:38 (1763402438) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.258281 s, 40.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:00:48 (1763402448) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa0246459b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.259902 s, 40.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa02444dc1000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:01:01 (1763402461) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:01:35 (1763402495) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:02:09 (1763402529) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 13:02:12 (1763402532) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.185186 s, 17.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.108915 s, 28.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb5201751 (stale) Mirror 2: 0xb5201751 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: c1db1e15b797b6dd853c7fd2bd7ffc1e - mirror 1 checksum: 9222949400fa75b8ad24ebb09459313f - mirror 2 checksum: c1db1e15b797b6dd853c7fd2bd7ffc1e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Nov 17 13:02:13 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Nov 17 13:02:13 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c1db1e15b797b6dd853c7fd2bd7ffc1e - mirror 1 checksum: 9222949400fa75b8ad24ebb09459313f - mirror 2 checksum: 9222949400fa75b8ad24ebb09459313f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Nov 17 13:02:13 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4e:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Nov 17 13:02:13 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:02:20 (1763402540) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291068 s, 36.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (5s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:02:25 (1763402545) Failing mds1 on oleg245-server Stopping /mnt/lustre-mds1 (opts:) on oleg245-server 13:02:36 (1763402556) shut down facet: mds1 facet_host: oleg245-server facet_failover_host: oleg245-server Failover mds1 to oleg245-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 13:03:07 (1763402587) targets are mounted 13:03:07 (1763402587) facet_failover done oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250392 s, 41.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x51:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:03:18 (1763402598) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0412256 s, 497 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.015229 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:03:26 (1763402606) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0620786 s, 330 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00849324 s, 2.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:03:35 (1763402615) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0115336 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:03:42 (1763402622) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0750922 s, 273 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:03:49 (1763402629) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00600716 s, 682 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0182457 s, 224 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:03:55 (1763402635) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:04:13 (1763402653) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:04:41 (1763402681) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:05:03 (1763402703) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:05:24 (1763402724) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 13:05:30 (1763402730) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 55905 mirror split pid 55906 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55905 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55906 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 13:06:36 (1763402796) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2798 skipping zero length zero range truncating to largest ever: 0x11b8e 000002 1763402798.648664 trunc from 00000000 to 0x011b8d (0x11b8e bytes) 000003 1763402798.679724 punch from 0x00391f to 0x007f18 (0x45fa bytes) 000004 1763402798.690278 mapread 0x00aa9c thru 0x00b548 (0xaad bytes) 000005 1763402798.699726 read 0x00fd38 thru 0x011b8d (0x1e56 bytes) 000006 1763402798.704744 write 0x006a55 thru 0x015e28 (0xf3d4 bytes) 000008 1763402798.716600 read 0x00225a thru 0x0114da (0xf281 bytes) 000009 1763402798.717258 mapwrite 0x03f1ec thru 0x03ffff (0xe14 bytes) 000010 1763402798.767728 write 0x0046eb thru 0x00e51b (0x9e31 bytes) 000011 1763402798.769783 mapwrite 0x011035 thru 0x015e18 (0x4de4 bytes) 000014 1763402798.789621 read 0x011ec0 thru 0x0175b4 (0x56f5 bytes) 000015 1763402798.790262 write 0x038d64 thru 0x03ffff (0x729c bytes) zero_range to largest ever: 0x23832 000016 1763402798.791351 trunc from 0x01465c to 0x023831 (0xf1d6 bytes) 000017 1763402798.799234 trunc from 0x040000 to 0x009fde (0x36021 bytes) 000018 1763402798.833697 write 0x00c86d thru 0x0110bf (0x4853 bytes) truncating to largest ever: 0x3e714 000019 1763402798.834983 trunc from 0x0110c0 to 0x03e713 (0x2d654 bytes) 000020 1763402798.858628 trunc from 0x01e84f to 0x02b237 (0xc9e9 bytes) 000021 1763402798.865647 read 0x03030c thru 0x0309e7 (0x6dc bytes) 000022 1763402798.887473 MIRROR_EXTEND to 2 mirrors 000023 1763402799.604715 mapread 0x00ba47 thru 0x016882 (0xae3c bytes) 000024 1763402799.644236 mapread 0x0116ea thru 0x01b572 (0x9e89 bytes) 000025 1763402799.662414 punch from 0x0342bf to 0x03e713 (0xa455 bytes) 000026 1763402799.706557 trunc from 0x02b6cd to 0x02f1a1 (0x3ad5 bytes) 000027 1763402799.711367 read 0x0284ec thru 0x02fc82 (0x7797 bytes) 000028 1763402799.719517 trunc from 0x03e714 to 0x022f85 (0x1b78e bytes) 000029 1763402799.796178 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) 000030 1763402800.076286 punch from 0x009b31 to 0x00cfad (0x347d bytes) 000031 1763402800.090158 mapread 0x0090ab thru 0x011b7c (0x8ad2 bytes) 000033 1763402800.134230 mapwrite 0x009ce4 thru 0x010aa6 (0x6dc3 bytes) 000034 1763402800.184188 trunc from 0x022f86 to 0x010d60 (0x12225 bytes) 000036 1763402800.232489 trunc from 0x010d61 to 0x010b78 (0x1e8 bytes) 000037 1763402800.268243 mapwrite 0x0357de thru 0x0378e4 (0x2107 bytes) 000038 1763402800.307467 punch from 0x029b7a to 0x02cd72 (0x31f9 bytes) 000039 1763402800.312749 write 0x01e0db thru 0x029133 (0xb059 bytes) 000040 1763402800.345045 mapread 0x0047e1 thru 0x01368d (0xeead bytes) 000042 1763402800.367168 write 0x0148c7 thru 0x01dfcf (0x9709 bytes) 000044 1763402800.368865 mapread 0x01a15f thru 0x02830d (0xe1af bytes) 000045 1763402800.370935 trunc from 0x0378e5 to 0x020732 (0x171b2 bytes) 000047 1763402800.400596 punch from 0x015ca9 to 0x019da0 (0x40f8 bytes) 000048 1763402800.454455 trunc from 0x001716 to 0x009d5d (0x8648 bytes) 000049 1763402800.467101 mapread 0x0032ab thru 0x0112a9 (0xdfff bytes) 000050 1763402800.482042 read 0x0038dd thru 0x00e9d6 (0xb0fa bytes) 000051 1763402800.486338 fallocate 0x010514 thru 0x016100 (0x5bed bytes) 000052 1763402800.492333 trunc from 0x01631d to 0x01dcaa (0x798e bytes) 000053 1763402800.500451 write 0x03123c thru 0x03ffff (0xedc4 bytes) 000054 1763402800.502244 trunc from 0x00b77e to 0x01b75f (0xffe2 bytes) 000055 1763402800.517447 trunc from 0x016c1e to 0x01a376 (0x3759 bytes) 000056 1763402800.522379 trunc from 0x040000 to 0x03db4e (0x24b1 bytes) 000057 1763402800.564678 fallocate 0x019c8e thru 0x01c00c (0x237f bytes) 000058 1763402800.571517 fallocate 0x0145e8 thru 0x01cad7 (0x84f0 bytes) 000059 1763402800.576478 write 0x03a7ce thru 0x03ffff (0x5832 bytes) 000060 1763402800.578120 read 0x036c61 thru 0x03ffff (0x939f bytes) 000061 1763402800.579447 trunc from 0x040000 to 0x02881b (0x177e4 bytes) 000062 1763402800.604363 write 0x01612c thru 0x025942 (0xf817 bytes) 000063 1763402800.617607 trunc from 0x02881c to 0x02f06a (0x684f bytes) 000064 1763402800.639727 mapwrite 0x000903 thru 0x0055f9 (0x4cf7 bytes) 000065 1763402800.659097 punch from 0x01302b to 0x01f665 (0xc63b bytes) 000067 1763402800.665259 mapwrite 0x0210cf thru 0x021f62 (0xe94 bytes) 000068 1763402800.677496 mapread 0x027d29 thru 0x02f06a (0x7342 bytes) 000069 1763402800.692279 mapwrite 0x00fb4d thru 0x0130a9 (0x355d bytes) 000070 1763402800.711429 mapread 0x006a6a thru 0x00b84b (0x4de2 bytes) 000071 1763402800.712299 trunc from 0x027bb4 to 0x02c454 (0x48a1 bytes) 000072 1763402800.719379 read 0x006c0a thru 0x014a98 (0xde8f bytes) 000073 1763402800.729381 fallocate 0x00460f thru 0x00a208 (0x5bfa bytes) 000074 1763402800.734342 mapwrite 0x02e2e1 thru 0x030111 (0x1e31 bytes) 000075 1763402800.775138 trunc from 0x004070 to 0x009df8 (0x5d89 bytes) 000076 1763402800.781235 mapwrite 0x012ec5 thru 0x019ee0 (0x701c bytes) 000077 1763402800.836346 trunc from 0x030112 to 0x018d4b (0x173c6 bytes) 000079 1763402800.859592 read 0x018d29 thru 0x018d4b (0x23 bytes) 000081 1763402800.862077 mapread 0x0081e4 thru 0x00c5f7 (0x4414 bytes) fallocating to largest ever: 0x40000 000083 1763402800.871399 fallocate 0x03f692 thru 0x03ffff (0x96e bytes) 000084 1763402800.878132 trunc from 0x040000 to 0x033b7f (0xc480 bytes) 000085 1763402800.902793 punch from 0x01f8f3 to 0x0202a0 (0x9ae bytes) 000086 1763402800.911691 mapread 0x00968e thru 0x00f451 (0x5dc4 bytes) 000087 1763402800.912644 trunc from 0x019ea0 to 0x025b83 (0xbce4 bytes) 000088 1763402800.921147 write 0x009c14 thru 0x01765d (0xda4a bytes) 000089 1763402800.924660 mapread 0x00792e thru 0x00b32f (0x3a02 bytes) 000090 1763402800.926574 trunc from 0x013165 to 0x01a7a8 (0x7644 bytes) 000091 1763402800.945130 trunc from 0x01125b to 0x013a4a (0x27f0 bytes) 000093 1763402800.952764 trunc from 0x006b70 to 0x00aee2 (0x4373 bytes) 000094 1763402800.959522 trunc from 0x025230 to 0x0263aa (0x117b bytes) 000095 1763402800.968580 write 0x005d82 thru 0x008200 (0x247f bytes) 000096 1763402800.984258 read 0x00f596 thru 0x01beb9 (0xc924 bytes) 000097 1763402800.997694 write 0x005c71 thru 0x007027 (0x13b7 bytes) 000098 1763402800.999430 mapwrite 0x005c91 thru 0x00762f (0x199f bytes) 000099 1763402801.015412 fallocate 0x02cb53 thru 0x039968 (0xce16 bytes) 000100 1763402801.024439 mapwrite 0x034617 thru 0x03ffff (0xb9e9 bytes) 000101 1763402801.117131 trunc from 0x040000 to 0x025f9a (0x1a065 bytes) 000102 1763402801.156596 mapwrite 0x0354d2 thru 0x03bcee (0x681d bytes) 000103 1763402801.242564 trunc from 0x03bcef to 0x0061eb (0x35b03 bytes) 000104 1763402801.275595 write 0x013dc2 thru 0x022341 (0xe580 bytes) 000105 1763402801.277402 fallocate 0x017e2c thru 0x02652e (0xe703 bytes) 000106 1763402801.282958 write 0x0355dc thru 0x03ffff (0xaa24 bytes) 000107 1763402801.284302 trunc from 0x040000 to 0x0001ff (0x3fe00 bytes) 000109 1763402801.311604 write 0x008d90 thru 0x00a0a6 (0x1317 bytes) 000110 1763402801.312636 mapread 0x00751d thru 0x0083cb (0xeaf bytes) 000111 1763402801.327124 trunc from 0x008d6b to 0x00a0a6 (0x133c bytes) 000112 1763402801.348195 trunc from 0x007ba3 to 0x00a0a6 (0x2504 bytes) 000113 1763402801.358495 trunc from 0x00a0a7 to 0x01d5ac (0x13506 bytes) 000115 1763402801.391765 punch from 0x008a70 to 0x00efb7 (0x6548 bytes) 000116 1763402801.405524 mapread 0x0079bf thru 0x012ae4 (0xb126 bytes) 000117 1763402801.486226 write 0x02abe2 thru 0x0322da (0x76f9 bytes) 000118 1763402801.487540 punch from 0x001a5f to 0x0054a1 (0x3a43 bytes) 000120 1763402801.497848 read 0x022c71 thru 0x031a5b (0xedeb bytes) 000121 1763402801.514203 trunc from 0x0269cb to 0x02cb57 (0x618d bytes) 000124 1763402801.526106 punch from 0x00aef2 to 0x016a5d (0xbb6c bytes) 000125 1763402801.533874 read 0x01386c thru 0x014645 (0xdda bytes) 000126 1763402801.545276 trunc from 0x0322db to 0x003346 (0x2ef94 bytes) 000127 1763402801.562409 fallocate 0x02f28d thru 0x03ef0b (0xfc7f bytes) 000128 1763402801.571046 MIRROR_EXTEND to 2 mirrors 000129 1763402802.132458 mapwrite 0x00ce21 thru 0x00de32 (0x1012 bytes) 000130 1763402802.282195 mapwrite 0x027210 thru 0x02a0c1 (0x2eb2 bytes) 000131 1763402802.305783 fallocate 0x0304f0 thru 0x0376cd (0x71de bytes) 000132 1763402802.328341 MIRROR_RESYNC 2 mirrors 000133 1763402803.027970 mapread 0x008467 thru 0x00daed (0x5687 bytes) 000134 1763402803.075649 punch from 0x020a30 to 0x026c95 (0x6266 bytes) 000135 1763402803.144778 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ef0c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef0c): Mirror 2: 0x26ee86b2 (stale) CRC-32 checksum value for chunk [0, 0x3ef0c): Mirror 1: 0x26ee86b2 Mirror 3: 0x26ee86b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef0c: skipped 000136 1763402804.163971 fallocate 0x0334d6 thru 0x034830 (0x135b bytes) 000138 1763402804.201331 punch from 0x017823 to 0x021578 (0x9d56 bytes) 000139 1763402804.205849 mapwrite 0x0180b8 thru 0x01c94a (0x4893 bytes) 000142 1763402804.253696 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ef0c): Mirror 3: 0x26ee86b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef0c): Mirror 3: 0x26ee86b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef0c: skipped 000143 1763402804.507257 punch from 0x02c3f6 to 0x03628c (0x9e97 bytes) 000144 1763402804.513379 read 0x03332f thru 0x03ef0b (0xbbdd bytes) 000146 1763402804.522643 mapread 0x001385 thru 0x00f06d (0xdce9 bytes) 000148 1763402804.542599 mapread 0x0384f9 thru 0x03ef0b (0x6a13 bytes) 000149 1763402804.543553 fallocate 0x031ca5 thru 0x03560c (0x3968 bytes) 000150 1763402804.548832 fallocate 0x0024b6 thru 0x009ac1 (0x760c bytes) 000151 1763402804.554514 read 0x017935 thru 0x01b8e6 (0x3fb2 bytes) 000152 1763402804.565286 mapread 0x00754c thru 0x010738 (0x91ed bytes) 000153 1763402804.566249 trunc from 0x03ef0c to 0x02a026 (0x14ee5 bytes) 000154 1763402804.614513 fallocate 0x03657d thru 0x03c84f (0x62d3 bytes) 000155 1763402804.626052 read 0x01fdf7 thru 0x028a43 (0x8c4d bytes) 000157 1763402804.627134 mapread 0x01c6b2 thru 0x021847 (0x5196 bytes) 000158 1763402804.628037 read 0x03498c thru 0x0396c4 (0x4d39 bytes) 000159 1763402804.638907 mapread 0x016ddd thru 0x01b243 (0x4467 bytes) 000160 1763402804.639769 mapread 0x036f6c thru 0x03c84f (0x58e4 bytes) 000161 1763402804.640607 write 0x03f62a thru 0x03ffff (0x9d6 bytes) 000162 1763402804.641482 read 0x03f322 thru 0x03ffff (0xcde bytes) 000163 1763402804.642303 punch from 0x007bf4 to 0x00f806 (0x7c13 bytes) 000164 1763402804.647376 mapwrite 0x01f2b9 thru 0x021ed6 (0x2c1e bytes) 000165 1763402804.665066 trunc from 0x040000 to 0x03ff67 (0x98 bytes) 000166 1763402804.678003 trunc from 0x03ff68 to 0x036c6d (0x92fa bytes) 000167 1763402804.694682 write 0x020f20 thru 0x02e336 (0xd417 bytes) 000168 1763402804.696155 write 0x020a16 thru 0x029091 (0x867c bytes) 000169 1763402804.697684 fallocate 0x00cb0b thru 0x015d7f (0x9275 bytes) 000170 1763402804.705236 mapread 0x014bd7 thru 0x01818c (0x35b6 bytes) 000171 1763402804.706039 punch from 0x0031b9 to 0x012cae (0xfaf6 bytes) 000172 1763402804.711081 write 0x02d707 thru 0x03a726 (0xd020 bytes) 000173 1763402804.713634 read 0x0056a2 thru 0x00ccdf (0x763e bytes) 000174 1763402804.726087 write 0x00b12a thru 0x013f52 (0x8e29 bytes) 000175 1763402804.727210 read 0x00cef4 thru 0x00dd8d (0xe9a bytes) 000176 1763402804.727706 read 0x02a720 thru 0x02f7f8 (0x50d9 bytes) 000178 1763402804.728615 write 0x01296f thru 0x01e3f6 (0xba88 bytes) 000179 1763402804.729930 write 0x009992 thru 0x00bdfa (0x2469 bytes) 000180 1763402804.730780 trunc from 0x01d660 to 0x01f888 (0x2229 bytes) 000181 1763402804.751518 read 0x002b88 thru 0x012944 (0xfdbd bytes) 000182 1763402804.753531 trunc from 0x03a727 to 0x021bbc (0x18b6a bytes) 000183 1763402804.773546 write 0x02a7fc thru 0x0323ac (0x7bb1 bytes) 000184 1763402804.774896 fallocate 0x00e45b thru 0x0134a4 (0x504a bytes) 000185 1763402804.781199 trunc from 0x0323ad to 0x031360 (0x104c bytes) 000186 1763402804.800204 read 0x002699 thru 0x00f02d (0xc995 bytes) 000187 1763402804.801138 trunc from 0x031361 to 0x02e283 (0x30dd bytes) 000188 1763402804.817964 trunc from 0x02e284 to 0x0169b2 (0x178d1 bytes) 000189 1763402804.836835 trunc from 0x013bbb to 0x0169b2 (0x2df8 bytes) 000191 1763402804.848632 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x169b3): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169b3): Mirror 3: 0xb0cef561 (stale) CRC-32 checksum value for chunk [0, 0x169b3): Mirror 1: 0x69fdee4c Mirror 4: 0x69fdee4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169b3: skipped 000192 1763402805.600313 read 0x00d40f thru 0x01114a (0x3d3c bytes) 000193 1763402805.613788 write 0x0131df thru 0x013e1f (0xc41 bytes) 000194 1763402805.665855 punch from 0x001755 to 0x00c1fb (0xaaa7 bytes) 000195 1763402805.670725 mapread 0x00937c thru 0x009430 (0xb5 bytes) 000196 1763402805.691710 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x169b3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x169b3): Mirror 3: 0xb0cef561 (stale) Mirror 4: 0x69fdee4c (stale) CRC-32 checksum value for chunk [0, 0x169b3): Mirror 1: 0xae7f9c20 Mirror 5: 0xae7f9c20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x169b3: skipped 000198 1763402806.729195 fallocate 0x018112 thru 0x021238 (0x9127 bytes) 000199 1763402806.764038 fallocate 0x02d9f1 thru 0x039b5b (0xc16b bytes) 000200 1763402806.770485 mapwrite 0x014d46 thru 0x017765 (0x2a20 bytes) 000202 1763402806.819871 fallocate 0x01499e thru 0x01cf9d (0x8600 bytes) 000203 1763402806.828606 mapwrite 0x01608a thru 0x01926d (0x31e4 bytes) 000204 1763402806.852902 read 0x001e81 thru 0x00adcb (0x8f4b bytes) 000205 1763402806.860897 trunc from 0x021239 to 0x034533 (0x132fb bytes) 000206 1763402806.898385 read 0x0067f2 thru 0x015cce (0xf4dd bytes) 000207 1763402806.913116 trunc from 0x001460 to 0x0036f7 (0x2298 bytes) 000208 1763402806.921931 fallocate 0x000a90 thru 0x00733e (0x68af bytes) 000209 1763402806.929431 write 0x031c8c thru 0x0338d7 (0x1c4c bytes) 000210 1763402806.930752 mapwrite 0x009201 thru 0x00d55f (0x435f bytes) 000211 1763402806.969649 mapwrite 0x0342a4 thru 0x03ffff (0xbd5c bytes) 000212 1763402807.041004 mapread 0x017079 thru 0x020914 (0x989c bytes) 000213 1763402807.041972 trunc from 0x040000 to 0x03e2b7 (0x1d48 bytes) 000214 1763402807.057326 punch from 0x027767 to 0x032449 (0xace3 bytes) 000215 1763402807.063697 trunc from 0x038566 to 0x03e2b7 (0x5d52 bytes) 000216 1763402807.069022 mapread 0x02947d thru 0x035dfd (0xc981 bytes) 000217 1763402807.132151 trunc from 0x03e2b8 to 0x00132d (0x3cf8a bytes) 000218 1763402807.159384 mapwrite 0x02bd82 thru 0x02f873 (0x3af2 bytes) 000220 1763402807.219108 write 0x02cc29 thru 0x02f4bd (0x2895 bytes) 000222 1763402807.221679 read 0x008a6e thru 0x008bc3 (0x156 bytes) 000223 1763402807.241480 fallocate 0x010c99 thru 0x01a247 (0x95af bytes) 000224 1763402807.255970 MIRROR_RESYNC 4 mirrors 000226 1763402807.982032 mapwrite 0x02fb37 thru 0x03f811 (0xfcdb bytes) 000227 1763402808.113320 mapwrite 0x024ad5 thru 0x032552 (0xda7e bytes) 000229 1763402808.141768 trunc from 0x022498 to 0x03127c (0xede5 bytes) 000230 1763402808.146685 punch from 0x018394 to 0x019779 (0x13e6 bytes) 000231 1763402808.155640 punch from 0x0295e5 to 0x0331a5 (0x9bc1 bytes) 000233 1763402808.162346 write 0x01decf thru 0x027e40 (0x9f72 bytes) 000234 1763402808.168904 fallocate 0x00819f thru 0x00b133 (0x2f95 bytes) 000235 1763402808.173459 write 0x021bc1 thru 0x027973 (0x5db3 bytes) 000236 1763402808.180689 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f812): Mirror 4: 0x98ba3ccf (stale) Mirror 5: 0x98ba3ccf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f812): Mirror 4: 0x98ba3ccf (stale) Mirror 5: 0x98ba3ccf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f812: skipped 000237 1763402808.670537 mapwrite 0x02682f thru 0x02ce91 (0x6663 bytes) 000238 1763402808.777789 punch from 0x01f78c to 0x02d888 (0xe0fd bytes) 000239 1763402808.802199 write 0x0123aa thru 0x0167b3 (0x440a bytes) 000240 1763402808.832045 fallocate 0x03279c thru 0x034b0c (0x2371 bytes) 000241 1763402808.839738 fallocate 0x01828a thru 0x019126 (0xe9d bytes) 000242 1763402808.847304 read 0x031b5b thru 0x03f811 (0xdcb7 bytes) 000243 1763402808.861543 fallocate 0x020d32 thru 0x027d29 (0x6ff8 bytes) 000244 1763402808.868525 fallocate 0x02f1d4 thru 0x039f45 (0xad72 bytes) 000245 1763402808.877586 fallocate 0x01057c thru 0x01efa6 (0xea2b bytes) 000247 1763402808.889497 trunc from 0x038d1a to 0x03f811 (0x6af8 bytes) 000248 1763402808.896023 write 0x02295f thru 0x02638a (0x3a2c bytes) 000249 1763402808.904598 mapwrite 0x00176a thru 0x00f5e1 (0xde78 bytes) 000250 1763402808.935189 read 0x0254af thru 0x0325ed (0xd13f bytes) 000251 1763402808.943616 trunc from 0x03f812 to 0x0370d6 (0x873b bytes) 000252 1763402808.961600 read 0x034cc3 thru 0x0370d6 (0x2414 bytes) 000254 1763402808.963143 read 0x00b0d6 thru 0x00f949 (0x4874 bytes) 000255 1763402808.963643 mapwrite 0x03819f thru 0x03922f (0x1091 bytes) 000256 1763402809.026271 fallocate 0x0267ab thru 0x02fdfd (0x9653 bytes) 000257 1763402809.035135 fallocate 0x03057d thru 0x0320e3 (0x1b67 bytes) 000258 1763402809.041129 trunc from 0x0254fb to 0x0300ce (0xabd4 bytes) 000259 1763402809.053421 MIRROR_RESYNC 3 mirrors 000260 1763402809.950308 punch from 0x0124cd to 0x015280 (0x2db4 bytes) 000261 1763402809.992885 punch from 0x00bde0 to 0x018ed9 (0xd0fa bytes) 000262 1763402809.998362 trunc from 0x033e2a to 0x03922f (0x5406 bytes) 000263 1763402810.003261 punch from 0x02e497 to 0x03922f (0xad99 bytes) 000264 1763402810.008446 read 0x001c80 thru 0x003240 (0x15c1 bytes) 000265 1763402810.016684 trunc from 0x031dbd to 0x0384de (0x6722 bytes) 000267 1763402810.026533 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39230): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39230): Mirror 4: 0xcdf622c2 (stale) Mirror 5: 0xcdf622c2 (stale) CRC-32 checksum value for chunk [0, 0x39230): Mirror 1: 0xf7c537a4 Mirror 6: 0xf7c537a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39230: skipped 000268 1763402811.054596 punch from 0x013fe0 to 0x01771d (0x373e bytes) 000269 1763402811.101182 mapread 0x029ba3 thru 0x0358f7 (0xbd55 bytes) 000270 1763402811.134372 trunc from 0x039230 to 0x03422b (0x5004 bytes) 000271 1763402811.155187 read 0x011760 thru 0x01d034 (0xb8d5 bytes) 000272 1763402811.168622 fallocate 0x020d17 thru 0x027415 (0x66ff bytes) 000274 1763402811.176541 write 0x02441e thru 0x02754f (0x3132 bytes) 000275 1763402811.178659 read 0x0078fb thru 0x008eef (0x15f5 bytes) 000276 1763402811.188008 mapread 0x0153ea thru 0x01667d (0x1294 bytes) 000277 1763402811.188955 mapread 0x02c457 thru 0x03422b (0x7dd5 bytes) 000278 1763402811.190642 trunc from 0x021d3d to 0x02253b (0x7ff bytes) 000279 1763402811.200793 MIRROR_RESYNC 4 mirrors 000280 1763402812.275458 trunc from 0x0072d5 to 0x011817 (0xa543 bytes) 000281 1763402812.322484 mapread 0x0085a3 thru 0x0172be (0xed1c bytes) 000282 1763402812.353289 MIRROR_RESYNC 4 mirrors 000284 1763402813.344619 mapread 0x03403d thru 0x03422b (0x1ef bytes) 000285 1763402813.349336 mapwrite 0x026f0d thru 0x02a1f6 (0x32ea bytes) 000286 1763402813.434292 MIRROR_RESYNC 4 mirrors 000287 1763402814.519299 read 0x02bef5 thru 0x03422b (0x8337 bytes) 000288 1763402814.537383 mapwrite 0x032c41 thru 0x03b69b (0x8a5b bytes) 000289 1763402814.632799 mapread 0x038b2d thru 0x03b69b (0x2b6f bytes) 000290 1763402814.634206 mapread 0x03af0e thru 0x03b69b (0x78e bytes) 000291 1763402814.635793 punch from 0x012bf4 to 0x01ffe6 (0xd3f3 bytes) 000292 1763402814.641120 mapwrite 0x02522c thru 0x03435e (0xf133 bytes) 000293 1763402814.679851 write 0x039242 thru 0x03ffff (0x6dbe bytes) 000294 1763402814.681170 trunc from 0x02e4bf to 0x03e016 (0xfb58 bytes) 000295 1763402814.696585 mapwrite 0x02da31 thru 0x03c03b (0xe60b bytes) 000296 1763402814.746602 trunc from 0x040000 to 0x00c8d4 (0x3372b bytes) 000298 1763402814.778018 write 0x03c1f4 thru 0x03ffff (0x3e0c bytes) 000299 1763402814.780246 trunc from 0x0042c2 to 0x00b9d6 (0x7715 bytes) 000300 1763402814.796805 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb3cca65f (stale) Mirror 6: 0xb3cca65f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb3cca65f (stale) Mirror 6: 0xb3cca65f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1763402815.135668 mapwrite 0x005623 thru 0x010698 (0xb076 bytes) 000302 1763402815.196618 MIRROR_RESYNC 3 mirrors 000303 1763402815.810742 mapwrite 0x00232d thru 0x00b4de (0x91b2 bytes) 000304 1763402815.920528 fallocate 0x0199b3 thru 0x01fe4d (0x649b bytes) 000305 1763402815.925679 write 0x01e636 thru 0x029b40 (0xb50b bytes) 000306 1763402815.938873 read 0x039aa6 thru 0x03ffff (0x655a bytes) 000307 1763402815.948046 punch from 0x00aace to 0x00b849 (0xd7c bytes) 000308 1763402815.954158 read 0x0344be thru 0x0368cb (0x240e bytes) 000309 1763402815.962766 read 0x01e13b thru 0x020716 (0x25dc bytes) 000310 1763402815.963363 mapread 0x030bf0 thru 0x037616 (0x6a27 bytes) 000313 1763402815.965497 punch from 0x03febd to 0x03ffff (0x143 bytes) 000314 1763402815.975260 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x898fbefb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x898fbefb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000315 1763402816.268359 punch from 0x039f85 to 0x03ffff (0x607b bytes) 000316 1763402816.273922 read 0x03ea84 thru 0x03fe3a (0x13b7 bytes) 000317 1763402816.279664 trunc from 0x040000 to 0x00e2d4 (0x31d2b bytes) 000319 1763402816.326913 mapwrite 0x000138 thru 0x00a3d9 (0xa2a2 bytes) 000320 1763402816.367885 trunc from 0x003b04 to 0x00e2d4 (0xa7d1 bytes) 000321 1763402816.375122 trunc from 0x00e2d5 to 0x039da7 (0x2bad3 bytes) 000322 1763402816.392926 trunc from 0x018bd4 to 0x024709 (0xbb36 bytes) 000323 1763402816.397602 write 0x0363d9 thru 0x038b36 (0x275e bytes) 000326 1763402816.409000 mapread 0x015a0a thru 0x019aa6 (0x409d bytes) 000327 1763402816.417785 punch from 0x011626 to 0x012957 (0x1332 bytes) 000328 1763402816.430390 MIRROR_RESYNC 2 mirrors 000329 1763402817.107351 mapwrite 0x035ed1 thru 0x0363f6 (0x526 bytes) 000330 1763402817.219882 mapread 0x019c6b thru 0x0284f3 (0xe889 bytes) 000331 1763402817.240774 read 0x008a67 thru 0x015d21 (0xd2bb bytes) 000332 1763402817.254606 trunc from 0x02d2ef to 0x032400 (0x5112 bytes) 000333 1763402817.258497 mapread 0x001278 thru 0x001e8e (0xc17 bytes) 000334 1763402817.266681 mapread 0x038d18 thru 0x039da7 (0x1090 bytes) 000335 1763402817.267451 mapread 0x00bb6d thru 0x00e0d8 (0x256c bytes) 000336 1763402817.269126 trunc from 0x001426 to 0x00e4db (0xd0b6 bytes) 000337 1763402817.273378 trunc from 0x039da8 to 0x016eb2 (0x22ef5 bytes) 000338 1763402817.329778 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16eb3): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16eb3): Mirror 6: 0x8a8af365 (stale) CRC-32 checksum value for chunk [0, 0x16eb3): Mirror 1: 0xc426170a Mirror 7: 0xc426170a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16eb3: skipped 000339 1763402818.081522 mapread 0x00734d thru 0x00fb29 (0x87dd bytes) 000341 1763402818.106735 mapwrite 0x022825 thru 0x023bb5 (0x1391 bytes) 000342 1763402818.223299 fallocate 0x01776a thru 0x021989 (0xa220 bytes) 000343 1763402818.230976 trunc from 0x023bb6 to 0x00e39e (0x15817 bytes) 000344 1763402818.255931 mapread 0x0090c8 thru 0x00e39e (0x52d7 bytes) 000345 1763402818.264144 write 0x009776 thru 0x00e0d8 (0x4963 bytes) 000346 1763402818.265230 fallocate 0x020f4b thru 0x024f99 (0x404f bytes) 000347 1763402818.270922 mapwrite 0x0347be thru 0x03b68c (0x6ecf bytes) 000349 1763402818.359568 read 0x01178b thru 0x01819b (0x6a11 bytes) 000350 1763402818.379345 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b68d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b68d): Mirror 6: 0xab7e9d42 (stale) Mirror 7: 0xf88ab55d (stale) CRC-32 checksum value for chunk [0, 0x3b68d): Mirror 1: 0xf4c75521 Mirror 8: 0xf4c75521 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b68d: skipped 000351 1763402819.072710 fallocate 0x02d77b thru 0x031265 (0x3aeb bytes) 000352 1763402819.147120 punch from 0x013595 to 0x022bca (0xf636 bytes) 000353 1763402819.151338 write 0x030a71 thru 0x03d62c (0xcbbc bytes) 000354 1763402819.168698 fallocate 0x01715e thru 0x021852 (0xa6f5 bytes) 000355 1763402819.185769 trunc from 0x03d62d to 0x02cc4c (0x109e0 bytes) 000356 1763402819.236911 read 0x0210bf thru 0x022ac2 (0x1a04 bytes) 000357 1763402819.248967 read 0x018cc6 thru 0x01dac3 (0x4dfe bytes) 000358 1763402819.275133 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2cc4d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc4d): Mirror 6: 0x91c88ef6 (stale) Mirror 7: 0xd0028a91 (stale) Mirror 8: 0x14b1775 (stale) CRC-32 checksum value for chunk [0, 0x2cc4d): Mirror 1: 0x14b1775 Mirror 9: 0x14b1775 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc4d: skipped 000359 1763402820.000319 trunc from 0x007f99 to 0x0168ea (0xe952 bytes) 000360 1763402820.041692 fallocate 0x0154ae thru 0x02415a (0xecad bytes) 000361 1763402820.046493 trunc from 0x0149de to 0x01f6aa (0xaccd bytes) 000362 1763402820.058672 read 0x024e13 thru 0x02cc4c (0x7e3a bytes) 000364 1763402820.071095 punch from 0x00b753 to 0x017c62 (0xc510 bytes) 000365 1763402820.079531 write 0x029114 thru 0x036444 (0xd331 bytes) 000366 1763402820.087643 write 0x02fca1 thru 0x034ccc (0x502c bytes) 000367 1763402820.088798 trunc from 0x036445 to 0x03a9f7 (0x45b3 bytes) 000368 1763402820.126530 read 0x029879 thru 0x030193 (0x691b bytes) 000369 1763402820.127181 write 0x0343ff thru 0x034716 (0x318 bytes) 000370 1763402820.127706 punch from 0x01da02 to 0x0274e9 (0x9ae8 bytes) 000371 1763402820.134506 fallocate 0x035af2 thru 0x0366fe (0xc0d bytes) 000372 1763402820.141544 trunc from 0x005415 to 0x00939b (0x3f87 bytes) 000374 1763402820.152292 mapread 0x02fe73 thru 0x03a9f7 (0xab85 bytes) 000375 1763402820.170715 mapread 0x015305 thru 0x01af2c (0x5c28 bytes) 000376 1763402820.171802 mapread 0x002fea thru 0x0065dd (0x35f4 bytes) 000377 1763402820.172728 read 0x00fdc1 thru 0x013982 (0x3bc2 bytes) 000378 1763402820.173301 mapread 0x02b8a7 thru 0x0350cd (0x9827 bytes) 000379 1763402820.178223 MIRROR_RESYNC 5 mirrors 000380 1763402821.199101 mapwrite 0x00dc5a thru 0x016ff3 (0x939a bytes) 000381 1763402821.315237 read 0x03161a thru 0x03a9f7 (0x93de bytes) 000382 1763402821.335900 fallocate 0x0395e8 thru 0x03ffff (0x6a18 bytes) 000384 1763402821.341922 punch from 0x007695 to 0x00c201 (0x4b6d bytes) 000385 1763402821.348444 trunc from 0x03a9f8 to 0x0307f4 (0xa203 bytes) 000386 1763402821.406796 trunc from 0x028d08 to 0x0307f4 (0x7aed bytes) 000387 1763402821.417114 MIRROR_RESYNC 5 mirrors 000388 1763402822.497978 mapwrite 0x02cbe1 thru 0x030762 (0x3b82 bytes) 000389 1763402822.579487 fallocate 0x023a42 thru 0x026a2f (0x2fee bytes) 000390 1763402822.589359 trunc from 0x01844f to 0x01c17c (0x3d2e bytes) 000392 1763402822.597431 mapread 0x00d9b4 thru 0x0190e7 (0xb734 bytes) 000393 1763402822.633723 mapread 0x00ffa8 thru 0x01dfb1 (0xe00a bytes) 000395 1763402822.635613 read 0x02335e thru 0x02fb1f (0xc7c2 bytes) 000396 1763402822.644349 read 0x020995 thru 0x02aaa5 (0xa111 bytes) 000397 1763402822.653580 mapread 0x00e978 thru 0x01e68c (0xfd15 bytes) 000398 1763402822.654597 write 0x010cbd thru 0x011f25 (0x1269 bytes) 000399 1763402822.655469 trunc from 0x01954f to 0x0252b3 (0xbd65 bytes) 000400 1763402822.662263 read 0x0087fb thru 0x014579 (0xbd7f bytes) 000401 1763402822.662927 punch from 0x01366b to 0x01faae (0xc444 bytes) 000402 1763402822.668643 trunc from 0x02b182 to 0x0307f4 (0x5673 bytes) 000403 1763402822.679878 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x307f5): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307f5): Mirror 6: 0xb0b263f8 (stale) Mirror 7: 0xb0b263f8 (stale) Mirror 8: 0xb0b263f8 (stale) Mirror 9: 0xb0b263f8 (stale) CRC-32 checksum value for chunk [0, 0x307f5): Mirror 1: 0x6eded35 Mirror 10: 0x6eded35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307f5: skipped 000404 1763402823.495379 write 0x0355ea thru 0x03ffff (0xaa16 bytes) 000405 1763402823.534799 mapread 0x0345dc thru 0x03ffff (0xba24 bytes) 000406 1763402823.539781 trunc from 0x040000 to 0x0081dc (0x37e23 bytes) 000407 1763402823.573225 mapread 0x0040b7 thru 0x0081dc (0x4126 bytes) 000408 1763402823.595976 trunc from 0x004e2e to 0x0081dc (0x33af bytes) 000409 1763402823.603095 punch from 0x004d5d to 0x0081dc (0x3480 bytes) 000410 1763402823.607325 write 0x02b5fe thru 0x02deee (0x28f1 bytes) 000411 1763402823.608382 read 0x00be1c thru 0x00e5de (0x27c3 bytes) 000412 1763402823.614440 mapread 0x01a526 thru 0x0268fa (0xc3d5 bytes) 000413 1763402823.627943 punch from 0x01b2b1 to 0x023400 (0x8150 bytes) 000414 1763402823.631754 mapread 0x0280a9 thru 0x02deee (0x5e46 bytes) 000415 1763402823.632883 read 0x017ee0 thru 0x01c11b (0x423c bytes) 000416 1763402823.641011 mapread 0x02168f thru 0x02b1cd (0x9b3f bytes) 000417 1763402823.642164 fallocate 0x02bc92 thru 0x032d29 (0x7098 bytes) 000418 1763402823.648855 punch from 0x0196f1 to 0x01b21a (0x1b2a bytes) 000419 1763402823.653134 read 0x02d038 thru 0x02deee (0xeb7 bytes) 000420 1763402823.653735 mapwrite 0x01268e thru 0x01caf1 (0xa464 bytes) 000423 1763402823.695712 fallocate 0x036cf0 thru 0x036f5a (0x26b bytes) 000424 1763402823.704589 mapwrite 0x00ca29 thru 0x01991d (0xcef5 bytes) 000425 1763402823.743364 read 0x0188c6 thru 0x0213ab (0x8ae6 bytes) 000426 1763402823.743990 write 0x031f62 thru 0x03608a (0x4129 bytes) 000427 1763402823.744997 punch from 0x0330a6 to 0x03608a (0x2fe5 bytes) 000428 1763402823.760685 write 0x029270 thru 0x037dd7 (0xeb68 bytes) 000429 1763402823.762300 fallocate 0x031d93 thru 0x039e62 (0x80d0 bytes) 000430 1763402823.773122 MIRROR_RESYNC 6 mirrors 000431 1763402824.992178 MIRROR_EXTEND to 7 mirrors 000432 1763402825.818402 MIRROR_EXTEND to 8 mirrors 000433 1763402826.455562 mapread 0x028245 thru 0x02f1af (0x6f6b bytes) 000434 1763402826.476862 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) 000436 1763402826.972148 fallocate 0x017574 thru 0x0215f9 (0xa086 bytes) 000438 1763402827.052367 mapwrite 0x002271 thru 0x011667 (0xf3f7 bytes) 000439 1763402827.097996 mapread 0x01be75 thru 0x029383 (0xd50f bytes) 000440 1763402827.125721 mapwrite 0x03850d thru 0x03ffff (0x7af3 bytes) 000441 1763402827.187103 trunc from 0x040000 to 0x017604 (0x289fb bytes) 000442 1763402827.202332 read 0x008915 thru 0x015143 (0xc82f bytes) 000443 1763402827.205965 MIRROR_RESYNC 7 mirrors 000444 1763402827.901449 write 0x013170 thru 0x01f0df (0xbf70 bytes) 000445 1763402827.953453 mapread 0x00391f thru 0x00e969 (0xb04b bytes) 000446 1763402827.970535 read 0x0184d6 thru 0x01f0df (0x6c0a bytes) 000447 1763402827.971121 fallocate 0x01c284 thru 0x02ad93 (0xeb10 bytes) 000448 1763402827.981700 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ad94): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad94): Mirror 6: 0xe5599d30 (stale) Mirror 7: 0xe5599d30 (stale) Mirror 8: 0xe5599d30 (stale) Mirror 10: 0xe5599d30 (stale) Mirror 11: 0xe5599d30 (stale) Mirror 12: 0xe5599d30 (stale) CRC-32 checksum value for chunk [0, 0x2ad94): Mirror 1: 0x147c767 Mirror 13: 0x147c767 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad94: skipped 000449 1763402828.876185 read 0x0235ff thru 0x02ad93 (0x7795 bytes) 000450 1763402828.888446 trunc from 0x02ad94 to 0x00b14f (0x1fc44 bytes) 000451 1763402828.945641 trunc from 0x000145 to 0x0016d3 (0x158f bytes) 000452 1763402828.958355 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xb150): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb150): Mirror 6: 0xdb5cc19 (stale) Mirror 7: 0xdb5cc19 (stale) Mirror 8: 0xdb5cc19 (stale) Mirror 10: 0xdb5cc19 (stale) Mirror 11: 0xdb5cc19 (stale) Mirror 12: 0xdb5cc19 (stale) Mirror 13: 0xdb5cc19 (stale) CRC-32 checksum value for chunk [0, 0xb150): Mirror 1: 0x606f033f Mirror 14: 0x606f033f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb150: skipped 000453 1763402829.771438 trunc from 0x00b150 to 0x03c24e (0x310ff bytes) 000454 1763402829.842266 read 0x0377cc thru 0x03787a (0xaf bytes) 000455 1763402829.853319 punch from 0x02c68a to 0x03115e (0x4ad5 bytes) 000457 1763402829.872158 fallocate 0x0274b6 thru 0x0347e2 (0xd32d bytes) 000458 1763402829.882336 trunc from 0x00ea97 to 0x01bb96 (0xd100 bytes) 000459 1763402829.895616 read 0x01ab9d thru 0x026788 (0xbbec bytes) 000460 1763402829.920309 mapread 0x012626 thru 0x01f375 (0xcd50 bytes) 000461 1763402829.921277 trunc from 0x03c24f to 0x003fd5 (0x38279 bytes) 000462 1763402829.981419 punch from 0x0017c6 to 0x002109 (0x944 bytes) 000463 1763402829.993037 read 0x0035a3 thru 0x003fd5 (0xa33 bytes) 000464 1763402829.993606 punch from 0x000ad4 to 0x003fd5 (0x3502 bytes) 000465 1763402830.006651 read 0x003933 thru 0x003fd5 (0x6a3 bytes) 000466 1763402830.016087 fallocate 0x039482 thru 0x03ffff (0x6b7e bytes) 000467 1763402830.029706 mapwrite 0x02a79f thru 0x036f7e (0xc7e0 bytes) 000468 1763402830.106397 write 0x02edf3 thru 0x030a32 (0x1c40 bytes) 000469 1763402830.107479 trunc from 0x01e7cd to 0x0233c3 (0x4bf7 bytes) 000470 1763402830.112587 trunc from 0x036f7f to 0x00064a (0x36934 bytes) 000471 1763402830.132361 read 0x0002be thru 0x00064a (0x38d bytes) 000472 1763402830.133005 trunc from 0x0005eb to 0x00064a (0x60 bytes) 000473 1763402830.145412 read 0x0004a3 thru 0x00064a (0x1a8 bytes) 000474 1763402830.151487 trunc from 0x000409 to 0x00064a (0x242 bytes) 000475 1763402830.165704 mapwrite 0x035609 thru 0x03b4f6 (0x5eee bytes) 000476 1763402830.248457 trunc from 0x03b4f7 to 0x017290 (0x24266 bytes) 000477 1763402830.267897 read 0x00ab85 thru 0x00e8b6 (0x3d32 bytes) 000478 1763402830.277446 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x17291): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17291): Mirror 6: 0x932a3116 (stale) Mirror 7: 0x932a3116 (stale) Mirror 8: 0x932a3116 (stale) Mirror 10: 0x932a3116 (stale) Mirror 11: 0x932a3116 (stale) Mirror 12: 0x932a3116 (stale) Mirror 13: 0x3525d45c (stale) Mirror 14: 0xa7b66280 (stale) CRC-32 checksum value for chunk [0, 0x17291): Mirror 1: 0xc31344bb Mirror 15: 0xc31344bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17291: skipped 000480 1763402831.295348 trunc from 0x017291 to 0x02b51e (0x1428e bytes) 000481 1763402831.382059 trunc from 0x026c46 to 0x02b51e (0x48d9 bytes) 000482 1763402831.393055 read 0x009bc3 thru 0x013d68 (0xa1a6 bytes) 000484 1763402831.411993 MIRROR_RESYNC 10 mirrors 000485 1763402832.846093 write 0x026cc1 thru 0x02d59e (0x68de bytes) 000486 1763402832.879536 trunc from 0x02d59f to 0x019a44 (0x13b5a bytes) 000487 1763402832.896386 write 0x03ce36 thru 0x03ffff (0x31ca bytes) 000488 1763402832.907148 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb5e62d1f (stale) Mirror 7: 0xb5e62d1f (stale) Mirror 8: 0xb5e62d1f (stale) Mirror 10: 0xb5e62d1f (stale) Mirror 11: 0xb5e62d1f (stale) Mirror 13: 0xb5e62d1f (stale) Mirror 14: 0xb5e62d1f (stale) Mirror 15: 0xb5e62d1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb5e62d1f (stale) Mirror 7: 0xb5e62d1f (stale) Mirror 8: 0xb5e62d1f (stale) Mirror 10: 0xb5e62d1f (stale) Mirror 11: 0xb5e62d1f (stale) Mirror 13: 0xb5e62d1f (stale) Mirror 14: 0xb5e62d1f (stale) Mirror 15: 0xb5e62d1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000489 1763402833.341399 mapread 0x034a83 thru 0x0378c0 (0x2e3e bytes) 000490 1763402833.360371 fallocate 0x02e3dd thru 0x03a850 (0xc474 bytes) 000491 1763402833.364639 mapread 0x020e57 thru 0x02a08e (0x9238 bytes) 000492 1763402833.379350 mapread 0x0242b5 thru 0x0306a7 (0xc3f3 bytes) 000493 1763402833.380203 read 0x0128b9 thru 0x01ab4a (0x8292 bytes) 000494 1763402833.386732 punch from 0x010ad6 to 0x01910b (0x8636 bytes) 000495 1763402833.393205 read 0x0270c6 thru 0x02e475 (0x73b0 bytes) 000496 1763402833.393861 read 0x0334a2 thru 0x03ffff (0xcb5e bytes) 000499 1763402833.404975 read 0x0395cb thru 0x03ffff (0x6a35 bytes) 000500 1763402833.405532 mapwrite 0x025d69 thru 0x033639 (0xd8d1 bytes) 000501 1763402833.425542 mapwrite 0x007d4b thru 0x00a9dc (0x2c92 bytes) 000502 1763402833.444977 trunc from 0x02228d to 0x025c50 (0x39c4 bytes) 000503 1763402833.451906 write 0x01c245 thru 0x026ebf (0xac7b bytes) 000504 1763402833.456634 read 0x033775 thru 0x03819d (0x4a29 bytes) 000505 1763402833.457272 read 0x0184fe thru 0x022535 (0xa038 bytes) 000507 1763402833.459799 punch from 0x035042 to 0x03ffff (0xafbe bytes) 000508 1763402833.466885 mapread 0x03b7ae thru 0x03f5b8 (0x3e0b bytes) 000509 1763402833.480451 punch from 0x00d41c to 0x01c7cf (0xf3b4 bytes) 000510 1763402833.494211 mapwrite 0x0142c6 thru 0x0159b5 (0x16f0 bytes) 000511 1763402833.524342 punch from 0x03e1c9 to 0x03eec4 (0xcfc bytes) 000512 1763402833.529565 trunc from 0x040000 to 0x03190d (0xe6f2 bytes) 000513 1763402833.553315 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3190e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3190e): Mirror 6: 0x61beb518 (stale) Mirror 7: 0x61beb518 (stale) Mirror 8: 0x61beb518 (stale) Mirror 10: 0x61beb518 (stale) Mirror 11: 0x61beb518 (stale) Mirror 13: 0x61beb518 (stale) Mirror 14: 0x61beb518 (stale) Mirror 15: 0x61beb518 (stale) CRC-32 checksum value for chunk [0, 0x3190e): Mirror 1: 0xb00b5f46 Mirror 16: 0xb00b5f46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3190e: skipped 000515 1763402834.446664 MIRROR_RESYNC 10 mirrors 000516 1763402836.186184 punch from 0x019b75 to 0x01cddd (0x3269 bytes) 000517 1763402836.228094 mapread 0x0046be thru 0x01118d (0xcad0 bytes) 000518 1763402836.252371 fallocate 0x0133cd thru 0x019318 (0x5f4c bytes) 000519 1763402836.257393 mapwrite 0x00ce91 thru 0x018390 (0xb500 bytes) 000520 1763402836.279060 trunc from 0x03190e to 0x00648a (0x2b483 bytes) 000521 1763402836.319610 punch from 0x004e1e to 0x00648a (0x166d bytes) 000522 1763402836.325001 mapwrite 0x0284cc thru 0x02c888 (0x43bd bytes) 000523 1763402836.366073 trunc from 0x0187f1 to 0x020d9b (0x85ab bytes) 000524 1763402836.370300 trunc from 0x01ec3e to 0x02b284 (0xc647 bytes) 000525 1763402836.377201 write 0x020f6b thru 0x02f280 (0xe316 bytes) 000527 1763402836.388280 punch from 0x001033 to 0x008c96 (0x7c64 bytes) 000528 1763402836.395177 mapwrite 0x01ebac thru 0x02e184 (0xf5d9 bytes) 000530 1763402836.414448 mapread 0x00dee4 thru 0x017aaa (0x9bc7 bytes) 000531 1763402836.438153 trunc from 0x006990 to 0x016526 (0xfb97 bytes) 000532 1763402836.445662 read 0x02dd92 thru 0x02f280 (0x14ef bytes) 000533 1763402836.446305 trunc from 0x02f281 to 0x01beb1 (0x133cf bytes) 000534 1763402836.467507 MIRROR_RESYNC 10 mirrors 000535 1763402837.556227 trunc from 0x01beb2 to 0x01c0d3 (0x222 bytes) 000536 1763402837.585177 mapread 0x01143e thru 0x015635 (0x41f8 bytes) 000537 1763402837.610586 read 0x019f31 thru 0x01c0d3 (0x21a3 bytes) 000538 1763402837.616256 trunc from 0x01c0d4 to 0x023ac4 (0x79f1 bytes) 000539 1763402837.639290 trunc from 0x023ac5 to 0x039ecd (0x16409 bytes) 000540 1763402837.654231 punch from 0x0304c1 to 0x032694 (0x21d4 bytes) 000541 1763402837.660533 mapwrite 0x02d4ef thru 0x030736 (0x3248 bytes) 000542 1763402837.712907 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39ece): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ece): Mirror 6: 0x951ddb32 (stale) Mirror 7: 0x951ddb32 (stale) Mirror 8: 0x951ddb32 (stale) Mirror 10: 0x951ddb32 (stale) Mirror 11: 0x951ddb32 (stale) Mirror 13: 0x951ddb32 (stale) Mirror 14: 0x951ddb32 (stale) Mirror 15: 0x951ddb32 (stale) Mirror 16: 0x951ddb32 (stale) CRC-32 checksum value for chunk [0, 0x39ece): Mirror 1: 0xc87b6b3a Mirror 17: 0xc87b6b3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ece: skipped 000543 1763402838.571563 trunc from 0x00d981 to 0x00fc26 (0x22a6 bytes) 000544 1763402838.627221 punch from 0x01b1be to 0x021d11 (0x6b54 bytes) 000545 1763402838.638002 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39ece): Mirror 6: 0x951ddb32 (stale) Mirror 7: 0x951ddb32 (stale) Mirror 8: 0x951ddb32 (stale) Mirror 10: 0x951ddb32 (stale) Mirror 11: 0x951ddb32 (stale) Mirror 13: 0x951ddb32 (stale) Mirror 15: 0x951ddb32 (stale) Mirror 16: 0x951ddb32 (stale) Mirror 17: 0xc87b6b3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ece): Mirror 6: 0x951ddb32 (stale) Mirror 7: 0x951ddb32 (stale) Mirror 8: 0x951ddb32 (stale) Mirror 10: 0x951ddb32 (stale) Mirror 11: 0x951ddb32 (stale) Mirror 13: 0x951ddb32 (stale) Mirror 15: 0x951ddb32 (stale) Mirror 16: 0x951ddb32 (stale) Mirror 17: 0xc87b6b3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ece: skipped 000547 1763402839.077323 trunc from 0x036898 to 0x039a1e (0x3187 bytes) 000548 1763402839.083188 punch from 0x02f2dd to 0x03402c (0x4d50 bytes) 000549 1763402839.090116 trunc from 0x010035 to 0x017307 (0x72d3 bytes) 000550 1763402839.098210 read 0x0244e5 thru 0x02ae6a (0x6986 bytes) 000551 1763402839.109995 trunc from 0x00a02b to 0x00ae5a (0xe30 bytes) 000552 1763402839.113940 trunc from 0x039ece to 0x01e599 (0x1b934 bytes) 000554 1763402839.136696 MIRROR_RESYNC 10 mirrors 000555 1763402840.229313 trunc from 0x004e8b to 0x00612b (0x12a1 bytes) 000556 1763402840.268216 mapread 0x01355b thru 0x01e599 (0xb03f bytes) 000557 1763402840.300087 mapread 0x007328 thru 0x00afd3 (0x3cac bytes) 000559 1763402840.311017 fallocate 0x03d5c5 thru 0x03ffff (0x2a3b bytes) 000560 1763402840.317577 trunc from 0x040000 to 0x00d4c8 (0x32b37 bytes) 000561 1763402840.364220 read 0x00a443 thru 0x00d4c8 (0x3086 bytes) 000562 1763402840.364863 fallocate 0x005e5e thru 0x00f4d9 (0x967c bytes) 000563 1763402840.373508 trunc from 0x00f4da to 0x00feb1 (0x9d8 bytes) 000565 1763402840.394487 trunc from 0x00f420 to 0x00feb1 (0xa92 bytes) 000566 1763402840.398851 write 0x0315b2 thru 0x0358ae (0x42fd bytes) 000567 1763402840.400183 trunc from 0x0358af to 0x008f01 (0x2c9ad bytes) 000568 1763402840.420382 write 0x01d38f thru 0x020a90 (0x3702 bytes) 000569 1763402840.422793 write 0x003496 thru 0x008228 (0x4d93 bytes) 000570 1763402840.433880 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x20a91): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20a91): Mirror 6: 0x3302fb79 (stale) Mirror 7: 0x3302fb79 (stale) Mirror 8: 0x3302fb79 (stale) Mirror 10: 0x3302fb79 (stale) Mirror 11: 0x3302fb79 (stale) Mirror 13: 0x3302fb79 (stale) Mirror 15: 0x3302fb79 (stale) Mirror 16: 0x3302fb79 (stale) Mirror 17: 0x3302fb79 (stale) CRC-32 checksum value for chunk [0, 0x20a91): Mirror 1: 0x32a39f1b Mirror 18: 0x32a39f1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20a91: skipped 000571 1763402841.266207 mapread 0x020974 thru 0x020a90 (0x11d bytes) 000572 1763402841.273378 trunc from 0x00ac2e to 0x00c708 (0x1adb bytes) 000573 1763402841.314589 punch from 0x011448 to 0x01700b (0x5bc4 bytes) 000574 1763402841.328997 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x20a91): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20a91): Mirror 6: 0x3302fb79 (stale) Mirror 7: 0x3302fb79 (stale) Mirror 8: 0x3302fb79 (stale) Mirror 10: 0x3302fb79 (stale) Mirror 11: 0x3302fb79 (stale) Mirror 13: 0x3302fb79 (stale) Mirror 15: 0x3302fb79 (stale) Mirror 16: 0x3302fb79 (stale) Mirror 17: 0x3302fb79 (stale) Mirror 18: 0x32a39f1b (stale) CRC-32 checksum value for chunk [0, 0x20a91): Mirror 1: 0x32a39f1b Mirror 19: 0x32a39f1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20a91: skipped 000575 1763402842.212049 trunc from 0x020a91 to 0x01aef3 (0x5b9d bytes) 000576 1763402842.269354 trunc from 0x01aef4 to 0x0174b8 (0x3a3b bytes) 000578 1763402842.290102 trunc from 0x013aec to 0x0174b8 (0x39cd bytes) 000579 1763402842.300482 read 0x010f7b thru 0x0174b8 (0x653e bytes) 000580 1763402842.310731 fallocate 0x01eccd thru 0x02d774 (0xeaa8 bytes) 000581 1763402842.315973 write 0x01e566 thru 0x029323 (0xadbe bytes) 000582 1763402842.317755 fallocate 0x0135be thru 0x01e06b (0xaaae bytes) 000583 1763402842.324302 write 0x005ced thru 0x00b808 (0x5b1c bytes) 000584 1763402842.333066 mapread 0x01feee thru 0x029323 (0x9436 bytes) 000586 1763402842.334394 mapread 0x0168ba thru 0x01b3d5 (0x4b1c bytes) 000587 1763402842.340951 mapwrite 0x014838 thru 0x018a79 (0x4242 bytes) 000588 1763402842.372786 mapread 0x010bd3 thru 0x01cf65 (0xc393 bytes) 000589 1763402842.373765 trunc from 0x00c5d1 to 0x014375 (0x7da5 bytes) 000590 1763402842.383496 write 0x01211c thru 0x02106e (0xef53 bytes) 000591 1763402842.403254 mapread 0x01202c thru 0x01ed68 (0xcd3d bytes) 000592 1763402842.404696 read 0x00372a thru 0x010cec (0xd5c3 bytes) 000593 1763402842.418996 trunc from 0x029324 to 0x012769 (0x16bba bytes) 000594 1763402842.437253 fallocate 0x001098 thru 0x00986e (0x87d7 bytes) 000595 1763402842.443177 fallocate 0x0320e7 thru 0x036e82 (0x4d9c bytes) 000596 1763402842.447703 trunc from 0x036e83 to 0x03b862 (0x49e0 bytes) 000598 1763402842.461231 trunc from 0x03b863 to 0x0111f3 (0x2a66f bytes) 000599 1763402842.492085 write 0x02856b thru 0x02a9c7 (0x245d bytes) 000600 1763402842.493306 mapread 0x0259c5 thru 0x02a9c7 (0x5003 bytes) 000601 1763402842.525096 punch from 0x00e5bc to 0x01e2a0 (0xfce5 bytes) 000602 1763402842.537032 trunc from 0x02a9c8 to 0x03cf2d (0x12566 bytes) 000603 1763402842.559383 read 0x007b45 thru 0x015c2c (0xe0e8 bytes) 000604 1763402842.574719 trunc from 0x0072b9 to 0x01084c (0x9594 bytes) 000605 1763402842.582885 mapwrite 0x00e3fe thru 0x01a40e (0xc011 bytes) 000606 1763402842.622659 write 0x01d067 thru 0x029b76 (0xcb10 bytes) 000607 1763402842.624342 write 0x00c3e2 thru 0x012567 (0x6186 bytes) 000608 1763402842.630647 read 0x02b35f thru 0x02c268 (0xf0a bytes) 000610 1763402842.636388 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf2e): Mirror 6: 0xe51f093 (stale) Mirror 7: 0xe51f093 (stale) Mirror 8: 0xe51f093 (stale) Mirror 10: 0xe51f093 (stale) Mirror 11: 0xe51f093 (stale) Mirror 13: 0xe51f093 (stale) Mirror 15: 0xe51f093 (stale) Mirror 16: 0xe51f093 (stale) Mirror 17: 0xe51f093 (stale) Mirror 19: 0x33092d2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf2e): Mirror 6: 0xe51f093 (stale) Mirror 7: 0xe51f093 (stale) Mirror 8: 0xe51f093 (stale) Mirror 10: 0xe51f093 (stale) Mirror 11: 0xe51f093 (stale) Mirror 13: 0xe51f093 (stale) Mirror 15: 0xe51f093 (stale) Mirror 16: 0xe51f093 (stale) Mirror 17: 0xe51f093 (stale) Mirror 19: 0x33092d2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf2e: skipped 000611 1763402843.205501 read 0x003f57 thru 0x008bde (0x4c88 bytes) 000613 1763402843.213949 mapread 0x007897 thru 0x009201 (0x196b bytes) 000614 1763402843.222439 read 0x03381e thru 0x03cf2d (0x9710 bytes) 000615 1763402843.235751 punch from 0x0049a8 to 0x01441f (0xfa78 bytes) 000617 1763402843.241598 trunc from 0x02baf5 to 0x02bbb8 (0xc4 bytes) 000618 1763402843.246666 mapwrite 0x02c0a5 thru 0x03b858 (0xf7b4 bytes) 000619 1763402843.271354 read 0x021474 thru 0x029823 (0x83b0 bytes) 000620 1763402843.272039 write 0x010202 thru 0x011d03 (0x1b02 bytes) 000621 1763402843.286444 mapwrite 0x036759 thru 0x03fc5b (0x9503 bytes) 000623 1763402843.330597 fallocate 0x022ae3 thru 0x02ad02 (0x8220 bytes) 000624 1763402843.336228 punch from 0x0024aa to 0x00a9c6 (0x851d bytes) 000625 1763402843.341241 trunc from 0x01f6d3 to 0x02634a (0x6c78 bytes) 000626 1763402843.348389 MIRROR_RESYNC 11 mirrors 000627 1763402846.360154 trunc from 0x012976 to 0x019bfb (0x7286 bytes) 000628 1763402846.406399 mapwrite 0x00e364 thru 0x00e6a9 (0x346 bytes) 000629 1763402846.422797 trunc from 0x03fc5c to 0x025948 (0x1a313 bytes) 000630 1763402846.455666 fallocate 0x0228d3 thru 0x02f347 (0xca75 bytes) 000631 1763402846.464285 mapwrite 0x0050fd thru 0x00fa30 (0xa934 bytes) 000632 1763402846.494400 write 0x011b14 thru 0x01a7c1 (0x8cae bytes) 000633 1763402846.506453 trunc from 0x025949 to 0x039d35 (0x143ed bytes) 000634 1763402846.520401 trunc from 0x03004b to 0x035725 (0x56db bytes) 000635 1763402846.524486 fallocate 0x0157d8 thru 0x01ffc2 (0xa7eb bytes) 000636 1763402846.529346 trunc from 0x039d36 to 0x0156ac (0x24689 bytes) 000637 1763402846.547178 write 0x0287df thru 0x02b110 (0x2932 bytes) 000638 1763402846.549404 read 0x009a52 thru 0x016098 (0xc647 bytes) 000639 1763402846.556514 trunc from 0x02b111 to 0x01ed4d (0xc3c3 bytes) 000640 1763402846.571362 trunc from 0x01ed4e to 0x0305a4 (0x11857 bytes) 000641 1763402846.588662 write 0x024c43 thru 0x0330d0 (0xe48e bytes) 000642 1763402846.596531 mapread 0x0179a4 thru 0x027789 (0xfde6 bytes) 000643 1763402846.603713 mapread 0x02624a thru 0x0294ef (0x32a6 bytes) 000644 1763402846.604478 punch from 0x017b05 to 0x01a962 (0x2e5e bytes) 000645 1763402846.613634 trunc from 0x02c0bb to 0x0330d0 (0x7016 bytes) 000647 1763402846.637627 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x330d1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330d1): Mirror 6: 0x53feb4bd (stale) Mirror 7: 0x53feb4bd (stale) Mirror 8: 0x53feb4bd (stale) Mirror 10: 0x53feb4bd (stale) Mirror 11: 0x53feb4bd (stale) Mirror 13: 0x53feb4bd (stale) Mirror 15: 0x53feb4bd (stale) Mirror 16: 0x53feb4bd (stale) Mirror 17: 0x53feb4bd (stale) Mirror 19: 0x53feb4bd (stale) CRC-32 checksum value for chunk [0, 0x330d1): Mirror 1: 0x765ad319 Mirror 20: 0x765ad319 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330d1: skipped 000648 1763402847.524270 punch from 0x00e546 to 0x00fa04 (0x14bf bytes) 000649 1763402847.566495 trunc from 0x01c12e to 0x02505a (0x8f2d bytes) 000650 1763402847.576978 trunc from 0x0330d1 to 0x000cf3 (0x323dd bytes) 000652 1763402847.641289 fallocate 0x01ae70 thru 0x01d9e1 (0x2b72 bytes) 000653 1763402847.649247 mapwrite 0x003899 thru 0x00f8c0 (0xc028 bytes) 000654 1763402847.699719 fallocate 0x030009 thru 0x035650 (0x5648 bytes) 000655 1763402847.708402 punch from 0x01b183 to 0x01d9e1 (0x285f bytes) 000656 1763402847.718943 trunc from 0x0121a3 to 0x015e6d (0x3ccb bytes) 000657 1763402847.728695 trunc from 0x01d9e2 to 0x010034 (0xd9ad bytes) 000658 1763402847.752295 mapread 0x005d49 thru 0x00cb4e (0x6e06 bytes) 000659 1763402847.753159 mapread 0x00239d thru 0x010034 (0xdc98 bytes) 000660 1763402847.753980 trunc from 0x00704e to 0x00bd46 (0x4cf9 bytes) 000661 1763402847.762440 trunc from 0x010035 to 0x00393a (0xc6fa bytes) 000662 1763402847.782286 mapread 0x00156d thru 0x00393a (0x23ce bytes) 000663 1763402847.783228 mapread 0x0029e5 thru 0x00393a (0xf56 bytes) 000664 1763402847.783989 trunc from 0x0017e0 to 0x00393a (0x215b bytes) 000665 1763402847.797484 MIRROR_RESYNC 12 mirrors 000666 1763402849.233164 mapread 0x0029cc thru 0x00393a (0xf6f bytes) 000667 1763402849.253699 read 0x0026d3 thru 0x00393a (0x1268 bytes) 000668 1763402849.254049 fallocate 0x02d3bf thru 0x03ae88 (0xdaca bytes) 000669 1763402849.289561 write 0x03252d thru 0x037966 (0x543a bytes) 000670 1763402849.297888 MIRROR_RESYNC 12 mirrors 000671 1763402851.203736 punch from 0x0255d6 to 0x033f0c (0xe937 bytes) 000672 1763402851.244129 mapwrite 0x034082 thru 0x03e20c (0xa18b bytes) 000673 1763402851.308704 write 0x000765 thru 0x00837d (0x7c19 bytes) 000674 1763402851.318218 trunc from 0x000543 to 0x00dca8 (0xd766 bytes) 000675 1763402851.332095 trunc from 0x018ba3 to 0x01ac20 (0x207e bytes) 000676 1763402851.336658 trunc from 0x03e20d to 0x03ea56 (0x84a bytes) 000677 1763402851.353269 mapwrite 0x00b351 thru 0x01773d (0xc3ed bytes) 000678 1763402851.385877 trunc from 0x03ea57 to 0x02a341 (0x14715 bytes) 000679 1763402851.398957 mapread 0x020af7 thru 0x0293c6 (0x88d0 bytes) 000680 1763402851.417790 read 0x012a01 thru 0x014c5a (0x225a bytes) 000681 1763402851.418380 punch from 0x017ad6 to 0x01fc52 (0x817d bytes) 000682 1763402851.428183 fallocate 0x006694 thru 0x00b76d (0x50da bytes) 000683 1763402851.436585 mapwrite 0x019816 thru 0x01b797 (0x1f82 bytes) 000684 1763402851.457455 trunc from 0x00d60c to 0x010756 (0x314b bytes) 000685 1763402851.462261 mapwrite 0x0037f4 thru 0x00f690 (0xbe9d bytes) 000686 1763402851.485706 write 0x024f28 thru 0x030bf6 (0xbccf bytes) 000688 1763402851.487315 trunc from 0x0173fd to 0x026d04 (0xf908 bytes) 000690 1763402851.507137 mapread 0x02e07b thru 0x030bf6 (0x2b7c bytes) 000691 1763402851.508029 trunc from 0x01660c to 0x025643 (0xf038 bytes) 000692 1763402851.530059 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30bf7): Mirror 6: 0x415f8720 (stale) Mirror 7: 0x415f8720 (stale) Mirror 10: 0x415f8720 (stale) Mirror 11: 0x415f8720 (stale) Mirror 13: 0x415f8720 (stale) Mirror 15: 0x415f8720 (stale) Mirror 16: 0x415f8720 (stale) Mirror 17: 0x415f8720 (stale) Mirror 19: 0x415f8720 (stale) Mirror 20: 0x415f8720 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30bf7): Mirror 6: 0x415f8720 (stale) Mirror 7: 0x415f8720 (stale) Mirror 10: 0x415f8720 (stale) Mirror 11: 0x415f8720 (stale) Mirror 13: 0x415f8720 (stale) Mirror 15: 0x415f8720 (stale) Mirror 16: 0x415f8720 (stale) Mirror 17: 0x415f8720 (stale) Mirror 19: 0x415f8720 (stale) Mirror 20: 0x415f8720 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30bf7: skipped 000693 1763402852.108906 punch from 0x020f51 to 0x0239a9 (0x2a59 bytes) 000694 1763402852.114681 trunc from 0x02c51b to 0x030bf6 (0x46dc bytes) 000695 1763402852.120386 trunc from 0x030bf7 to 0x039f9f (0x93a9 bytes) 000696 1763402852.154133 read 0x038326 thru 0x039f9f (0x1c7a bytes) 000697 1763402852.160551 fallocate 0x005de7 thru 0x00f6a9 (0x98c3 bytes) 000698 1763402852.167874 mapread 0x029a19 thru 0x036eea (0xd4d2 bytes) 000699 1763402852.195870 mapread 0x025eac thru 0x027f53 (0x20a8 bytes) 000700 1763402852.196638 fallocate 0x011132 thru 0x0166ec (0x55bb bytes) 000701 1763402852.203239 punch from 0x039cb9 to 0x039f9f (0x2e7 bytes) 000702 1763402852.208072 mapwrite 0x008f70 thru 0x011cd2 (0x8d63 bytes) 000703 1763402852.247903 mapread 0x00372e thru 0x00c017 (0x88ea bytes) 000704 1763402852.248866 punch from 0x0048d1 to 0x01118d (0xc8bd bytes) 000705 1763402852.256770 trunc from 0x039fa0 to 0x00bfb0 (0x2dfef bytes) 000706 1763402852.276394 punch from 0x005506 to 0x00bfb0 (0x6aab bytes) 000707 1763402852.280997 fallocate 0x039519 thru 0x03ffff (0x6ae7 bytes) 000708 1763402852.285605 mapread 0x005870 thru 0x0139e8 (0xe179 bytes) 000709 1763402852.302140 trunc from 0x040000 to 0x01643b (0x29bc4 bytes) 000710 1763402852.316552 mapwrite 0x0129e2 thru 0x0149af (0x1fce bytes) 000711 1763402852.332722 trunc from 0x01643c to 0x0070b9 (0xf382 bytes) 000712 1763402852.344788 fallocate 0x03f690 thru 0x03ffff (0x970 bytes) 000713 1763402852.352549 write 0x003049 thru 0x004f11 (0x1ec9 bytes) 000714 1763402852.353890 write 0x006ff7 thru 0x01181b (0xa825 bytes) 000715 1763402852.355396 punch from 0x00f2b8 to 0x010d27 (0x1a70 bytes) 000716 1763402852.368165 trunc from 0x0091ff to 0x01181b (0x861d bytes) 000717 1763402852.372937 write 0x00836b thru 0x0083aa (0x40 bytes) 000718 1763402852.374058 mapwrite 0x028eb6 thru 0x0348c6 (0xba11 bytes) 000719 1763402852.427991 write 0x02cf46 thru 0x02e827 (0x18e2 bytes) 000720 1763402852.428945 trunc from 0x0348c7 to 0x02063c (0x1428a bytes) 000721 1763402852.456199 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2063d): Mirror 7: 0xe2d9c97e (stale) Mirror 10: 0xe2d9c97e (stale) Mirror 11: 0xe2d9c97e (stale) Mirror 13: 0xe2d9c97e (stale) Mirror 15: 0xe2d9c97e (stale) Mirror 16: 0xe2d9c97e (stale) Mirror 17: 0xe2d9c97e (stale) Mirror 19: 0xe2d9c97e (stale) Mirror 20: 0xe2d9c97e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2063d): Mirror 7: 0xe2d9c97e (stale) Mirror 10: 0xe2d9c97e (stale) Mirror 11: 0xe2d9c97e (stale) Mirror 13: 0xe2d9c97e (stale) Mirror 15: 0xe2d9c97e (stale) Mirror 16: 0xe2d9c97e (stale) Mirror 17: 0xe2d9c97e (stale) Mirror 19: 0xe2d9c97e (stale) Mirror 20: 0xe2d9c97e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2063d: skipped 000722 1763402852.949512 fallocate 0x0115b2 thru 0x01e84f (0xd29e bytes) 000723 1763402852.954296 trunc from 0x00ce1b to 0x012a9a (0x5c80 bytes) 000724 1763402852.961439 write 0x03a129 thru 0x03ffff (0x5ed7 bytes) 000725 1763402852.967784 trunc from 0x040000 to 0x038ab0 (0x754f bytes) 000726 1763402852.988651 mapwrite 0x030844 thru 0x03329f (0x2a5c bytes) 000727 1763402853.033059 trunc from 0x02650d to 0x02d71a (0x720e bytes) 000728 1763402853.040510 trunc from 0x038ab1 to 0x037492 (0x161e bytes) 000729 1763402853.057984 fallocate 0x033c9f thru 0x03ffff (0xc361 bytes) 000730 1763402853.064534 mapwrite 0x0045f2 thru 0x013d40 (0xf74f bytes) 000731 1763402853.106514 mapwrite 0x0393c4 thru 0x03c056 (0x2c93 bytes) 000732 1763402853.150023 write 0x027d2d thru 0x02eee3 (0x71b7 bytes) 000733 1763402853.162887 write 0x00ab39 thru 0x010604 (0x5acc bytes) 000734 1763402853.164606 write 0x0124bb thru 0x0184d2 (0x6018 bytes) 000735 1763402853.165576 mapread 0x023afb thru 0x02cc8d (0x9193 bytes) 000736 1763402853.166564 punch from 0x02a2b6 to 0x02cdd7 (0x2b22 bytes) 000737 1763402853.178933 fallocate 0x0092b7 thru 0x00aa56 (0x17a0 bytes) 000738 1763402853.183479 write 0x023235 thru 0x024dfb (0x1bc7 bytes) 000739 1763402853.184338 mapread 0x02c2a4 thru 0x02f555 (0x32b2 bytes) 000740 1763402853.189780 trunc from 0x00952f to 0x0114e4 (0x7fb6 bytes) 000741 1763402853.198602 punch from 0x02dc86 to 0x030626 (0x29a1 bytes) 000742 1763402853.205370 trunc from 0x03c057 to 0x00eef4 (0x2d162 bytes) 000743 1763402853.224583 fallocate 0x0366cb thru 0x03ee02 (0x8738 bytes) 000744 1763402853.230092 mapread 0x00954b thru 0x00eef4 (0x59aa bytes) 000745 1763402853.240052 punch from 0x009057 to 0x00eef4 (0x5e9e bytes) 000746 1763402853.244364 trunc from 0x00eef5 to 0x00053c (0xe9b8 bytes) 000747 1763402853.259897 write 0x039424 thru 0x03ffff (0x6bdc bytes) 000748 1763402853.261670 trunc from 0x01113c to 0x0204fe (0xf3c3 bytes) 000749 1763402853.267106 write 0x004ecf thru 0x00b2e4 (0x6416 bytes) 000750 1763402853.276649 read 0x028c84 thru 0x03509d (0xc41a bytes) 000751 1763402853.283845 write 0x0241e0 thru 0x02d118 (0x8f39 bytes) 000752 1763402853.290090 punch from 0x02d7cb to 0x02e67e (0xeb4 bytes) 000753 1763402853.306051 write 0x028502 thru 0x0286f0 (0x1ef bytes) 000754 1763402853.307719 write 0x012c3f thru 0x019f5f (0x7321 bytes) 000755 1763402853.320038 mapwrite 0x00d269 thru 0x0112bb (0x4053 bytes) 000757 1763402853.345130 fallocate 0x02eb38 thru 0x03b7f9 (0xccc2 bytes) 000758 1763402853.353627 fallocate 0x0274ce thru 0x035773 (0xe2a6 bytes) 000759 1763402853.365391 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3c0a4c39 (stale) Mirror 10: 0x3c0a4c39 (stale) Mirror 11: 0x3c0a4c39 (stale) Mirror 13: 0x3c0a4c39 (stale) Mirror 15: 0x3c0a4c39 (stale) Mirror 17: 0x3c0a4c39 (stale) Mirror 19: 0x3c0a4c39 (stale) Mirror 20: 0x3c0a4c39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x3c0a4c39 (stale) Mirror 10: 0x3c0a4c39 (stale) Mirror 11: 0x3c0a4c39 (stale) Mirror 13: 0x3c0a4c39 (stale) Mirror 15: 0x3c0a4c39 (stale) Mirror 17: 0x3c0a4c39 (stale) Mirror 19: 0x3c0a4c39 (stale) Mirror 20: 0x3c0a4c39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1763402853.831743 punch from 0x007d5a to 0x00e176 (0x641d bytes) 000761 1763402853.835152 mapwrite 0x00f8ab thru 0x01265b (0x2db1 bytes) 000762 1763402853.871556 MIRROR_RESYNC 9 mirrors 000763 1763402856.143643 mapread 0x03eb3a thru 0x03ffff (0x14c6 bytes) 000764 1763402856.153760 read 0x0104b4 thru 0x012ee4 (0x2a31 bytes) 000765 1763402856.159446 read 0x00994f thru 0x00f5ae (0x5c60 bytes) 000766 1763402856.172313 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) 000767 1763402856.507612 punch from 0x03aa76 to 0x03ffff (0x558a bytes) 000769 1763402856.544634 mapread 0x0203aa thru 0x025985 (0x55dc bytes) 000770 1763402856.562168 trunc from 0x00ffaa to 0x01d697 (0xd6ee bytes) 000771 1763402856.569748 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x2d4f0691 (stale) Mirror 10: 0x2d4f0691 (stale) Mirror 11: 0x2d4f0691 (stale) Mirror 15: 0x2d4f0691 (stale) Mirror 17: 0x2d4f0691 (stale) Mirror 19: 0x2d4f0691 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x2d4f0691 (stale) Mirror 10: 0x2d4f0691 (stale) Mirror 11: 0x2d4f0691 (stale) Mirror 15: 0x2d4f0691 (stale) Mirror 17: 0x2d4f0691 (stale) Mirror 19: 0x2d4f0691 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000772 1763402856.969857 trunc from 0x040000 to 0x0188bf (0x27740 bytes) 000773 1763402856.996308 trunc from 0x00745b to 0x00a13f (0x2ce5 bytes) 000774 1763402857.004617 trunc from 0x00aa8d to 0x0117b7 (0x6d2b bytes) 000775 1763402857.008835 read 0x0092ac thru 0x012ac5 (0x981a bytes) 000777 1763402857.022507 mapread 0x001083 thru 0x009941 (0x88bf bytes) 000778 1763402857.040008 read 0x017e2f thru 0x0188bf (0xa91 bytes) 000779 1763402857.046332 read 0x00b827 thru 0x01137e (0x5b58 bytes) 000780 1763402857.046951 trunc from 0x0188c0 to 0x010acb (0x7df4 bytes) 000782 1763402857.060458 write 0x01e2fd thru 0x023b08 (0x580c bytes) 000783 1763402857.061660 trunc from 0x023b09 to 0x0010f8 (0x22a10 bytes) 000784 1763402857.077377 mapwrite 0x0136a0 thru 0x013dab (0x70c bytes) 000785 1763402857.138144 trunc from 0x013dac to 0x03bcfa (0x27f4f bytes) 000786 1763402857.164332 write 0x014ce3 thru 0x01d96d (0x8c8b bytes) 000788 1763402857.174464 read 0x037f6d thru 0x03bcfa (0x3d8e bytes) 000789 1763402857.176061 mapwrite 0x0045ac thru 0x00de80 (0x98d5 bytes) 000790 1763402857.194187 fallocate 0x01d2af thru 0x026a39 (0x978b bytes) 000791 1763402857.198848 mapwrite 0x020378 thru 0x02842d (0x80b6 bytes) 000793 1763402857.219050 MIRROR_RESYNC 7 mirrors 000794 1763402858.813445 mapwrite 0x037327 thru 0x03ffff (0x8cd9 bytes) 000795 1763402858.881340 write 0x0288ab thru 0x036ea9 (0xe5ff bytes) 000796 1763402858.893405 punch from 0x03083e to 0x034af5 (0x42b8 bytes) 000797 1763402858.913637 write 0x010cec thru 0x0201e6 (0xf4fb bytes) 000798 1763402858.930670 write 0x02f85d thru 0x034a1e (0x51c2 bytes) 000799 1763402858.936156 fallocate 0x027a7c thru 0x02d471 (0x59f6 bytes) 000800 1763402858.944766 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x529fbeed (stale) Mirror 10: 0x529fbeed (stale) Mirror 11: 0x529fbeed (stale) Mirror 15: 0x529fbeed (stale) Mirror 17: 0x529fbeed (stale) Mirror 19: 0x529fbeed (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaf9c37a3 Mirror 20: 0xaf9c37a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000801 1763402859.681788 write 0x023f31 thru 0x026877 (0x2947 bytes) 000803 1763402859.725391 read 0x03c72b thru 0x03ffff (0x38d5 bytes) 000804 1763402859.730736 read 0x003fae thru 0x00b085 (0x70d8 bytes) 000805 1763402859.745597 mapread 0x006583 thru 0x0161f0 (0xfc6e bytes) 000807 1763402859.747106 trunc from 0x03c7f5 to 0x03d0af (0x8bb bytes) 000808 1763402859.759744 MIRROR_RESYNC 8 mirrors 000809 1763402861.007019 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) 000810 1763402861.300274 trunc from 0x040000 to 0x03fcdc (0x323 bytes) 000811 1763402861.350789 punch from 0x011f93 to 0x01a889 (0x88f7 bytes) 000812 1763402861.356083 fallocate 0x0238b2 thru 0x029c1d (0x636c bytes) 000813 1763402861.361310 write 0x02f750 thru 0x03594d (0x61fe bytes) 000814 1763402861.378184 MIRROR_RESYNC 7 mirrors 000815 1763402862.863173 trunc from 0x03fcdd to 0x012849 (0x2d493 bytes) 000817 1763402862.893004 read 0x0040cf thru 0x00dc42 (0x9b74 bytes) 000818 1763402862.899785 fallocate 0x01911f thru 0x025ef5 (0xcdd7 bytes) 000819 1763402862.906602 fallocate 0x03539a thru 0x03ef93 (0x9bfa bytes) 000820 1763402862.911130 mapwrite 0x009bb0 thru 0x00bfe4 (0x2435 bytes) 000821 1763402862.927110 trunc from 0x010702 to 0x012849 (0x2148 bytes) 000822 1763402862.932190 trunc from 0x01284a to 0x0239d4 (0x1118b bytes) 000823 1763402862.952315 fallocate 0x02f045 thru 0x02f345 (0x301 bytes) 000824 1763402862.958761 trunc from 0x012192 to 0x01b59d (0x940c bytes) 000825 1763402862.963849 write 0x005abe thru 0x00f693 (0x9bd6 bytes) 000826 1763402862.975147 read 0x0178c0 thru 0x0208d4 (0x9015 bytes) 000828 1763402862.988113 trunc from 0x00530a to 0x00818b (0x2e82 bytes) 000829 1763402863.005039 trunc from 0x02f346 to 0x03bc5b (0xc916 bytes) 000830 1763402863.017485 trunc from 0x03bc5c to 0x0016c1 (0x3a59a bytes) 000831 1763402863.033358 mapwrite 0x025f06 thru 0x030030 (0xa12b bytes) 000832 1763402863.151264 trunc from 0x030031 to 0x029654 (0x69dc bytes) 000833 1763402863.166585 trunc from 0x029655 to 0x03dadc (0x14488 bytes) 000834 1763402863.190038 trunc from 0x000e05 to 0x00986b (0x8a67 bytes) 000835 1763402863.196337 punch from 0x01155f to 0x01971c (0x81be bytes) 000836 1763402863.206619 read 0x01d9e3 thru 0x026de4 (0x9402 bytes) 000837 1763402863.223058 read 0x023cc9 thru 0x02922a (0x5562 bytes) 000838 1763402863.224252 fallocate 0x01975f thru 0x0215ac (0x7e4e bytes) 000839 1763402863.229114 write 0x013b35 thru 0x015707 (0x1bd3 bytes) 000840 1763402863.230021 mapwrite 0x020741 thru 0x02dfa3 (0xd863 bytes) 000841 1763402863.251577 mapwrite 0x02dcb5 thru 0x0392f4 (0xb640 bytes) 000842 1763402863.278335 MIRROR_RESYNC 7 mirrors 000843 1763402864.660560 write 0x034654 thru 0x038b66 (0x4513 bytes) 000844 1763402864.706177 mapread 0x026c8f thru 0x035557 (0xe8c9 bytes) 000845 1763402864.724973 fallocate 0x01f1e6 thru 0x022af9 (0x3914 bytes) 000846 1763402864.728883 trunc from 0x03a3b4 to 0x03dadc (0x3729 bytes) 000847 1763402864.735058 read 0x02f762 thru 0x038d32 (0x95d1 bytes) 000848 1763402864.735662 punch from 0x01a03a to 0x023ec8 (0x9e8f bytes) 000849 1763402864.741362 fallocate 0x003251 thru 0x008dc2 (0x5b72 bytes) 000850 1763402864.756027 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 7: 0xa2875308 (stale) Mirror 10: 0xa2875308 (stale) Mirror 15: 0xa2875308 (stale) Mirror 17: 0xa2875308 (stale) Mirror 19: 0xa2875308 (stale) Mirror 20: 0xa2875308 (stale) CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 1: 0xf7608c8f Mirror 21: 0xf7608c8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dadd: skipped 000851 1763402865.600444 punch from 0x004dda to 0x010e6f (0xc096 bytes) 000852 1763402865.651034 MIRROR_RESYNC 8 mirrors 000853 1763402867.280729 mapwrite 0x002808 thru 0x00c20a (0x9a03 bytes) 000854 1763402867.372446 read 0x0004fe thru 0x00d47d (0xcf80 bytes) 000855 1763402867.385496 trunc from 0x011f15 to 0x01d3df (0xb4cb bytes) 000858 1763402867.403146 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: 0xf7608c8f (stale) Mirror 15: 0xf7608c8f (stale) Mirror 17: 0xf7608c8f (stale) Mirror 19: 0xf7608c8f (stale) Mirror 20: 0xf7608c8f (stale) Mirror 21: 0xf7608c8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: 0xf7608c8f (stale) Mirror 15: 0xf7608c8f (stale) Mirror 17: 0xf7608c8f (stale) Mirror 19: 0xf7608c8f (stale) Mirror 20: 0xf7608c8f (stale) Mirror 21: 0xf7608c8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dadd: skipped 000859 1763402867.772172 mapwrite 0x00d5f6 thru 0x017eda (0xa8e5 bytes) 000860 1763402867.813164 fallocate 0x011a57 thru 0x01c8cb (0xae75 bytes) 000861 1763402867.819789 write 0x03a7a7 thru 0x03b999 (0x11f3 bytes) 000862 1763402867.835469 read 0x001c52 thru 0x00b5ee (0x999d bytes) 000863 1763402867.842761 trunc from 0x03c94a to 0x03dadc (0x1193 bytes) 000864 1763402867.846416 read 0x01827a thru 0x01d7c0 (0x5547 bytes) 000865 1763402867.857741 fallocate 0x00fd4f thru 0x01162f (0x18e1 bytes) 000866 1763402867.862943 write 0x01fced thru 0x02b687 (0xb99b bytes) 000867 1763402867.864261 mapread 0x02d0ab thru 0x0393c7 (0xc31d bytes) 000868 1763402867.865179 punch from 0x01f02f to 0x02e490 (0xf462 bytes) 000869 1763402867.876971 trunc from 0x0139f9 to 0x022b9e (0xf1a6 bytes) 000870 1763402867.883996 mapwrite 0x01e512 thru 0x022554 (0x4043 bytes) 000871 1763402867.932291 mapread 0x039434 thru 0x03dadc (0x46a9 bytes) 000873 1763402867.943329 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: 0xf7608c8f (stale) Mirror 15: 0xf7608c8f (stale) Mirror 19: 0xf7608c8f (stale) Mirror 20: 0xf7608c8f (stale) Mirror 21: 0xf7608c8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: 0xf7608c8f (stale) Mirror 15: 0xf7608c8f (stale) Mirror 19: 0xf7608c8f (stale) Mirror 20: 0xf7608c8f (stale) Mirror 21: 0xf7608c8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dadd: skipped 000874 1763402868.321139 fallocate 0x0296d5 thru 0x038cba (0xf5e6 bytes) 000875 1763402868.324758 mapread 0x00ca7a thru 0x018b63 (0xc0ea bytes) 000876 1763402868.346974 read 0x020951 thru 0x023d00 (0x33b0 bytes) 000877 1763402868.352014 punch from 0x039c17 to 0x03dadc (0x3ec6 bytes) 000878 1763402868.358246 write 0x02b624 thru 0x02edfb (0x37d8 bytes) 000879 1763402868.378228 trunc from 0x03d869 to 0x03dadc (0x274 bytes) 000880 1763402868.383270 mapread 0x034264 thru 0x039f6e (0x5d0b bytes) 000881 1763402868.394788 punch from 0x021838 to 0x02522d (0x39f6 bytes) 000882 1763402868.399068 mapread 0x00b6a7 thru 0x01a5f5 (0xef4f bytes) 000883 1763402868.400057 punch from 0x03a0b1 to 0x03bdaa (0x1cfa bytes) 000884 1763402868.411482 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: 0xf7608c8f (stale) Mirror 15: 0xf7608c8f (stale) Mirror 19: 0xf7608c8f (stale) Mirror 20: 0xf7608c8f (stale) Mirror 21: 0xf7608c8f (stale) CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 1: 0x16ca0002 Mirror 22: 0x16ca0002 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dadd: skipped 000885 1763402869.364177 read 0x00f605 thru 0x01f41d (0xfe19 bytes) 000886 1763402869.379697 write 0x037b23 thru 0x03a92a (0x2e08 bytes) 000887 1763402869.451686 mapwrite 0x002726 thru 0x004608 (0x1ee3 bytes) 000888 1763402869.493411 mapwrite 0x019e0c thru 0x027943 (0xdb38 bytes) 000889 1763402869.539697 write 0x028604 thru 0x02e522 (0x5f1f bytes) 000890 1763402869.547308 mapread 0x000c28 thru 0x009270 (0x8649 bytes) 000891 1763402869.552879 fallocate 0x02dd87 thru 0x0395be (0xb838 bytes) 000892 1763402869.561885 mapwrite 0x013dd1 thru 0x01b5f4 (0x7824 bytes) 000893 1763402869.589981 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 10: 0xf7608c8f (stale) Mirror 15: 0xf7608c8f (stale) Mirror 19: 0xf7608c8f (stale) Mirror 20: 0xf7608c8f (stale) Mirror 21: 0xf7608c8f (stale) Mirror 22: 0x16ca0002 (stale) CRC-32 checksum value for chunk [0, 0x3dadd): Mirror 1: 0xf6e36320 Mirror 23: 0xf6e36320 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dadd: skipped 000894 1763402870.596541 fallocate 0x0302d8 thru 0x03d26b (0xcf94 bytes) 000895 1763402870.642497 trunc from 0x03dadd to 0x038763 (0x5379 bytes) 000896 1763402870.699613 mapread 0x0353f2 thru 0x038763 (0x3372 bytes) 000897 1763402870.724184 punch from 0x02fbff to 0x038763 (0x8b65 bytes) 000898 1763402870.734714 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x38764): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38764): Mirror 10: 0xb7451a06 (stale) Mirror 15: 0xb7451a06 (stale) Mirror 19: 0xb7451a06 (stale) Mirror 20: 0xb7451a06 (stale) Mirror 21: 0xb7451a06 (stale) Mirror 22: 0xdb35203a (stale) Mirror 23: 0xb508098f (stale) CRC-32 checksum value for chunk [0, 0x38764): Mirror 1: 0x70f0937d Mirror 24: 0x70f0937d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38764: skipped 000899 1763402871.677101 fallocate 0x00837d thru 0x01580c (0xd490 bytes) 000900 1763402871.759083 write 0x01d6ba thru 0x01d960 (0x2a7 bytes) 000901 1763402871.782450 trunc from 0x0329e3 to 0x03607d (0x369b bytes) 000902 1763402871.796208 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38764): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38764): Mirror 10: 0xb7451a06 (stale) Mirror 15: 0xb7451a06 (stale) Mirror 19: 0xb7451a06 (stale) Mirror 20: 0xb7451a06 (stale) Mirror 21: 0xb7451a06 (stale) Mirror 22: 0xdb35203a (stale) Mirror 23: 0xb508098f (stale) Mirror 24: 0x70f0937d (stale) CRC-32 checksum value for chunk [0, 0x38764): Mirror 1: 0xb5952d10 Mirror 25: 0xb5952d10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38764: skipped 000903 1763402872.790378 trunc from 0x038764 to 0x01a6e0 (0x1e083 bytes) 000904 1763402872.838996 write 0x03563c thru 0x03c389 (0x6d4e bytes) 000905 1763402872.840237 read 0x0036bc thru 0x00faf0 (0xc435 bytes) 000906 1763402872.849619 mapwrite 0x00709b thru 0x00b978 (0x48de bytes) 000907 1763402872.868345 trunc from 0x03c38a to 0x039b7c (0x280d bytes) 000910 1763402872.889576 trunc from 0x039b7d to 0x030b4e (0x902e bytes) 000911 1763402872.911505 mapwrite 0x02571c thru 0x02e1d0 (0x8ab5 bytes) 000912 1763402872.958045 fallocate 0x03550d thru 0x037b29 (0x261d bytes) 000913 1763402872.970469 write 0x0191f6 thru 0x01f48a (0x6295 bytes) 000914 1763402872.978034 trunc from 0x030b4f to 0x005d58 (0x2adf6 bytes) 000915 1763402873.004377 mapwrite 0x005e82 thru 0x008735 (0x28b4 bytes) 000916 1763402873.068294 read 0x00012d thru 0x00123f (0x1113 bytes) 000917 1763402873.074361 mapwrite 0x0104b6 thru 0x014fbb (0x4b06 bytes) 000919 1763402873.153926 fallocate 0x03fe76 thru 0x03ffff (0x18a bytes) 000920 1763402873.166151 read 0x00acb3 thru 0x013d4f (0x909d bytes) 000921 1763402873.167557 read 0x00ac15 thru 0x00edca (0x41b6 bytes) 000922 1763402873.168979 mapread 0x00ae56 thru 0x01369d (0x8848 bytes) 000923 1763402873.170140 trunc from 0x014fbc to 0x0266e8 (0x1172d bytes) 000924 1763402873.188436 fallocate 0x01d723 thru 0x0295c0 (0xbe9e bytes) 000925 1763402873.194280 fallocate 0x01b9d4 thru 0x022816 (0x6e43 bytes) 000926 1763402873.200018 trunc from 0x0295c1 to 0x00253e (0x27082 bytes) 000927 1763402873.216470 read 0x0003d4 thru 0x00253e (0x216b bytes) 000928 1763402873.217151 mapread 0x001366 thru 0x00253e (0x11d9 bytes) 000929 1763402873.217981 trunc from 0x0005a3 to 0x00253e (0x1f9c bytes) 000930 1763402873.228778 write 0x03935c thru 0x03ffff (0x6ca4 bytes) 000931 1763402873.230182 trunc from 0x032652 to 0x0393cb (0x6d7a bytes) 000932 1763402873.249287 read 0x00b878 thru 0x015f42 (0xa6cb bytes) 000933 1763402873.258984 trunc from 0x00a95a to 0x011f8b (0x7632 bytes) 000934 1763402873.264610 mapwrite 0x00acbf thru 0x00bc3f (0xf81 bytes) 000935 1763402873.291467 fallocate 0x012b38 thru 0x01e578 (0xba41 bytes) 000936 1763402873.302451 write 0x03cec0 thru 0x03ffff (0x3140 bytes) 000937 1763402873.303426 trunc from 0x034cef to 0x0373c6 (0x26d8 bytes) 000938 1763402873.312081 read 0x02592d thru 0x033d60 (0xe434 bytes) 000940 1763402873.313292 read 0x00ccfe thru 0x00e976 (0x1c79 bytes) 000941 1763402873.338709 mapread 0x02fddf thru 0x030e6c (0x108e bytes) 000942 1763402873.339626 mapread 0x03f5c9 thru 0x03ffff (0xa37 bytes) 000943 1763402873.340312 punch from 0x00b4e3 to 0x01a0e5 (0xec03 bytes) 000944 1763402873.348282 trunc from 0x040000 to 0x038c00 (0x73ff bytes) 000946 1763402873.368554 mapwrite 0x02b519 thru 0x02ee10 (0x38f8 bytes) 000947 1763402873.384663 mapread 0x025ebe thru 0x02d8ca (0x7a0d bytes) 000948 1763402873.385557 punch from 0x0056ad to 0x010248 (0xab9c bytes) 000949 1763402873.391660 trunc from 0x038c01 to 0x035375 (0x388b bytes) 000950 1763402873.417311 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x35376): Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35376): Mirror 10: 0xaed25ae8 (stale) Mirror 15: 0xaed25ae8 (stale) Mirror 19: 0xaed25ae8 (stale) Mirror 20: 0xaed25ae8 (stale) Mirror 21: 0xaed25ae8 (stale) Mirror 22: 0x4e971ddd (stale) Mirror 23: 0x7e1d4714 (stale) Mirror 24: 0x24a495ea (stale) Mirror 25: 0x5a30d9a2 (stale) CRC-32 checksum value for chunk [0, 0x35376): Mirror 1: 0xa5ce89c6 Mirror 26: 0xa5ce89c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35376: skipped 000951 1763402874.294933 mapread 0x01d9f8 thru 0x02c3ba (0xe9c3 bytes) 000952 1763402874.327778 write 0x031973 thru 0x0398e4 (0x7f72 bytes) 000954 1763402874.420741 mapread 0x031c78 thru 0x0398e4 (0x7c6d bytes) 000955 1763402874.424475 fallocate 0x03fa48 thru 0x03ffff (0x5b8 bytes) 000956 1763402874.431715 fallocate 0x010c74 thru 0x015e7e (0x520b bytes) 000957 1763402874.441292 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x244:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x398e5): Mirror 10: 0xfb4a41fe (stale) Mirror 15: 0xfb4a41fe (stale) Mirror 19: 0xfb4a41fe (stale) Mirror 20: 0xfb4a41fe (stale) Mirror 21: 0xfb4a41fe (stale) Mirror 23: 0xdf3b2149 (stale) Mirror 24: 0x2b7d8f72 (stale) Mirror 25: 0x96ca3157 (stale) Mirror 26: 0x10bd1028 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398e5): Mirror 10: 0xfb4a41fe (stale) Mirror 15: 0xfb4a41fe (stale) Mirror 19: 0xfb4a41fe (stale) Mirror 20: 0xfb4a41fe (stale) Mirror 21: 0xfb4a41fe (stale) Mirror 23: 0xdf3b2149 (stale) Mirror 24: 0x2b7d8f72 (stale) Mirror 25: 0x96ca3157 (stale) Mirror 26: 0x10bd1028 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398e5: skipped 000958 1763402875.031841 MIRROR_RESYNC 10 mirrors 000959 1763402877.437548 trunc from 0x003f8a to 0x008bec (0x4c63 bytes) 000960 1763402877.479831 fallocate 0x03e5ae thru 0x03ffff (0x1a52 bytes) 000961 1763402877.484705 mapwrite 0x0041df thru 0x0109d3 (0xc7f5 bytes) 000962 1763402877.528612 punch from 0x0243a5 to 0x02fb09 (0xb765 bytes) 000963 1763402877.544922 write 0x01c72b thru 0x0282ff (0xbbd5 bytes) 000964 1763402877.564799 fallocate 0x03e3bc thru 0x03ffff (0x1c44 bytes) 000965 1763402877.570387 trunc from 0x040000 to 0x02c16a (0x13e95 bytes) 000966 1763402877.594386 write 0x016770 thru 0x01737d (0xc0e bytes) 000967 1763402877.595330 read 0x0020fe thru 0x00d040 (0xaf43 bytes) 000968 1763402877.596766 punch from 0x0139d8 to 0x02295f (0xef88 bytes) 000969 1763402877.615419 MIRROR_RESYNC 10 mirrors 000970 1763402879.959479 mapwrite 0x00cafc thru 0x015f10 (0x9415 bytes) 000971 1763402880.089236 trunc from 0x02c16b to 0x00e0df (0x1e08b bytes) 000972 1763402880.120387 fallocate 0x033540 thru 0x03ffff (0xcac0 bytes) 000973 1763402880.131499 trunc from 0x00e0e0 to 0x00ba0f (0x26d0 bytes) 000975 1763402880.151197 read 0x001db1 thru 0x00ba0f (0x9c5f bytes) 000976 1763402880.160461 read 0x0091fa thru 0x00ba0f (0x2816 bytes) 000977 1763402880.161049 mapwrite 0x023e8b thru 0x02f865 (0xb9db bytes) 000978 1763402880.213607 mapwrite 0x024a7c thru 0x027572 (0x2af7 bytes) 000979 1763402880.231006 fallocate 0x03d2ab thru 0x03ffff (0x2d55 bytes) 000980 1763402880.240599 MIRROR_RESYNC 10 mirrors 000982 1763402882.785474 mapread 0x01d407 thru 0x026f1c (0x9b16 bytes) 000983 1763402882.811756 read 0x00d34e thru 0x015879 (0x852c bytes) 000984 1763402882.826046 punch from 0x0362e4 to 0x03ffff (0x9d1c bytes) 000985 1763402882.872399 trunc from 0x040000 to 0x0045f2 (0x3ba0d bytes) 000986 1763402882.894927 write 0x011753 thru 0x01c8e0 (0xb18e bytes) 000989 1763402882.898210 fallocate 0x0230d9 thru 0x027efd (0x4e25 bytes) 000990 1763402882.902683 trunc from 0x016246 to 0x01a678 (0x4433 bytes) 000991 1763402882.915256 fallocate 0x02abdd thru 0x036442 (0xb866 bytes) 000992 1763402882.920666 trunc from 0x036443 to 0x009c6b (0x2c7d7 bytes) 000993 1763402882.933696 trunc from 0x000d73 to 0x007f82 (0x7210 bytes) 000994 1763402882.939855 read 0x002cd2 thru 0x007e60 (0x518f bytes) 000996 1763402882.950951 trunc from 0x001dd2 to 0x009c6b (0x7e9a bytes) 000997 1763402882.956571 fallocate 0x0046b6 thru 0x007475 (0x2dc0 bytes) 000998 1763402882.961689 trunc from 0x009c6c to 0x016c36 (0xcfcb bytes) 000999 1763402882.983326 mapwrite 0x013bfe thru 0x01d4ca (0x98cd bytes) All operations completed A-OK! PASS 70a (91s) == sanity-flr test 71: check mirror extend parent fid ==== 13:08:07 (1763402887) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x276:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 200a: stress test ===================== 13:08:15 (1763402895) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1970240 .. Extending file size to 4129952 .. Extending file size to 6183808 .. Extending file size to 4491392 .. Extending file size to 6746848 .. Extending file size to 7587552 .. Extending file size to 7638400 .. Extending file size to 6756992 .. Extending file size to 8094656 .. Extending file size to 7308000 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7958752 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8667424 .. Extending file size to 8595040 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8734528 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8706048 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9241824 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59438 59439 59442 59443 59445 done 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (74s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:09:29 (1763402969) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8535008 .. Extending file size to 5824480 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8975488 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8808768 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9164640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61849 61850 61853 61854 61855 done 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:10:42 (1763403042) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1268960 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22668 3584352 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 14716 3592304 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37384 7176656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1268960 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2348 1268956 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 22668 3584352 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 14716 3592304 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37384 7176656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0644407 s, 32.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114608 s, 18.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf56d8c3b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:10:54 (1763403054) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108615 s, 19.3 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 13:11:00 (1763403060) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xa9:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:11:05 (1763403065) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0682809 s, 30.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:11:11 (1763403071) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.076215 s, 27.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x313:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x313:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x314:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:11:16 (1763403076) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0807734 s, 26.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:11:46 (1763403106) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583027 s, 18.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x314:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x314:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x315:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:12:29 (1763403149) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0990759 s, 21.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:13:17 (1763403197) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg245-server: Pool lustre.test_204f created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0799915 s, 26.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg245-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:14:25 (1763403265) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:14:32 (1763403272) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:14:39 (1763403279) oleg245-server: Pool lustre.test_206 created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg245-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:15:03 (1763403303) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428182 s, 24.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262866 s, 39.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa879f70a Mirror 3: 0xa879f70a PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:15:13 (1763403313) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:15:17 (1763403317) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:15:21 (1763403321) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0606954 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2070000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1da0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1da0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1da0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe20000 Read this (4 bytes): BBBB 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 209a (31s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:15:52 (1763403352) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0399609 s, 205 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:15:58 (1763403358) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240254 s, 43.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611260 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-17 13:15:59.000000000 -0500 Modify: 2025-11-17 13:15:59.000000000 -0500 Change: 2025-11-17 13:16:00.000000000 -0500 Birth: 2025-11-17 13:15:59.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:16:05 (1763403365) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295742 s, 35.5 MB/s oleg245-server: debugfs 1.47.3-wc1 (8-Aug-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg245-server: debugfs 1.47.3-wc1 (8-Aug-2025) oleg245-server: O/280000401/d27/bb: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:16:23 (1763403383) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.152017 s, 269 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0252149 s, 162 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 17 13:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x81:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 17 13:16 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:16:30 (1763403390) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:16:36 (1763403396) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1527 sec =========== 13:16:45 (1763403405) === sanity-flr: start cleanup 13:16:46 (1763403406) === === sanity-flr: finish cleanup 13:16:54 (1763403414) ===