-----============= acceptance-small: sanity-flr ============----- Thu Jan 22 04:21:27 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg210-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:21:53 (1769073713) === oleg210-client.virtnet: executing check_config_client /mnt/lustre oleg210-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg210-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92830a4af800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92830a4af800.idle_timeout=debug disable quota as required oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 04:22:37 (1769073757) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8220] == sanity-flr test 0a: lfs mirror create with -N option == 04:22:39 (1769073759) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:22:58 (1769073778) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:23:02 (1769073782) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:23:07 (1769073787) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:23:33 (1769073813) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:23:37 (1769073817) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 04:23:42 (1769073822) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff92830a4af800.stats=clear osc.lustre-OST0001-osc-ffff92830a4af800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:23:52 (1769073832) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:24:04 (1769073844) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:24:12 (1769073852) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:24:16 (1769073856) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 04:24:26 (1769073866) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:24:34 (1769073874) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:24:44 (1769073884) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:24:53 (1769073893) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (10s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:25:07 (1769073907) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:25:16 (1769073916) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.65299 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932646 s, 11.2 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:25:27 (1769073927) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.609826 s, 29.2 MB/s PASS 22 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:25:37 (1769073937) Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:26:58 (1769074018) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.304163 s, 13.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:27:45 (1769074065) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:29:05 (1769074145) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.36917 s, 15.6 MB/s -rw-r--r-- 1 root root 52428800 Jan 22 04:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.998s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92830a4af800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.094s user 0m0.001s sys 0m0.173s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.215s user 0m0.001s sys 0m0.184s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92830a4af800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:30:37 (1769074237) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:30:41 (1769074241) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:30:46 (1769074246) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 04:30:50 (1769074250) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 04:30:59 (1769074259) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.752374 s, 16.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:31:12 (1769074272) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046415 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259188 s, 40.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:31:20 (1769074280) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:31:29 (1769074289) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0007432 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012373 s, 828 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000830769 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000846722 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 04:31:38 (1769074298) create test files with size 15740248 .. 38b9aa4f598b4c97bdc76412ec00c667 - 15026444562455f45685962b5b46969f - a3467154b13a30b9ad8251ebb51a7a6a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 04:32:09 (1769074329) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:32:23 (1769074343) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:32:31 (1769074351) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807859 s, 13.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x37:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510298 s, 20.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x38:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x39:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x39:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 04:32:40 (1769074360) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0741481 s, 28.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.321493 s, 13.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13376 s, 15.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 04:32:53 (1769074373) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 04:32:58 (1769074378) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:33:01 (1769074381) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981129 s, 10.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 04:33:10 (1769074390) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.360626 s, 8.7 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:33:19 (1769074399) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x42:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x43:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x43:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:33:29 (1769074409) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.39554 s, 26.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:33:43 (1769074423) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff928304d1c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.277307 s, 37.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff928306bbe000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:33:59 (1769074439) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:34:44 (1769074484) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:35:23 (1769074523) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:35:26 (1769074526) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.21438 s, 14.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.127031 s, 24.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd8f171e (stale) Mirror 2: 0xd8f171e (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 3b3a8e36426875114558e092f4c16d3d - mirror 1 checksum: 7e3312bc8253af38b132e8f2b2b4da95 - mirror 2 checksum: 3b3a8e36426875114558e092f4c16d3d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 22 04:35:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 22 04:35:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3b3a8e36426875114558e092f4c16d3d - mirror 1 checksum: 7e3312bc8253af38b132e8f2b2b4da95 - mirror 2 checksum: 7e3312bc8253af38b132e8f2b2b4da95 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 22 04:35:29 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 22 04:35:29 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:35:38 (1769074538) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243148 s, 43.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:35:46 (1769074546) Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 04:36:01 (1769074561) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 04:36:30 (1769074590) targets are mounted 04:36:30 (1769074590) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887032 s, 11.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:36:46 (1769074606) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0384069 s, 533 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0125801 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:36:56 (1769074616) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0510207 s, 401 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0108755 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:37:05 (1769074625) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00785087 s, 2.1 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:37:13 (1769074633) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0795982 s, 257 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:37:21 (1769074641) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.010245 s, 400 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0481714 s, 85.0 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:37:29 (1769074649) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:37:51 (1769074671) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:38:20 (1769074700) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (28s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:38:48 (1769074728) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:39:14 (1769074754) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 04:39:22 (1769074762) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55955 mirror split pid 55956 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55955 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 55956 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 04:40:31 (1769074831) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4833 skipping zero length punch hole skipping zero length zero range skipping zero size read 000004 1769074833.691652 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x40000 000005 1769074834.016808 fallocate 0x03bf18 thru 0x03ffff (0x40e8 bytes) 000006 1769074834.073185 write 0x02feb8 thru 0x03993d (0x9a86 bytes) 000008 1769074834.117762 trunc from 0x03b473 to 0x03ffff (0x4b8d bytes) 000009 1769074834.129473 punch from 0x01397b to 0x015ec5 (0x254b bytes) 000010 1769074834.152530 write 0x031340 thru 0x031b92 (0x853 bytes) 000011 1769074834.154485 write 0x01b559 thru 0x0230d3 (0x7b7b bytes) 000012 1769074834.178936 trunc from 0x040000 to 0x00bcf8 (0x34307 bytes) 000013 1769074834.249784 trunc from 0x00bcf9 to 0x02cc20 (0x20f28 bytes) 000014 1769074834.283570 read 0x0112bb thru 0x01cd38 (0xba7e bytes) 000015 1769074834.316587 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) 000016 1769074834.575849 mapread 0x0020d9 thru 0x006ee5 (0x4e0d bytes) 000017 1769074834.629936 mapwrite 0x00a496 thru 0x00ae56 (0x9c1 bytes) 000019 1769074834.660296 mapread 0x028b0d thru 0x02cc20 (0x4114 bytes) 000020 1769074834.670651 trunc from 0x00fb29 to 0x01ee58 (0xf330 bytes) 000021 1769074834.684283 read 0x02470b thru 0x02cc20 (0x8516 bytes) 000022 1769074834.685131 write 0x01acf8 thru 0x02ab21 (0xfe2a bytes) 000023 1769074834.704648 trunc from 0x02cc21 to 0x01e164 (0xeabc bytes) 000024 1769074834.728912 trunc from 0x0110c7 to 0x01e164 (0xd09e bytes) 000025 1769074834.749475 punch from 0x007cd0 to 0x016e68 (0xf199 bytes) 000026 1769074834.756589 mapwrite 0x0003d0 thru 0x008990 (0x85c1 bytes) 000027 1769074834.798259 trunc from 0x01e165 to 0x0292bc (0xb158 bytes) 000028 1769074834.819400 mapread 0x0253bf thru 0x0292bc (0x3efe bytes) 000029 1769074834.835982 fallocate 0x00a20e thru 0x00f8de (0x56d1 bytes) 000030 1769074834.846653 punch from 0x017d1b to 0x01f209 (0x74ef bytes) 000031 1769074834.854596 read 0x01f339 thru 0x023376 (0x403e bytes) 000033 1769074834.869814 mapread 0x010a5a thru 0x0161cb (0x5772 bytes) 000034 1769074834.878505 punch from 0x0054e7 to 0x010ed8 (0xb9f2 bytes) 000035 1769074834.890779 write 0x02bdbb thru 0x037f91 (0xc1d7 bytes) 000036 1769074834.892892 mapread 0x00d0a1 thru 0x017041 (0x9fa1 bytes) 000038 1769074834.927787 MIRROR_EXTEND to 2 mirrors 000039 1769074835.967181 fallocate 0x026136 thru 0x02e5ac (0x8477 bytes) 000040 1769074836.034272 punch from 0x037bc1 to 0x037f91 (0x3d1 bytes) 000041 1769074836.052149 mapread 0x031336 thru 0x037f91 (0x6c5c bytes) 000042 1769074836.099002 MIRROR_RESYNC 2 mirrors 000043 1769074836.971367 punch from 0x01f680 to 0x020965 (0x12e6 bytes) 000044 1769074837.024015 read 0x006306 thru 0x012502 (0xc1fd bytes) 000045 1769074837.039665 MIRROR_RESYNC 2 mirrors 000046 1769074837.561119 read 0x037416 thru 0x037f91 (0xb7c bytes) 000047 1769074837.566108 mapwrite 0x00dca4 thru 0x0154dd (0x783a bytes) 000048 1769074837.660393 trunc from 0x014ba6 to 0x01fa1c (0xae77 bytes) 000049 1769074837.666538 read 0x000f02 thru 0x00cb8c (0xbc8b bytes) 000050 1769074837.676557 write 0x03e6db thru 0x03ffff (0x1925 bytes) 000051 1769074837.685507 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4f65decc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x310155a Mirror 3: 0x310155a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000053 1769074838.736811 write 0x029d84 thru 0x02caea (0x2d67 bytes) 000054 1769074838.793647 read 0x031efe thru 0x03ce0f (0xaf12 bytes) 000055 1769074838.805025 trunc from 0x02b66a to 0x02dc4e (0x25e5 bytes) 000056 1769074838.810586 trunc from 0x0359d7 to 0x03c83d (0x6e67 bytes) 000057 1769074838.816353 mapwrite 0x005f53 thru 0x00b422 (0x54d0 bytes) 000058 1769074838.846671 punch from 0x01afa0 to 0x01f2e8 (0x4349 bytes) 000059 1769074838.853001 write 0x019e74 thru 0x01b960 (0x1aed bytes) 000060 1769074838.862889 fallocate 0x0139e3 thru 0x01f546 (0xbb64 bytes) 000061 1769074838.868426 trunc from 0x03676b to 0x03ffff (0x9895 bytes) 000062 1769074838.873677 fallocate 0x024e18 thru 0x02d50f (0x86f8 bytes) 000063 1769074838.887650 mapread 0x01062b thru 0x01db11 (0xd4e7 bytes) 000064 1769074838.895335 trunc from 0x040000 to 0x01c2da (0x23d25 bytes) 000065 1769074838.931725 read 0x002c8a thru 0x005f57 (0x32ce bytes) 000066 1769074838.937963 MIRROR_RESYNC 3 mirrors 000067 1769074840.152487 trunc from 0x01c2db to 0x02f116 (0x12e3c bytes) 000068 1769074840.188147 mapwrite 0x036a2c thru 0x03ffff (0x95d4 bytes) 000069 1769074840.235762 trunc from 0x01e2b4 to 0x02c4ad (0xe1fa bytes) 000071 1769074840.241548 trunc from 0x024ca9 to 0x02f68e (0xa9e6 bytes) 000072 1769074840.249971 MIRROR_RESYNC 3 mirrors 000073 1769074841.111583 MIRROR_EXTEND to 4 mirrors 000075 1769074841.936737 fallocate 0x037148 thru 0x03bebb (0x4d74 bytes) 000076 1769074841.980753 read 0x0275c1 thru 0x0285c6 (0x1006 bytes) 000077 1769074841.989652 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc9cfacf4 (stale) Mirror 4: 0xc9cfacf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc9cfacf4 (stale) Mirror 4: 0xc9cfacf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000078 1769074842.578237 fallocate 0x00e852 thru 0x01a8ef (0xc09e bytes) 000079 1769074842.593242 read 0x03e6c0 thru 0x03ffff (0x1940 bytes) 000080 1769074842.622570 read 0x01d7cd thru 0x0242ae (0x6ae2 bytes) 000081 1769074842.648223 trunc from 0x040000 to 0x013640 (0x2c9bf bytes) 000082 1769074842.731011 fallocate 0x01e3d9 thru 0x0226f6 (0x431e bytes) 000083 1769074842.741427 trunc from 0x007a30 to 0x00f053 (0x7624 bytes) 000084 1769074842.747786 mapwrite 0x02f751 thru 0x03dc19 (0xe4c9 bytes) 000085 1769074842.976613 mapread 0x00d813 thru 0x016e06 (0x95f4 bytes) 000086 1769074843.018865 write 0x0109b6 thru 0x01896a (0x7fb5 bytes) 000087 1769074843.031675 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc1a): Mirror 3: 0xaf33dae0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc1a): Mirror 3: 0xaf33dae0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc1a: skipped 000088 1769074843.321972 MIRROR_RESYNC 2 mirrors 000089 1769074844.140558 trunc from 0x021be8 to 0x02a3d1 (0x87ea bytes) 000090 1769074844.207566 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) 000091 1769074844.432361 trunc from 0x01e5ad to 0x02a28a (0xbcde bytes) 000092 1769074844.436378 fallocate 0x039f60 thru 0x03ffff (0x60a0 bytes) 000093 1769074844.441463 mapwrite 0x00cf10 thru 0x016810 (0x9901 bytes) 000094 1769074844.475728 punch from 0x00fd7e to 0x016bd7 (0x6e5a bytes) 000095 1769074844.482878 fallocate 0x0109dd thru 0x01b4d6 (0xaafa bytes) 000096 1769074844.487752 fallocate 0x0055bc thru 0x00fb8f (0xa5d4 bytes) 000097 1769074844.492411 punch from 0x028926 to 0x037783 (0xee5e bytes) 000098 1769074844.496320 read 0x00e60b thru 0x012337 (0x3d2d bytes) 000101 1769074844.504312 fallocate 0x037418 thru 0x03a3c7 (0x2fb0 bytes) 000102 1769074844.508820 trunc from 0x03dc1a to 0x023da7 (0x19e72 bytes) 000103 1769074844.525010 trunc from 0x023b6a to 0x023da7 (0x23e bytes) 000104 1769074844.529368 trunc from 0x021150 to 0x023da7 (0x2c58 bytes) 000106 1769074844.535914 punch from 0x01207f to 0x01a9de (0x8960 bytes) 000107 1769074844.539955 mapread 0x0232e8 thru 0x023da7 (0xac0 bytes) 000108 1769074844.547943 write 0x009663 thru 0x010870 (0x720e bytes) 000109 1769074844.549647 mapwrite 0x018774 thru 0x0204e8 (0x7d75 bytes) 000110 1769074844.574607 fallocate 0x01e7fd thru 0x02a5cc (0xbdd0 bytes) 000114 1769074844.581017 trunc from 0x01c0ed to 0x01c296 (0x1aa bytes) 000115 1769074844.586209 write 0x006e65 thru 0x016e06 (0xffa2 bytes) 000116 1769074844.595157 fallocate 0x00797f thru 0x00f4e5 (0x7b67 bytes) 000117 1769074844.599559 read 0x00c934 thru 0x019e09 (0xd4d6 bytes) 000118 1769074844.600380 punch from 0x006c6c to 0x014832 (0xdbc7 bytes) 000119 1769074844.612846 write 0x001faf thru 0x008071 (0x60c3 bytes) 000120 1769074844.620406 punch from 0x01115d to 0x0181a4 (0x7048 bytes) 000121 1769074844.625522 write 0x023780 thru 0x028f59 (0x57da bytes) 000122 1769074844.634572 fallocate 0x031a4b thru 0x03e8a2 (0xce58 bytes) 000123 1769074844.638790 mapwrite 0x02711c thru 0x02ada5 (0x3c8a bytes) 000124 1769074844.675476 punch from 0x00ced2 to 0x0170c5 (0xa1f4 bytes) 000125 1769074844.680777 trunc from 0x02ada6 to 0x0247df (0x65c6 bytes) 000126 1769074844.697524 trunc from 0x0247e0 to 0x03d9d4 (0x191f5 bytes) 000127 1769074844.723762 mapwrite 0x008e58 thru 0x014147 (0xb2f0 bytes) 000128 1769074844.781682 mapread 0x0081b5 thru 0x013a0e (0xb85a bytes) 000129 1769074844.782805 mapwrite 0x034022 thru 0x036037 (0x2016 bytes) 000130 1769074844.795006 read 0x0301c2 thru 0x032928 (0x2767 bytes) 000131 1769074844.795680 mapwrite 0x03526a thru 0x03ed1e (0x9ab5 bytes) 000132 1769074844.830681 write 0x00a2f2 thru 0x01430e (0xa01d bytes) 000133 1769074844.831964 punch from 0x012b25 to 0x021167 (0xe643 bytes) 000134 1769074844.844615 mapwrite 0x020990 thru 0x022acd (0x213e bytes) 000135 1769074844.868291 punch from 0x02bfb8 to 0x039df9 (0xde42 bytes) 000137 1769074844.875279 fallocate 0x03acff thru 0x03ffff (0x5301 bytes) 000138 1769074844.880411 fallocate 0x00d9f6 thru 0x01c33e (0xe949 bytes) 000139 1769074844.887870 trunc from 0x040000 to 0x0211ea (0x1ee15 bytes) 000140 1769074844.909366 trunc from 0x00e851 to 0x010e07 (0x25b7 bytes) 000141 1769074844.917401 punch from 0x01e31c to 0x01e99c (0x681 bytes) 000142 1769074844.924470 punch from 0x005b95 to 0x007fe4 (0x2450 bytes) 000143 1769074844.931750 punch from 0x01fa18 to 0x0211ea (0x17d3 bytes) 000144 1769074844.941246 trunc from 0x01b3ae to 0x0211ea (0x5e3d bytes) 000145 1769074844.945889 mapread 0x00b8b4 thru 0x015b0c (0xa259 bytes) 000146 1769074844.989759 read 0x01518e thru 0x0211ea (0xc05d bytes) 000147 1769074845.001062 mapread 0x0056ee thru 0x014b83 (0xf496 bytes) 000148 1769074845.002157 trunc from 0x000da2 to 0x010130 (0xf38f bytes) 000149 1769074845.018664 trunc from 0x018332 to 0x01969b (0x136a bytes) 000150 1769074845.026245 mapread 0x020d24 thru 0x0211ea (0x4c7 bytes) 000151 1769074845.030453 mapwrite 0x02d3c4 thru 0x033ee5 (0x6b22 bytes) 000152 1769074845.088129 mapread 0x02401e thru 0x02a6b7 (0x669a bytes) 000153 1769074845.088999 write 0x01d9b7 thru 0x02684c (0x8e96 bytes) 000154 1769074845.091720 write 0x01e7d4 thru 0x023d10 (0x553d bytes) 000155 1769074845.093909 punch from 0x02bae8 to 0x033ee5 (0x83fe bytes) 000156 1769074845.099431 punch from 0x02c456 to 0x02d48d (0x1038 bytes) 000157 1769074845.103462 write 0x025915 thru 0x026ff3 (0x16df bytes) 000158 1769074845.104956 fallocate 0x01e90b thru 0x022a6b (0x4161 bytes) 000159 1769074845.110518 write 0x01d57f thru 0x0238d4 (0x6356 bytes) 000160 1769074845.111908 trunc from 0x02ead0 to 0x033ee5 (0x5416 bytes) 000161 1769074845.116464 trunc from 0x033ed7 to 0x033ee5 (0xf bytes) 000162 1769074845.120688 trunc from 0x033ee6 to 0x018ccc (0x1b219 bytes) 000163 1769074845.136505 trunc from 0x018ccd to 0x020298 (0x75cc bytes) 000164 1769074845.154988 write 0x03abde thru 0x03ffff (0x5422 bytes) 000165 1769074845.156323 punch from 0x037668 to 0x03906d (0x1a06 bytes) 000166 1769074845.164930 mapwrite 0x019163 thru 0x01c6f6 (0x3594 bytes) 000167 1769074845.233124 read 0x02141c thru 0x024b70 (0x3755 bytes) 000170 1769074845.247124 mapwrite 0x0115e4 thru 0x02076f (0xf18c bytes) 000171 1769074845.275451 write 0x034283 thru 0x035bcd (0x194b bytes) 000172 1769074845.276240 mapwrite 0x0003a9 thru 0x0023c4 (0x201c bytes) 000173 1769074845.291388 mapwrite 0x006ba4 thru 0x0127b0 (0xbc0d bytes) 000174 1769074845.313798 punch from 0x003e57 to 0x00dd10 (0x9eba bytes) 000175 1769074845.323907 trunc from 0x040000 to 0x029598 (0x16a67 bytes) 000176 1769074845.361006 mapread 0x011276 thru 0x019a6f (0x87fa bytes) 000178 1769074845.364469 mapread 0x01c13f thru 0x029598 (0xd45a bytes) 000179 1769074845.365769 punch from 0x001ed1 to 0x00c642 (0xa772 bytes) 000180 1769074845.378131 trunc from 0x029599 to 0x033628 (0xa090 bytes) 000181 1769074845.425745 fallocate 0x038fda thru 0x03cafc (0x3b23 bytes) 000182 1769074845.438620 read 0x0154d0 thru 0x01ecae (0x97df bytes) 000183 1769074845.442908 punch from 0x00739b to 0x00d784 (0x63ea bytes) 000184 1769074845.453237 fallocate 0x002e55 thru 0x00782a (0x49d6 bytes) 000185 1769074845.468922 fallocate 0x02287c thru 0x026301 (0x3a86 bytes) 000186 1769074845.474433 trunc from 0x03cafd to 0x01c636 (0x204c6 bytes) 000187 1769074845.522483 trunc from 0x01c637 to 0x02985f (0xd229 bytes) 000188 1769074845.547876 write 0x028012 thru 0x031ae3 (0x9ad2 bytes) 000190 1769074845.568936 trunc from 0x02550f to 0x027bdb (0x26cd bytes) 000191 1769074845.574260 trunc from 0x025d1e to 0x031ae3 (0xbdc6 bytes) 000192 1769074845.589497 read 0x00b820 thru 0x019e8a (0xe66b bytes) 000193 1769074845.591272 mapread 0x0227f2 thru 0x025e66 (0x3675 bytes) 000195 1769074845.605460 fallocate 0x006437 thru 0x0109d9 (0xa5a3 bytes) 000196 1769074845.612146 write 0x00e27c thru 0x015159 (0x6ede bytes) 000197 1769074845.613211 trunc from 0x0252d6 to 0x031ae3 (0xc80e bytes) 000199 1769074845.623454 write 0x008139 thru 0x015f4c (0xde14 bytes) 000200 1769074845.625741 mapwrite 0x01811e thru 0x026332 (0xe215 bytes) 000201 1769074845.686956 trunc from 0x029027 to 0x031ae3 (0x8abd bytes) 000202 1769074845.691986 mapread 0x003037 thru 0x00e253 (0xb21d bytes) 000204 1769074845.695664 fallocate 0x03c58d thru 0x03d8d5 (0x1349 bytes) 000205 1769074845.713466 fallocate 0x017b98 thru 0x019473 (0x18dc bytes) 000206 1769074845.731124 fallocate 0x0347c7 thru 0x037cca (0x3504 bytes) 000207 1769074845.738420 fallocate 0x0278c9 thru 0x0295e7 (0x1d1f bytes) 000208 1769074845.750253 mapwrite 0x03f654 thru 0x03fc8c (0x639 bytes) 000209 1769074845.836716 trunc from 0x03fc8d to 0x005ac7 (0x3a1c5 bytes) 000210 1769074845.873443 mapwrite 0x016e6f thru 0x0236f5 (0xc887 bytes) 000211 1769074846.015341 trunc from 0x00035b to 0x002e57 (0x2afd bytes) 000213 1769074846.021256 write 0x003be8 thru 0x01185f (0xdc78 bytes) 000214 1769074846.029408 write 0x02598c thru 0x027dd5 (0x244a bytes) 000215 1769074846.031325 mapwrite 0x010118 thru 0x016c29 (0x6b12 bytes) 000216 1769074846.075848 write 0x01540f thru 0x016af9 (0x16eb bytes) 000217 1769074846.078369 mapread 0x02339b thru 0x027dd5 (0x4a3b bytes) 000218 1769074846.087693 trunc from 0x017065 to 0x017cd0 (0xc6c bytes) 000219 1769074846.097435 mapread 0x004ae9 thru 0x00e835 (0x9d4d bytes) 000221 1769074846.099168 fallocate 0x0154c6 thru 0x021aba (0xc5f5 bytes) 000222 1769074846.106769 read 0x016f59 thru 0x01bd11 (0x4db9 bytes) 000223 1769074846.115835 read 0x019d50 thru 0x0204b8 (0x6769 bytes) 000224 1769074846.116473 mapwrite 0x00a1af thru 0x017833 (0xd685 bytes) 000227 1769074846.149194 MIRROR_EXTEND to 2 mirrors 000228 1769074846.714368 mapwrite 0x02322f thru 0x02c37f (0x9151 bytes) 000229 1769074846.858674 write 0x034020 thru 0x03c17b (0x815c bytes) 000230 1769074846.860072 write 0x00aa6f thru 0x00c2f0 (0x1882 bytes) 000231 1769074846.869321 mapread 0x022333 thru 0x0238c7 (0x1595 bytes) 000232 1769074846.870204 punch from 0x00fc27 to 0x01d629 (0xda03 bytes) 000233 1769074846.876622 write 0x01452f thru 0x0231a0 (0xec72 bytes) 000234 1769074846.892000 read 0x037c65 thru 0x03c17b (0x4517 bytes) 000235 1769074846.897627 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c17c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c17c): Mirror 2: 0xa0b1de3b (stale) CRC-32 checksum value for chunk [0, 0x3c17c): Mirror 1: 0x5d83bd5c Mirror 3: 0x5d83bd5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c17c: skipped 000238 1769074848.016130 mapwrite 0x026e79 thru 0x02c304 (0x548c bytes) 000239 1769074848.117219 mapwrite 0x03682e thru 0x03809a (0x186d bytes) 000240 1769074848.155411 punch from 0x004c27 to 0x0071e7 (0x25c1 bytes) 000242 1769074848.177583 mapwrite 0x0137c4 thru 0x019cdf (0x651c bytes) 000243 1769074848.213182 trunc from 0x03c17c to 0x021ab1 (0x1a6ca bytes) 000244 1769074848.288337 mapwrite 0x00ef07 thru 0x019e53 (0xaf4d bytes) 000245 1769074848.324576 write 0x03794d thru 0x03c8dd (0x4f91 bytes) 000246 1769074848.325874 fallocate 0x0322ad thru 0x039c5b (0x79af bytes) 000248 1769074848.337688 fallocate 0x0354ee thru 0x03ffff (0xab12 bytes) 000249 1769074848.346477 trunc from 0x018e5b to 0x0287ae (0xf954 bytes) 000251 1769074848.360539 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c8de): Mirror 2: 0x8e28ffe8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c8de): Mirror 2: 0x8e28ffe8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c8de: skipped 000252 1769074848.947263 MIRROR_RESYNC 2 mirrors 000253 1769074850.294108 mapread 0x036938 thru 0x03c8dd (0x5fa6 bytes) 000254 1769074850.328397 mapwrite 0x02b247 thru 0x031b67 (0x6921 bytes) 000255 1769074850.466255 write 0x02ee8d thru 0x03a074 (0xb1e8 bytes) 000257 1769074850.478662 fallocate 0x0156c7 thru 0x015d3a (0x674 bytes) 000258 1769074850.485193 mapwrite 0x03d431 thru 0x03ffff (0x2bcf bytes) 000259 1769074850.555339 mapwrite 0x0382a0 thru 0x03f1a6 (0x6f07 bytes) 000260 1769074850.585836 read 0x001981 thru 0x00ee6d (0xd4ed bytes) 000261 1769074850.608875 trunc from 0x01bcb3 to 0x01eec0 (0x320e bytes) 000262 1769074850.628668 trunc from 0x040000 to 0x00de9f (0x32160 bytes) 000263 1769074850.688890 trunc from 0x00dea0 to 0x00da45 (0x45a bytes) 000264 1769074850.721482 write 0x01f003 thru 0x0238b0 (0x48ae bytes) 000265 1769074850.722696 read 0x01bf36 thru 0x0238b0 (0x797b bytes) 000266 1769074850.730743 read 0x022fc6 thru 0x0238b0 (0x8eb bytes) 000267 1769074850.731465 mapwrite 0x01790d thru 0x019320 (0x1a14 bytes) 000268 1769074850.768585 fallocate 0x03b3f7 thru 0x03c429 (0x1033 bytes) 000269 1769074850.779708 fallocate 0x01c4ce thru 0x02ac9e (0xe7d1 bytes) 000270 1769074850.802922 punch from 0x010d3d to 0x01d861 (0xcb25 bytes) 000271 1769074850.815571 mapread 0x00106a thru 0x00e996 (0xd92d bytes) 000272 1769074850.817025 mapread 0x001bf4 thru 0x001f92 (0x39f bytes) 000273 1769074850.819800 fallocate 0x010328 thru 0x01d70c (0xd3e5 bytes) 000274 1769074850.842654 mapread 0x004b49 thru 0x012ae5 (0xdf9d bytes) 000275 1769074850.881338 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x238b1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x238b1): Mirror 2: 0xf9c2db8c (stale) CRC-32 checksum value for chunk [0, 0x238b1): Mirror 1: 0x4151bdb2 Mirror 3: 0x4151bdb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x238b1: skipped 000276 1769074852.243225 mapwrite 0x005418 thru 0x005f9a (0xb83 bytes) 000277 1769074852.372626 fallocate 0x031b6e thru 0x03ac53 (0x90e6 bytes) 000279 1769074852.387746 trunc from 0x00128a to 0x00d32d (0xc0a4 bytes) 000280 1769074852.399601 read 0x005081 thru 0x00e468 (0x93e8 bytes) 000281 1769074852.433241 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x238b1): Mirror 2: 0xf9c2db8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x238b1): Mirror 2: 0xf9c2db8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x238b1: skipped 000282 1769074852.971861 write 0x00e764 thru 0x014fd5 (0x6872 bytes) 000283 1769074853.019570 write 0x01a4e3 thru 0x024965 (0xa483 bytes) 000284 1769074853.042718 trunc from 0x01dfec to 0x020b99 (0x2bae bytes) 000285 1769074853.092821 mapwrite 0x027620 thru 0x02f211 (0x7bf2 bytes) 000286 1769074853.177407 trunc from 0x02f212 to 0x032bb9 (0x39a8 bytes) 000287 1769074853.205845 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) 000288 1769074853.383018 punch from 0x01a5a7 to 0x01a8a9 (0x303 bytes) 000289 1769074853.394527 write 0x013774 thru 0x018a7f (0x530c bytes) 000291 1769074853.419134 punch from 0x00e6bc to 0x019c7d (0xb5c2 bytes) 000292 1769074853.436144 mapread 0x0206ad thru 0x02351b (0x2e6f bytes) 000293 1769074853.447574 mapread 0x02cb3d thru 0x030180 (0x3644 bytes) 000294 1769074853.457347 punch from 0x00814d to 0x00f012 (0x6ec6 bytes) 000295 1769074853.463428 trunc from 0x032bba to 0x02e068 (0x4b51 bytes) 000296 1769074853.486003 read 0x02208b thru 0x02a5fc (0x8572 bytes) 000297 1769074853.490369 write 0x02338c thru 0x03317c (0xfdf1 bytes) 000299 1769074853.500911 read 0x0084aa thru 0x014992 (0xc4e9 bytes) 000301 1769074853.520722 punch from 0x01ea1e to 0x02d7e2 (0xedc5 bytes) 000302 1769074853.568390 trunc from 0x03317d to 0x03c5b2 (0x9436 bytes) 000303 1769074853.593388 mapwrite 0x01ec3e thru 0x022e1a (0x41dd bytes) 000304 1769074853.672031 fallocate 0x010e69 thru 0x01a54e (0x96e6 bytes) 000305 1769074853.678420 mapwrite 0x02e8f8 thru 0x03cca1 (0xe3aa bytes) 000306 1769074853.758343 trunc from 0x03cca2 to 0x01b2cc (0x219d5 bytes) 000308 1769074853.798672 punch from 0x01af05 to 0x01b2cc (0x3c8 bytes) 000309 1769074853.811494 read 0x0098f1 thru 0x0196c7 (0xfdd7 bytes) 000310 1769074853.816498 punch from 0x0147c4 to 0x01b2cc (0x6b09 bytes) 000311 1769074853.826653 fallocate 0x01d303 thru 0x02acc3 (0xd9c1 bytes) 000312 1769074853.836120 trunc from 0x01b2cd to 0x00ebda (0xc6f2 bytes) 000313 1769074853.861709 read 0x005f9e thru 0x00ad2f (0x4d92 bytes) 000314 1769074853.862363 read 0x0045a7 thru 0x00ebda (0xa634 bytes) 000315 1769074853.862855 read 0x007488 thru 0x00ebda (0x7753 bytes) 000316 1769074853.863391 fallocate 0x03e995 thru 0x03ffff (0x166b bytes) 000318 1769074853.874828 punch from 0x007377 to 0x00c169 (0x4df3 bytes) 000319 1769074853.882613 write 0x002dc2 thru 0x0042d3 (0x1512 bytes) 000320 1769074853.886419 trunc from 0x00ebdb to 0x00ab1d (0x40bd bytes) 000321 1769074853.921016 punch from 0x008f2e to 0x00ab1d (0x1bf0 bytes) 000323 1769074853.931637 read 0x00809f thru 0x00a02e (0x1f90 bytes) 000324 1769074853.936420 punch from 0x00304a to 0x00ab1d (0x7ad4 bytes) 000325 1769074853.950306 mapread 0x00587e thru 0x009556 (0x3cd9 bytes) 000326 1769074853.995477 punch from 0x009571 to 0x00ab1d (0x15ad bytes) 000327 1769074853.999669 write 0x036299 thru 0x03cc99 (0x6a01 bytes) 000328 1769074854.002284 trunc from 0x03cc9a to 0x00332f (0x3996a bytes) 000329 1769074854.025284 read 0x000c4a thru 0x00332f (0x26e6 bytes) 000330 1769074854.030519 punch from 0x0032b9 to 0x00332f (0x77 bytes) 000331 1769074854.039520 MIRROR_EXTEND to 2 mirrors 000332 1769074854.819891 trunc from 0x001825 to 0x00332f (0x1b0b bytes) 000333 1769074854.903657 write 0x03dcc3 thru 0x03ffff (0x233d bytes) 000334 1769074854.908702 mapwrite 0x03e74a thru 0x03f3c7 (0xc7e bytes) 000335 1769074854.937738 write 0x011ec8 thru 0x021a10 (0xfb49 bytes) 000336 1769074854.955585 write 0x005980 thru 0x010a77 (0xb0f8 bytes) 000337 1769074854.972564 mapwrite 0x0274d4 thru 0x031f4e (0xaa7b bytes) 000338 1769074855.052213 punch from 0x021f35 to 0x02a926 (0x89f2 bytes) 000339 1769074855.066028 punch from 0x0392ec to 0x03b0fd (0x1e12 bytes) 000340 1769074855.083725 mapread 0x03878f thru 0x03ffff (0x7871 bytes) 000341 1769074855.099614 punch from 0x015c0a to 0x017ef6 (0x22ed bytes) 000342 1769074855.118319 write 0x021809 thru 0x025d53 (0x454b bytes) 000343 1769074855.132549 mapread 0x03b378 thru 0x03ffff (0x4c88 bytes) 000344 1769074855.133455 fallocate 0x03cef1 thru 0x03ffff (0x310f bytes) 000345 1769074855.155873 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xad8e32b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8dcd5f39 Mirror 3: 0x8dcd5f39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000346 1769074856.980914 mapwrite 0x02dc7b thru 0x02ef3c (0x12c2 bytes) 000347 1769074857.164420 write 0x00a146 thru 0x017a89 (0xd944 bytes) 000348 1769074857.230385 read 0x02720a thru 0x030eb8 (0x9caf bytes) 000349 1769074857.233606 mapwrite 0x02e17e thru 0x02f8fb (0x177e bytes) 000350 1769074857.269503 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xad8e32b (stale) Mirror 3: 0x8dcd5f39 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8de3a362 Mirror 4: 0x8de3a362 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000351 1769074858.267396 mapwrite 0x0203ca thru 0x0267c6 (0x63fd bytes) 000352 1769074858.384246 mapwrite 0x031a98 thru 0x033a74 (0x1fdd bytes) 000355 1769074858.432382 fallocate 0x00f299 thru 0x01c3a5 (0xd10d bytes) 000356 1769074858.439915 write 0x009423 thru 0x01329d (0x9e7b bytes) 000357 1769074858.474525 mapread 0x01f7ac thru 0x025a7e (0x62d3 bytes) 000358 1769074858.478592 punch from 0x03ae22 to 0x03ffff (0x51de bytes) 000359 1769074858.493430 mapwrite 0x03a1c0 thru 0x03ffff (0x5e40 bytes) 000360 1769074858.557683 write 0x027947 thru 0x02d33e (0x59f8 bytes) 000361 1769074858.565252 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xad8e32b (stale) Mirror 3: 0x8dcd5f39 (stale) Mirror 4: 0x8de3a362 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8963e006 Mirror 5: 0x8963e006 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000362 1769074859.815664 read 0x03a47c thru 0x03fa48 (0x55cd bytes) 000363 1769074859.832641 write 0x0378e2 thru 0x0380c6 (0x7e5 bytes) 000364 1769074859.962405 MIRROR_RESYNC 5 mirrors 000365 1769074861.685428 MIRROR_RESYNC 5 mirrors 000366 1769074862.053719 trunc from 0x040000 to 0x00c5f2 (0x33a0d bytes) 000367 1769074862.109956 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xc5f3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc5f3): Mirror 2: 0xfbf5cd85 (stale) Mirror 3: 0xfbf5cd85 (stale) Mirror 4: 0xfbf5cd85 (stale) Mirror 5: 0xfbf5cd85 (stale) CRC-32 checksum value for chunk [0, 0xc5f3): Mirror 1: 0xfbf5cd85 Mirror 6: 0xfbf5cd85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc5f3: skipped 000368 1769074863.229562 punch from 0x008cca to 0x00c5f2 (0x3929 bytes) 000369 1769074863.337262 trunc from 0x00c5f3 to 0x032738 (0x26146 bytes) 000371 1769074863.465199 mapwrite 0x028b16 thru 0x02a536 (0x1a21 bytes) 000372 1769074863.551009 write 0x02de84 thru 0x03b5d6 (0xd753 bytes) 000373 1769074863.557879 write 0x03b4e4 thru 0x03ffff (0x4b1c bytes) 000374 1769074863.559878 trunc from 0x027645 to 0x02d020 (0x59dc bytes) 000375 1769074863.581061 mapread 0x015a6e thru 0x01a8d3 (0x4e66 bytes) 000376 1769074863.593377 write 0x004586 thru 0x010a55 (0xc4d0 bytes) 000377 1769074863.607891 fallocate 0x037db0 thru 0x03ffff (0x8250 bytes) 000378 1769074863.633558 read 0x0277da thru 0x02fac2 (0x82e9 bytes) 000379 1769074863.649309 punch from 0x0108bd to 0x01f7aa (0xeeee bytes) 000380 1769074863.680850 write 0x019ba1 thru 0x02294e (0x8dae bytes) 000381 1769074863.706833 mapwrite 0x02a66e thru 0x035512 (0xaea5 bytes) 000383 1769074863.757886 fallocate 0x014a34 thru 0x01e523 (0x9af0 bytes) 000384 1769074863.766059 mapwrite 0x02ff66 thru 0x033683 (0x371e bytes) 000385 1769074863.787099 punch from 0x014d35 to 0x01dbe9 (0x8eb5 bytes) 000387 1769074863.796853 fallocate 0x009335 thru 0x00a319 (0xfe5 bytes) 000388 1769074863.805656 read 0x02b6d8 thru 0x02f71f (0x4048 bytes) 000389 1769074863.807809 trunc from 0x01a39d to 0x020bc3 (0x6827 bytes) 000390 1769074863.818374 trunc from 0x040000 to 0x0219cd (0x1e632 bytes) 000391 1769074863.837358 punch from 0x013986 to 0x0201fb (0xc876 bytes) 000392 1769074863.843768 write 0x030a1d thru 0x037474 (0x6a58 bytes) 000393 1769074863.846590 fallocate 0x018467 thru 0x022564 (0xa0fe bytes) 000394 1769074863.851422 mapread 0x02c89f thru 0x031580 (0x4ce2 bytes) 000395 1769074863.860366 write 0x006300 thru 0x00a0a2 (0x3da3 bytes) 000396 1769074863.861527 write 0x01ae3b thru 0x01b604 (0x7ca bytes) 000397 1769074863.862845 fallocate 0x012227 thru 0x01abac (0x8986 bytes) 000398 1769074863.867387 mapread 0x00f8fe thru 0x01e5a9 (0xecac bytes) 000399 1769074863.868969 read 0x02bd71 thru 0x02f144 (0x33d4 bytes) 000400 1769074863.870130 trunc from 0x037475 to 0x03e410 (0x6f9c bytes) 000401 1769074863.888455 trunc from 0x00caf0 to 0x00e307 (0x1818 bytes) 000402 1769074863.896271 punch from 0x003a6a to 0x004c42 (0x11d9 bytes) 000403 1769074863.901563 fallocate 0x03c85d thru 0x03ffff (0x37a3 bytes) 000404 1769074863.911330 MIRROR_RESYNC 6 mirrors 000405 1769074866.278883 mapwrite 0x03ba84 thru 0x03bdd8 (0x355 bytes) 000406 1769074866.376568 write 0x02b55d thru 0x03af02 (0xf9a6 bytes) 000407 1769074866.397020 trunc from 0x0351f9 to 0x03ffff (0xae07 bytes) 000409 1769074866.441292 trunc from 0x02ff76 to 0x03710b (0x7196 bytes) 000410 1769074866.453479 read 0x00eebd thru 0x01b0d3 (0xc217 bytes) 000412 1769074866.482324 MIRROR_RESYNC 6 mirrors 000413 1769074868.836027 trunc from 0x012f2e to 0x015ce6 (0x2db9 bytes) 000414 1769074868.886457 mapread 0x02e3aa thru 0x032c9a (0x48f1 bytes) 000415 1769074868.925644 read 0x007569 thru 0x011000 (0x9a98 bytes) 000416 1769074868.938882 write 0x03749a thru 0x0382a9 (0xe10 bytes) 000417 1769074868.939854 punch from 0x006783 to 0x012dec (0xc66a bytes) 000418 1769074868.948420 write 0x00c0c1 thru 0x013fcb (0x7f0b bytes) 000419 1769074868.965669 punch from 0x002426 to 0x01038d (0xdf68 bytes) 000420 1769074869.042757 mapread 0x00d46d thru 0x01cf2d (0xfac1 bytes) 000422 1769074869.091916 read 0x0210ff thru 0x030fbb (0xfebd bytes) 000423 1769074869.126414 mapread 0x000636 thru 0x00c97e (0xc349 bytes) 000424 1769074869.130780 punch from 0x013710 to 0x0234af (0xfda0 bytes) 000425 1769074869.146435 trunc from 0x0361a0 to 0x03a507 (0x4368 bytes) 000426 1769074869.158905 mapread 0x00ad28 thru 0x015be5 (0xaebe bytes) 000427 1769074869.202599 trunc from 0x00b8ee to 0x01b74c (0xfe5f bytes) 000428 1769074869.216525 write 0x00b18d thru 0x00fa38 (0x48ac bytes) 000429 1769074869.254974 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xef8da21f (stale) Mirror 4: 0xef8da21f (stale) Mirror 5: 0xef8da21f (stale) Mirror 6: 0xef8da21f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xef8da21f (stale) Mirror 4: 0xef8da21f (stale) Mirror 5: 0xef8da21f (stale) Mirror 6: 0xef8da21f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000430 1769074869.792813 trunc from 0x00e475 to 0x01e363 (0xfeef bytes) 000433 1769074869.802016 punch from 0x02a10b to 0x02f776 (0x566c bytes) 000434 1769074869.807984 mapread 0x033a52 thru 0x038f01 (0x54b0 bytes) 000435 1769074869.823251 punch from 0x0059b4 to 0x010052 (0xa69f bytes) 000436 1769074869.826433 fallocate 0x009692 thru 0x01663e (0xcfad bytes) 000438 1769074869.830264 trunc from 0x0344c7 to 0x03ffff (0xbb39 bytes) 000439 1769074869.834086 punch from 0x00f235 to 0x01bc98 (0xca64 bytes) 000440 1769074869.837552 trunc from 0x01cbb8 to 0x027846 (0xac8f bytes) 000441 1769074869.841328 fallocate 0x0016aa thru 0x001cb1 (0x608 bytes) 000443 1769074869.845478 read 0x03ac60 thru 0x03ffff (0x53a0 bytes) 000444 1769074869.852144 read 0x01a54e thru 0x02980e (0xf2c1 bytes) 000445 1769074869.864410 write 0x03464e thru 0x03ffff (0xb9b2 bytes) 000446 1769074869.870269 trunc from 0x040000 to 0x024c54 (0x1b3ab bytes) 000447 1769074869.886184 read 0x010cf9 thru 0x01dc44 (0xcf4c bytes) 000448 1769074869.886963 punch from 0x00c44b to 0x01bdd4 (0xf98a bytes) 000450 1769074869.894339 read 0x00c146 thru 0x014f30 (0x8deb bytes) 000451 1769074869.908442 read 0x000539 thru 0x00baeb (0xb5b3 bytes) 000452 1769074869.909421 fallocate 0x03c6fc thru 0x03e6c4 (0x1fc9 bytes) 000453 1769074869.914497 punch from 0x015789 to 0x020567 (0xaddf bytes) 000455 1769074869.919231 fallocate 0x026145 thru 0x026f17 (0xdd3 bytes) 000456 1769074869.924072 mapread 0x019e31 thru 0x01d325 (0x34f5 bytes) 000457 1769074869.943084 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e6c5): Mirror 2: 0xa87c94fa (stale) Mirror 5: 0xa87c94fa (stale) Mirror 6: 0xa87c94fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6c5): Mirror 2: 0xa87c94fa (stale) Mirror 5: 0xa87c94fa (stale) Mirror 6: 0xa87c94fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6c5: skipped 000458 1769074870.256865 trunc from 0x03ccca to 0x03e6c4 (0x19fb bytes) 000459 1769074870.267446 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e6c5): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6c5): Mirror 2: 0xa87c94fa (stale) Mirror 5: 0xa87c94fa (stale) Mirror 6: 0xa87c94fa (stale) CRC-32 checksum value for chunk [0, 0x3e6c5): Mirror 1: 0x63046284 Mirror 7: 0x63046284 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6c5: skipped 000460 1769074871.380563 punch from 0x034824 to 0x03e6c4 (0x9ea1 bytes) 000461 1769074871.444864 read 0x01d2be thru 0x029ba3 (0xc8e6 bytes) 000462 1769074871.454979 trunc from 0x03e6c5 to 0x02b093 (0x13631 bytes) 000463 1769074871.483404 punch from 0x014c36 to 0x01b6ff (0x6aca bytes) 000464 1769074871.490265 mapwrite 0x02a5d7 thru 0x02ba00 (0x142a bytes) 000466 1769074871.531152 punch from 0x001138 to 0x006e59 (0x5d22 bytes) 000467 1769074871.539952 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ba01): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba01): Mirror 2: 0xbd2335bc (stale) Mirror 5: 0xbd2335bc (stale) Mirror 6: 0xbd2335bc (stale) Mirror 7: 0x5ec527a7 (stale) CRC-32 checksum value for chunk [0, 0x2ba01): Mirror 1: 0x223b088d Mirror 8: 0x223b088d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba01: skipped 000468 1769074872.325772 read 0x0102ea thru 0x012a12 (0x2729 bytes) 000469 1769074872.361725 write 0x020484 thru 0x02c4da (0xc057 bytes) 000470 1769074872.433486 fallocate 0x0374a4 thru 0x03ffff (0x8b5c bytes) 000471 1769074872.440791 trunc from 0x0035eb to 0x00a59f (0x6fb5 bytes) 000472 1769074872.447555 fallocate 0x0083db thru 0x014c15 (0xc83b bytes) 000473 1769074872.453811 trunc from 0x02c4db to 0x00907e (0x2345c bytes) 000474 1769074872.519477 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x907f): Mirror 2: 0xd4cd33a8 (stale) Mirror 5: 0xd4cd33a8 (stale) Mirror 6: 0xd4cd33a8 (stale) Mirror 8: 0x41fa17f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x907f): Mirror 2: 0xd4cd33a8 (stale) Mirror 5: 0xd4cd33a8 (stale) Mirror 6: 0xd4cd33a8 (stale) Mirror 8: 0x41fa17f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x907f: skipped 000475 1769074872.829971 trunc from 0x00907f to 0x03d81c (0x3479e bytes) 000476 1769074872.854107 trunc from 0x000aa7 to 0x0028db (0x1e35 bytes) 000477 1769074872.859171 write 0x03e9cb thru 0x03ffff (0x1635 bytes) 000478 1769074872.860517 write 0x0239fd thru 0x02f344 (0xb948 bytes) 000479 1769074872.869522 read 0x01fefb thru 0x028756 (0x885c bytes) 000480 1769074872.878612 MIRROR_RESYNC 5 mirrors 000481 1769074873.793282 write 0x03c625 thru 0x03e618 (0x1ff4 bytes) 000482 1769074873.840386 punch from 0x017ae4 to 0x01bab2 (0x3fcf bytes) 000483 1769074873.846142 trunc from 0x023f6c to 0x0305ad (0xc642 bytes) 000484 1769074873.855100 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbc150efb (stale) Mirror 5: 0xbc150efb (stale) Mirror 6: 0xbc150efb (stale) Mirror 8: 0xbc150efb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ea94a54 Mirror 9: 0x8ea94a54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000485 1769074874.588226 write 0x02a497 thru 0x039cbd (0xf827 bytes) 000486 1769074874.634230 trunc from 0x039181 to 0x03f788 (0x6608 bytes) 000487 1769074874.647000 fallocate 0x03488d thru 0x03b060 (0x67d4 bytes) 000488 1769074874.651472 trunc from 0x031c68 to 0x03962e (0x79c7 bytes) 000489 1769074874.657254 mapread 0x00efdf thru 0x00f856 (0x878 bytes) 000491 1769074874.668240 mapwrite 0x03b138 thru 0x03ffff (0x4ec8 bytes) 000492 1769074874.703266 trunc from 0x03db64 to 0x03fd44 (0x21e1 bytes) 000493 1769074874.709132 mapread 0x02c7ef thru 0x033112 (0x6924 bytes) 000494 1769074874.716384 mapwrite 0x010a10 thru 0x018d91 (0x8382 bytes) 000495 1769074874.750014 punch from 0x02f469 to 0x03518b (0x5d23 bytes) 000496 1769074874.756382 mapwrite 0x02ceaa thru 0x03aaa2 (0xdbf9 bytes) 000497 1769074874.781929 punch from 0x01501f to 0x022c85 (0xdc67 bytes) 000499 1769074874.789396 fallocate 0x03079e thru 0x038052 (0x78b5 bytes) 000500 1769074874.793830 fallocate 0x002fde thru 0x00512c (0x214f bytes) 000501 1769074874.799577 trunc from 0x040000 to 0x038c8a (0x7375 bytes) 000502 1769074874.816488 fallocate 0x03c2cc thru 0x03ffff (0x3d34 bytes) 000503 1769074874.821970 trunc from 0x040000 to 0x011afc (0x2e503 bytes) 000504 1769074874.838233 mapwrite 0x01b43a thru 0x01ced1 (0x1a98 bytes) 000505 1769074874.875456 write 0x01d8cc thru 0x02b300 (0xda35 bytes) 000506 1769074874.877168 write 0x03cd5d thru 0x03f585 (0x2829 bytes) 000507 1769074874.878152 mapread 0x00a194 thru 0x0164ee (0xc35b bytes) 000508 1769074874.890106 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f586): Mirror 2: 0xb74e9858 (stale) Mirror 5: 0xb74e9858 (stale) Mirror 6: 0xb74e9858 (stale) Mirror 8: 0xb74e9858 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f586): Mirror 2: 0xb74e9858 (stale) Mirror 5: 0xb74e9858 (stale) Mirror 6: 0xb74e9858 (stale) Mirror 8: 0xb74e9858 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f586: skipped 000509 1769074875.262525 punch from 0x013a2a to 0x01bb96 (0x816d bytes) 000510 1769074875.268115 mapread 0x003cf1 thru 0x007b96 (0x3ea6 bytes) 000511 1769074875.285338 read 0x002bca thru 0x011646 (0xea7d bytes) 000513 1769074875.298826 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f586): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f586): Mirror 2: 0xb74e9858 (stale) Mirror 5: 0xb74e9858 (stale) Mirror 6: 0xb74e9858 (stale) Mirror 8: 0xb74e9858 (stale) CRC-32 checksum value for chunk [0, 0x3f586): Mirror 1: 0xdf499c66 Mirror 9: 0xdf499c66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f586: skipped 000514 1769074876.124951 write 0x00e8be thru 0x01974a (0xae8d bytes) 000515 1769074876.196012 mapwrite 0x004557 thru 0x00d657 (0x9101 bytes) 000516 1769074876.226391 mapwrite 0x01d514 thru 0x028e75 (0xb962 bytes) 000517 1769074876.270225 MIRROR_RESYNC 6 mirrors 000518 1769074877.949006 mapwrite 0x02d27d thru 0x037af9 (0xa87d bytes) 000519 1769074878.167511 trunc from 0x03aaed to 0x03bd90 (0x12a4 bytes) 000520 1769074878.189441 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f586): Mirror 2: 0xd9daf15d (stale) Mirror 5: 0xd9daf15d (stale) Mirror 6: 0xd9daf15d (stale) Mirror 9: 0xd9daf15d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f586): Mirror 2: 0xd9daf15d (stale) Mirror 5: 0xd9daf15d (stale) Mirror 6: 0xd9daf15d (stale) Mirror 9: 0xd9daf15d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f586: skipped 000521 1769074878.522150 mapwrite 0x02f6f1 thru 0x03c37e (0xcc8e bytes) 000522 1769074878.576557 trunc from 0x00f395 to 0x01a9ec (0xb658 bytes) 000523 1769074878.586122 mapwrite 0x01ff4e thru 0x02a31c (0xa3cf bytes) 000524 1769074878.630394 MIRROR_RESYNC 5 mirrors 000525 1769074879.942723 mapread 0x017c1d thru 0x01ba81 (0x3e65 bytes) 000526 1769074879.971538 trunc from 0x03f586 to 0x032fe9 (0xc59c bytes) 000527 1769074880.029024 trunc from 0x01092a to 0x01f220 (0xe8f7 bytes) 000528 1769074880.033433 punch from 0x006fd7 to 0x011da8 (0xadd2 bytes) 000529 1769074880.043727 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32fea): Mirror 2: 0x538072e4 (stale) Mirror 5: 0x538072e4 (stale) Mirror 6: 0x538072e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32fea): Mirror 2: 0x538072e4 (stale) Mirror 5: 0x538072e4 (stale) Mirror 6: 0x538072e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32fea: skipped 000530 1769074880.426028 mapwrite 0x016aa2 thru 0x022c9b (0xc1fa bytes) 000531 1769074880.506523 mapwrite 0x029f0e thru 0x02fd0e (0x5e01 bytes) 000532 1769074880.592629 punch from 0x01aaa5 to 0x01c743 (0x1c9f bytes) 000533 1769074880.601993 read 0x030b6a thru 0x032fe9 (0x2480 bytes) 000534 1769074880.602936 punch from 0x0012ec to 0x0053f7 (0x410c bytes) 000535 1769074880.616560 fallocate 0x0296a8 thru 0x030f29 (0x7882 bytes) 000536 1769074880.623413 mapread 0x027097 thru 0x029586 (0x24f0 bytes) 000537 1769074880.631182 MIRROR_RESYNC 4 mirrors 000538 1769074881.559545 MIRROR_EXTEND to 5 mirrors 000539 1769074882.266957 mapread 0x006ba9 thru 0x00dc4b (0x70a3 bytes) 000541 1769074882.284086 write 0x03c670 thru 0x03ffff (0x3990 bytes) 000543 1769074882.324004 mapread 0x01948c thru 0x01a02c (0xba1 bytes) 000544 1769074882.332592 read 0x00a14c thru 0x013a2e (0x98e3 bytes) 000545 1769074882.341493 fallocate 0x00cbbe thru 0x015afa (0x8f3d bytes) 000546 1769074882.346269 trunc from 0x00d5d5 to 0x00f0bc (0x1ae8 bytes) 000547 1769074882.351229 mapread 0x011560 thru 0x015107 (0x3ba8 bytes) 000548 1769074882.352171 trunc from 0x00652b to 0x011d56 (0xb82c bytes) 000550 1769074882.358422 trunc from 0x03542c to 0x03ffff (0xabd4 bytes) 000551 1769074882.371503 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x17400aa1 (stale) Mirror 5: 0x17400aa1 (stale) Mirror 6: 0x17400aa1 (stale) Mirror 7: 0x17400aa1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa38d24a0 Mirror 8: 0xa38d24a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000553 1769074883.221761 fallocate 0x027779 thru 0x034161 (0xc9e9 bytes) 000554 1769074883.262178 mapwrite 0x02971b thru 0x0383c0 (0xeca6 bytes) 000555 1769074883.312920 trunc from 0x040000 to 0x03cbf6 (0x3409 bytes) 000557 1769074883.365431 read 0x033f23 thru 0x0346ee (0x7cc bytes) 000558 1769074883.367180 fallocate 0x025823 thru 0x027a92 (0x2270 bytes) 000559 1769074883.372395 fallocate 0x00a696 thru 0x00b7a3 (0x110e bytes) 000561 1769074883.377582 fallocate 0x02d559 thru 0x039af0 (0xc598 bytes) 000562 1769074883.386393 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 2: 0x4fb20ccf (stale) Mirror 5: 0x4fb20ccf (stale) Mirror 6: 0x4fb20ccf (stale) Mirror 7: 0x4fb20ccf (stale) Mirror 8: 0xb179aa7b (stale) CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 1: 0x9e885f85 Mirror 9: 0x9e885f85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbf7: skipped 000563 1769074884.130013 read 0x028621 thru 0x031a1c (0x93fc bytes) 000564 1769074884.146902 trunc from 0x0105f8 to 0x0175b1 (0x6fba bytes) 000565 1769074884.189332 mapread 0x02dfa4 thru 0x035d39 (0x7d96 bytes) 000566 1769074884.204596 trunc from 0x00e818 to 0x01619e (0x7987 bytes) 000567 1769074884.212375 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 2: 0x4fb20ccf (stale) Mirror 5: 0x4fb20ccf (stale) Mirror 6: 0x4fb20ccf (stale) Mirror 7: 0x4fb20ccf (stale) Mirror 8: 0xb179aa7b (stale) Mirror 9: 0x9e885f85 (stale) CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 1: 0x9a960edc Mirror 10: 0x9a960edc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbf7: skipped 000568 1769074885.056301 MIRROR_RESYNC 8 mirrors 000569 1769074886.641703 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) 000570 1769074887.011981 fallocate 0x022d24 thru 0x025c75 (0x2f52 bytes) 000571 1769074887.060240 fallocate 0x01a22e thru 0x01f0e8 (0x4ebb bytes) 000572 1769074887.070382 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 2: 0x9a960edc (stale) Mirror 5: 0x9a960edc (stale) Mirror 6: 0x9a960edc (stale) Mirror 7: 0x9a960edc (stale) Mirror 8: 0x9a960edc (stale) Mirror 9: 0x9a960edc (stale) CRC-32 checksum value for chunk [0, 0x3cbf7): Mirror 1: 0x9a960edc Mirror 10: 0x9a960edc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbf7: skipped 000573 1769074887.911521 read 0x033a68 thru 0x03cbf6 (0x918f bytes) 000575 1769074887.934920 trunc from 0x03cbf7 to 0x0163e4 (0x26812 bytes) 000576 1769074887.997203 mapread 0x00df07 thru 0x012a15 (0x4b0f bytes) 000577 1769074888.025982 fallocate 0x02e2cb thru 0x030550 (0x2286 bytes) 000578 1769074888.037697 mapread 0x00eedf thru 0x015a94 (0x6bb6 bytes) 000579 1769074888.038622 mapread 0x013cb1 thru 0x015085 (0x13d5 bytes) 000580 1769074888.039364 trunc from 0x0133b0 to 0x0163e4 (0x3035 bytes) 000581 1769074888.046748 trunc from 0x0163e5 to 0x00b58f (0xae55 bytes) 000582 1769074888.084501 mapread 0x005553 thru 0x00b58f (0x603d bytes) 000583 1769074888.106591 write 0x01b56c thru 0x01bb64 (0x5f9 bytes) 000584 1769074888.107624 fallocate 0x029a98 thru 0x02c305 (0x286e bytes) 000585 1769074888.117459 mapwrite 0x00ae87 thru 0x013f8a (0x9104 bytes) 000586 1769074888.176209 punch from 0x011cbc to 0x015eac (0x41f1 bytes) 000587 1769074888.195113 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2c306): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c306): Mirror 2: 0xdfa1996a (stale) Mirror 5: 0xdfa1996a (stale) Mirror 6: 0xdfa1996a (stale) Mirror 7: 0xdfa1996a (stale) Mirror 8: 0xdfa1996a (stale) Mirror 9: 0xdfa1996a (stale) Mirror 10: 0xdfa1996a (stale) CRC-32 checksum value for chunk [0, 0x2c306): Mirror 1: 0x5162a78f Mirror 11: 0x5162a78f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c306: skipped 000588 1769074889.540084 mapwrite 0x0340bd thru 0x0373f4 (0x3338 bytes) 000589 1769074889.738662 mapread 0x011607 thru 0x01eda3 (0xd79d bytes) 000590 1769074889.781747 read 0x00151d thru 0x0040e6 (0x2bca bytes) 000591 1769074889.815690 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x373f5): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x373f5): Mirror 2: 0x98601eb (stale) Mirror 5: 0x98601eb (stale) Mirror 6: 0x98601eb (stale) Mirror 7: 0x98601eb (stale) Mirror 8: 0x98601eb (stale) Mirror 9: 0x98601eb (stale) Mirror 10: 0x98601eb (stale) Mirror 11: 0x36791a5e (stale) CRC-32 checksum value for chunk [0, 0x373f5): Mirror 1: 0xa8be8425 Mirror 12: 0xa8be8425 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x373f5: skipped 000592 1769074890.795171 trunc from 0x0373f5 to 0x02c017 (0xb3dd bytes) 000593 1769074890.847286 mapread 0x008148 thru 0x00fbae (0x7a67 bytes) 000594 1769074890.865328 trunc from 0x02c018 to 0x03f95b (0x13944 bytes) 000595 1769074890.882491 trunc from 0x00a3ae to 0x01a2e0 (0xff33 bytes) 000596 1769074890.888250 trunc from 0x0156f7 to 0x0191e2 (0x3aec bytes) 000597 1769074890.891943 read 0x01252c thru 0x0183e9 (0x5ebe bytes) 000598 1769074890.899527 punch from 0x00a0e7 to 0x013716 (0x9630 bytes) 000599 1769074890.903949 read 0x0134fa thru 0x0205a8 (0xd0af bytes) 000600 1769074890.914071 mapwrite 0x0072b9 thru 0x014b58 (0xd8a0 bytes) 000601 1769074890.932867 mapwrite 0x006297 thru 0x00e65c (0x83c6 bytes) 000603 1769074890.957877 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f95c): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f95c): Mirror 2: 0x3c2f698a (stale) Mirror 5: 0x3c2f698a (stale) Mirror 6: 0x3c2f698a (stale) Mirror 7: 0x3c2f698a (stale) Mirror 8: 0x3c2f698a (stale) Mirror 9: 0x3c2f698a (stale) Mirror 10: 0x3c2f698a (stale) Mirror 11: 0x935d79e5 (stale) Mirror 12: 0x25531f5d (stale) CRC-32 checksum value for chunk [0, 0x3f95c): Mirror 1: 0x3c8e2d9c Mirror 13: 0x3c8e2d9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f95c: skipped 000605 1769074891.956619 read 0x01693d thru 0x023111 (0xc7d5 bytes) 000606 1769074891.984391 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f95c): Mirror 2: 0x3c2f698a (stale) Mirror 5: 0x3c2f698a (stale) Mirror 7: 0x3c2f698a (stale) Mirror 8: 0x3c2f698a (stale) Mirror 9: 0x3c2f698a (stale) Mirror 10: 0x3c2f698a (stale) Mirror 11: 0x935d79e5 (stale) Mirror 12: 0x25531f5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f95c): Mirror 2: 0x3c2f698a (stale) Mirror 5: 0x3c2f698a (stale) Mirror 7: 0x3c2f698a (stale) Mirror 8: 0x3c2f698a (stale) Mirror 9: 0x3c2f698a (stale) Mirror 10: 0x3c2f698a (stale) Mirror 11: 0x935d79e5 (stale) Mirror 12: 0x25531f5d (stale) CRC-32 checksum value for chunk [0, 0x3f95c): Mirror 1: 0x3c8e2d9c Mirror 13: 0x3c8e2d9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f95c: skipped 000607 1769074892.797530 read 0x012338 thru 0x0202de (0xdfa7 bytes) 000608 1769074892.807399 mapread 0x00f101 thru 0x015540 (0x6440 bytes) 000609 1769074892.832345 read 0x00a961 thru 0x0193ef (0xea8f bytes) 000610 1769074892.833379 write 0x03f51d thru 0x03ffff (0xae3 bytes) 000611 1769074892.897591 trunc from 0x02d294 to 0x034616 (0x7383 bytes) 000612 1769074892.903686 mapwrite 0x037017 thru 0x03ea24 (0x7a0e bytes) 000613 1769074892.930294 trunc from 0x040000 to 0x0319c4 (0xe63b bytes) 000614 1769074892.953316 mapwrite 0x00399d thru 0x008bd1 (0x5235 bytes) 000615 1769074892.971871 mapread 0x010e68 thru 0x01f89e (0xea37 bytes) 000616 1769074892.973013 read 0x008df2 thru 0x018d5d (0xff6c bytes) 000617 1769074892.974277 punch from 0x0169a3 to 0x022a72 (0xc0d0 bytes) 000618 1769074892.979206 fallocate 0x03d017 thru 0x03ffff (0x2fe9 bytes) 000619 1769074892.983360 mapwrite 0x015311 thru 0x01b3df (0x60cf bytes) 000620 1769074893.025368 write 0x00efb6 thru 0x01bbe1 (0xcc2c bytes) 000621 1769074893.031719 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x760318b9 (stale) Mirror 5: 0x760318b9 (stale) Mirror 7: 0x760318b9 (stale) Mirror 9: 0x760318b9 (stale) Mirror 10: 0x760318b9 (stale) Mirror 11: 0xc1bb7222 (stale) Mirror 12: 0xd4df99db (stale) Mirror 13: 0x1057e6d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x760318b9 (stale) Mirror 5: 0x760318b9 (stale) Mirror 7: 0x760318b9 (stale) Mirror 9: 0x760318b9 (stale) Mirror 10: 0x760318b9 (stale) Mirror 11: 0xc1bb7222 (stale) Mirror 12: 0xd4df99db (stale) Mirror 13: 0x1057e6d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000622 1769074893.575181 mapread 0x00e851 thru 0x01806c (0x981c bytes) 000623 1769074893.593421 fallocate 0x00e859 thru 0x016730 (0x7ed8 bytes) 000624 1769074893.598328 trunc from 0x040000 to 0x00106c (0x3ef93 bytes) 000625 1769074893.623099 trunc from 0x000cdc to 0x00106c (0x391 bytes) 000626 1769074893.630310 trunc from 0x00106d to 0x026897 (0x2582b bytes) 000628 1769074893.650387 read 0x00afb1 thru 0x00e62f (0x367f bytes) 000629 1769074893.662895 mapwrite 0x002d89 thru 0x00d45c (0xa6d4 bytes) 000630 1769074893.691963 mapread 0x00156b thru 0x004c76 (0x370c bytes) 000631 1769074893.693283 fallocate 0x00bb25 thru 0x00e514 (0x29f0 bytes) 000632 1769074893.698244 trunc from 0x026898 to 0x01f5a4 (0x72f3 bytes) 000633 1769074893.720738 fallocate 0x02d0d1 thru 0x03a24a (0xd17a bytes) 000634 1769074893.726493 fallocate 0x000bff thru 0x010066 (0xf468 bytes) 000635 1769074893.732203 trunc from 0x03a24b to 0x003e38 (0x36412 bytes) 000636 1769074893.754159 trunc from 0x001dfb to 0x003e38 (0x203e bytes) 000637 1769074893.760121 mapwrite 0x00ab3c thru 0x01001a (0x54df bytes) 000638 1769074893.804979 punch from 0x00eff9 to 0x01001a (0x1022 bytes) 000640 1769074893.811355 trunc from 0x01001b to 0x03e8d2 (0x2e8b8 bytes) 000641 1769074893.829280 read 0x035c39 thru 0x03e8d2 (0x8c9a bytes) 000642 1769074893.840896 trunc from 0x03e8d3 to 0x01b285 (0x2364d bytes) 000643 1769074893.863429 mapwrite 0x005d98 thru 0x011075 (0xb2de bytes) 000644 1769074893.881527 trunc from 0x004646 to 0x00d3f2 (0x8dad bytes) 000645 1769074893.887388 read 0x013bfa thru 0x018903 (0x4d0a bytes) 000646 1769074893.888361 write 0x01ed3e thru 0x021c7c (0x2f3f bytes) 000647 1769074893.889649 read 0x00763b thru 0x00d537 (0x5efd bytes) 000648 1769074893.898454 trunc from 0x01e23e to 0x021c7c (0x3a3f bytes) 000650 1769074893.909451 mapread 0x01ce32 thru 0x021c7c (0x4e4b bytes) 000652 1769074893.932704 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21c7d): Mirror 2: 0x778e7f12 (stale) Mirror 5: 0x778e7f12 (stale) Mirror 7: 0x778e7f12 (stale) Mirror 10: 0x778e7f12 (stale) Mirror 11: 0x36868b12 (stale) Mirror 12: 0x36868b12 (stale) Mirror 13: 0x20b053f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c7d): Mirror 2: 0x778e7f12 (stale) Mirror 5: 0x778e7f12 (stale) Mirror 7: 0x778e7f12 (stale) Mirror 10: 0x778e7f12 (stale) Mirror 11: 0x36868b12 (stale) Mirror 12: 0x36868b12 (stale) Mirror 13: 0x20b053f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c7d: skipped 000653 1769074894.419677 trunc from 0x021c7d to 0x010140 (0x11b3c bytes) 000654 1769074894.445440 punch from 0x0016fa to 0x00d434 (0xbd3b bytes) 000655 1769074894.465160 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10141): Mirror 2: 0x77e77544 (stale) Mirror 5: 0x77e77544 (stale) Mirror 10: 0x77e77544 (stale) Mirror 11: 0x2f46fda3 (stale) Mirror 12: 0x2f46fda3 (stale) Mirror 13: 0xc8717f52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10141): Mirror 2: 0x77e77544 (stale) Mirror 5: 0x77e77544 (stale) Mirror 10: 0x77e77544 (stale) Mirror 11: 0x2f46fda3 (stale) Mirror 12: 0x2f46fda3 (stale) Mirror 13: 0xc8717f52 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10141: skipped 000656 1769074895.015988 read 0x003c63 thru 0x00ba19 (0x7db7 bytes) 000657 1769074895.028634 mapwrite 0x00330d thru 0x00876b (0x545f bytes) 000658 1769074895.046201 read 0x0035d3 thru 0x009132 (0x5b60 bytes) 000659 1769074895.047265 punch from 0x00fb1e to 0x010140 (0x623 bytes) 000660 1769074895.052911 read 0x00ef41 thru 0x010140 (0x1200 bytes) 000661 1769074895.061137 mapwrite 0x00e80f thru 0x01e7df (0xffd1 bytes) 000662 1769074895.113114 fallocate 0x03bc01 thru 0x03ffff (0x43ff bytes) 000663 1769074895.118506 read 0x02a1ce thru 0x036b83 (0xc9b6 bytes) 000664 1769074895.127713 trunc from 0x000c52 to 0x00a899 (0x9c48 bytes) 000665 1769074895.133180 write 0x026f81 thru 0x0270c7 (0x147 bytes) 000666 1769074895.134120 mapwrite 0x010550 thru 0x0201cb (0xfc7c bytes) 000668 1769074895.154739 punch from 0x0296d7 to 0x034dbb (0xb6e5 bytes) 000669 1769074895.165953 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x760318b9 (stale) Mirror 5: 0x760318b9 (stale) Mirror 10: 0x760318b9 (stale) Mirror 11: 0xc1bb7222 (stale) Mirror 12: 0xd4df99db (stale) Mirror 13: 0x1057e6d6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7040e91f Mirror 14: 0x7040e91f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000670 1769074896.352354 write 0x0035ea thru 0x003d06 (0x71d bytes) 000671 1769074896.450588 read 0x02573c thru 0x033a4f (0xe314 bytes) 000672 1769074896.485435 read 0x00fcd2 thru 0x01bac5 (0xbdf4 bytes) 000674 1769074896.518642 trunc from 0x020e3e to 0x02bd97 (0xaf5a bytes) 000675 1769074896.527078 punch from 0x03c267 to 0x03ffff (0x3d99 bytes) 000676 1769074896.531501 trunc from 0x039bac to 0x03adb4 (0x1209 bytes) 000677 1769074896.554829 mapread 0x01b8fe thru 0x01d38f (0x1a92 bytes) 000678 1769074896.555730 read 0x01213e thru 0x015b4c (0x3a0f bytes) 000680 1769074896.556739 mapread 0x03cba1 thru 0x03ffff (0x345f bytes) 000681 1769074896.592803 read 0x038bfb thru 0x03ffff (0x7405 bytes) 000682 1769074896.593429 write 0x02bf19 thru 0x02e237 (0x231f bytes) 000683 1769074896.598983 mapwrite 0x0003a4 thru 0x0052f0 (0x4f4d bytes) 000684 1769074896.647723 fallocate 0x016947 thru 0x021411 (0xaacb bytes) 000685 1769074896.666185 punch from 0x03965a to 0x03b014 (0x19bb bytes) 000686 1769074896.681931 mapread 0x0172df thru 0x023917 (0xc639 bytes) 000687 1769074896.683012 trunc from 0x018597 to 0x028244 (0xfcae bytes) 000688 1769074896.702963 mapread 0x02cfe1 thru 0x02d856 (0x876 bytes) 000689 1769074896.703926 read 0x00b053 thru 0x0193f6 (0xe3a4 bytes) 000690 1769074896.726407 write 0x00a997 thru 0x00b965 (0xfcf bytes) 000691 1769074896.734612 trunc from 0x040000 to 0x019020 (0x26fdf bytes) 000692 1769074896.858737 fallocate 0x023689 thru 0x026c64 (0x35dc bytes) 000694 1769074896.884619 mapwrite 0x008e99 thru 0x009a5c (0xbc4 bytes) 000695 1769074896.932115 mapread 0x00be47 thru 0x0105ea (0x47a4 bytes) 000696 1769074896.935357 trunc from 0x019021 to 0x023a25 (0xaa05 bytes) 000698 1769074896.971250 mapread 0x0003ba thru 0x00aea1 (0xaae8 bytes) 000699 1769074896.972226 fallocate 0x02c84f thru 0x0303b9 (0x3b6b bytes) 000700 1769074896.986568 mapread 0x0004ab thru 0x00cfab (0xcb01 bytes) 000701 1769074896.991257 write 0x02d018 thru 0x03a531 (0xd51a bytes) 000702 1769074897.003384 trunc from 0x02a7fd to 0x02e6fc (0x3f00 bytes) 000703 1769074897.036537 trunc from 0x03140d to 0x037c24 (0x6818 bytes) 000704 1769074897.050492 MIRROR_RESYNC 8 mirrors 000705 1769074899.296997 trunc from 0x03a532 to 0x0155e4 (0x24f4d bytes) 000706 1769074899.352969 trunc from 0x011fa4 to 0x0155e4 (0x3641 bytes) 000707 1769074899.358264 trunc from 0x010276 to 0x0155e4 (0x536f bytes) 000708 1769074899.363589 mapread 0x00c670 thru 0x00dd49 (0x16da bytes) 000709 1769074899.377407 read 0x0152aa thru 0x0155e4 (0x33b bytes) 000710 1769074899.383328 mapwrite 0x012b41 thru 0x019378 (0x6838 bytes) 000712 1769074899.425080 mapwrite 0x036ec8 thru 0x03ffff (0x9138 bytes) 000713 1769074899.469313 MIRROR_RESYNC 8 mirrors 000714 1769074901.044660 trunc from 0x03dcb6 to 0x03ffff (0x234a bytes) 000715 1769074901.100697 mapread 0x0135bc thru 0x0139e4 (0x429 bytes) 000716 1769074901.108107 trunc from 0x040000 to 0x020f8f (0x1f070 bytes) 000717 1769074901.144417 mapwrite 0x036bd0 thru 0x03a887 (0x3cb8 bytes) 000718 1769074901.213772 punch from 0x010dce to 0x01c623 (0xb856 bytes) 000719 1769074901.222640 write 0x039e81 thru 0x03ffff (0x617f bytes) 000720 1769074901.248867 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3a33d43c (stale) Mirror 5: 0x3a33d43c (stale) Mirror 10: 0x3a33d43c (stale) Mirror 11: 0x3a33d43c (stale) Mirror 12: 0x3a33d43c (stale) Mirror 13: 0x3a33d43c (stale) Mirror 14: 0x3a33d43c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x68791f4c Mirror 15: 0x68791f4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000722 1769074902.277620 write 0x01b8ec thru 0x022d13 (0x7428 bytes) 000723 1769074902.344148 punch from 0x035947 to 0x03ffff (0xa6b9 bytes) 000724 1769074902.349650 read 0x008576 thru 0x013927 (0xb3b2 bytes) 000725 1769074902.359451 read 0x00684a thru 0x00cfb5 (0x676c bytes) 000726 1769074902.374838 punch from 0x032602 to 0x03ffff (0xd9fe bytes) 000727 1769074902.381731 punch from 0x039a9d to 0x03ffff (0x6563 bytes) 000729 1769074902.389217 fallocate 0x02cda5 thru 0x038c64 (0xbec0 bytes) 000730 1769074902.398611 fallocate 0x02d5d7 thru 0x035411 (0x7e3b bytes) 000731 1769074902.421188 MIRROR_RESYNC 9 mirrors 000732 1769074905.497876 mapwrite 0x0380e4 thru 0x03ffff (0x7f1c bytes) 000733 1769074905.582440 write 0x02539f thru 0x0301ad (0xae0f bytes) 000735 1769074905.596460 punch from 0x003f18 to 0x011022 (0xd10b bytes) 000737 1769074905.601453 read 0x022822 thru 0x030d99 (0xe578 bytes) 000738 1769074905.603068 write 0x038271 thru 0x03ffff (0x7d8f bytes) 000739 1769074905.608505 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x56b0f0e5 (stale) Mirror 10: 0x56b0f0e5 (stale) Mirror 11: 0x56b0f0e5 (stale) Mirror 12: 0x56b0f0e5 (stale) Mirror 13: 0x56b0f0e5 (stale) Mirror 14: 0x56b0f0e5 (stale) Mirror 15: 0x56b0f0e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x56b0f0e5 (stale) Mirror 10: 0x56b0f0e5 (stale) Mirror 11: 0x56b0f0e5 (stale) Mirror 12: 0x56b0f0e5 (stale) Mirror 13: 0x56b0f0e5 (stale) Mirror 14: 0x56b0f0e5 (stale) Mirror 15: 0x56b0f0e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1769074906.026720 punch from 0x034f73 to 0x03e04b (0x90d9 bytes) 000741 1769074906.033913 fallocate 0x03adf0 thru 0x03ed0e (0x3f1f bytes) 000742 1769074906.039844 mapread 0x021f93 thru 0x030d29 (0xed97 bytes) 000743 1769074906.068517 trunc from 0x040000 to 0x021a5a (0x1e5a5 bytes) 000744 1769074906.093095 fallocate 0x020a40 thru 0x02d73e (0xccff bytes) 000745 1769074906.098519 mapread 0x00ef3f thru 0x015af0 (0x6bb2 bytes) 000746 1769074906.108889 trunc from 0x021a5b to 0x01186c (0x101ee bytes) 000747 1769074906.129431 read 0x00d72c thru 0x01186c (0x4141 bytes) 000748 1769074906.131997 punch from 0x009e81 to 0x01186c (0x79ec bytes) 000749 1769074906.138556 mapwrite 0x0270e7 thru 0x03011b (0x9035 bytes) 000750 1769074906.208423 trunc from 0x01dabb to 0x02baab (0xdff1 bytes) 000751 1769074906.214438 read 0x0036d4 thru 0x0071cc (0x3af9 bytes) 000752 1769074906.230359 trunc from 0x01193a to 0x020ac9 (0xf190 bytes) 000753 1769074906.241303 read 0x0169dd thru 0x024576 (0xdb9a bytes) 000754 1769074906.260063 mapread 0x009de8 thru 0x00eb4a (0x4d63 bytes) 000756 1769074906.264972 trunc from 0x02bcd6 to 0x03011b (0x4446 bytes) 000757 1769074906.273142 write 0x029066 thru 0x02f4f6 (0x6491 bytes) 000758 1769074906.280561 read 0x00d3b6 thru 0x01d114 (0xfd5f bytes) 000759 1769074906.281227 write 0x006c22 thru 0x010421 (0x9800 bytes) 000760 1769074906.282356 mapread 0x024b41 thru 0x03011b (0xb5db bytes) 000761 1769074906.298479 read 0x020759 thru 0x0207e5 (0x8d bytes) 000762 1769074906.299510 mapwrite 0x03f15b thru 0x03ffff (0xea5 bytes) 000763 1769074906.367829 mapread 0x028fdd thru 0x02f1b2 (0x61d6 bytes) 000764 1769074906.370998 mapwrite 0x01d8ce thru 0x02ac2c (0xd35f bytes) 000765 1769074906.419775 trunc from 0x040000 to 0x03d108 (0x2ef7 bytes) 000767 1769074906.444946 write 0x0321f9 thru 0x03aac2 (0x88ca bytes) 000768 1769074906.446181 write 0x003250 thru 0x00f6fd (0xc4ae bytes) 000769 1769074906.447389 mapwrite 0x03873a thru 0x03d8c4 (0x518b bytes) 000770 1769074906.505451 trunc from 0x022240 to 0x02fbf2 (0xd9b3 bytes) 000771 1769074906.512164 mapread 0x000d38 thru 0x00ee13 (0xe0dc bytes) 000772 1769074906.512847 mapread 0x0115c0 thru 0x01ff67 (0xe9a8 bytes) 000773 1769074906.513602 mapwrite 0x03ec02 thru 0x03f83d (0xc3c bytes) 000775 1769074906.560968 trunc from 0x03f58f to 0x03f83d (0x2af bytes) 000776 1769074906.566429 trunc from 0x03f83e to 0x01693f (0x28efe bytes) 000777 1769074906.602605 trunc from 0x016940 to 0x02c9ff (0x160c0 bytes) 000778 1769074906.629191 mapwrite 0x030b68 thru 0x03159f (0xa38 bytes) 000779 1769074906.689274 write 0x013bc3 thru 0x0149f7 (0xe35 bytes) 000780 1769074906.692317 write 0x006a95 thru 0x00c479 (0x59e5 bytes) 000781 1769074906.696126 read 0x01f67a thru 0x02eac5 (0xf44c bytes) 000782 1769074906.710287 punch from 0x018cea to 0x026bce (0xdee5 bytes) 000783 1769074906.720348 MIRROR_RESYNC 8 mirrors 000784 1769074908.129180 trunc from 0x01bc9f to 0x02b1d1 (0xf533 bytes) 000785 1769074908.201484 write 0x0369ad thru 0x037e70 (0x14c4 bytes) 000786 1769074908.202472 trunc from 0x006c87 to 0x0104b1 (0x982b bytes) 000787 1769074908.212579 write 0x034e29 thru 0x03af10 (0x60e8 bytes) 000788 1769074908.218293 read 0x007485 thru 0x00b60a (0x4186 bytes) 000789 1769074908.247404 trunc from 0x0294fd to 0x032188 (0x8c8c bytes) 000790 1769074908.255308 trunc from 0x03af11 to 0x018299 (0x22c77 bytes) 000791 1769074908.318392 fallocate 0x03c03c thru 0x03ffff (0x3fc4 bytes) 000792 1769074908.328018 trunc from 0x015b76 to 0x018299 (0x2724 bytes) 000793 1769074908.341424 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1829a): Mirror 5: 0xffe16390 (stale) Mirror 10: 0xffe16390 (stale) Mirror 11: 0xffe16390 (stale) Mirror 13: 0xffe16390 (stale) Mirror 14: 0xffe16390 (stale) Mirror 15: 0xffe16390 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1829a): Mirror 5: 0xffe16390 (stale) Mirror 10: 0xffe16390 (stale) Mirror 11: 0xffe16390 (stale) Mirror 13: 0xffe16390 (stale) Mirror 14: 0xffe16390 (stale) Mirror 15: 0xffe16390 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1829a: skipped 000794 1769074908.772226 mapwrite 0x0254cf thru 0x02a121 (0x4c53 bytes) 000795 1769074908.839681 trunc from 0x011161 to 0x019765 (0x8605 bytes) 000796 1769074908.846116 mapwrite 0x01815e thru 0x02579d (0xd640 bytes) 000797 1769074908.881826 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a122): Mirror 5: 0x2348bded (stale) Mirror 10: 0x2348bded (stale) Mirror 13: 0x2348bded (stale) Mirror 14: 0x2348bded (stale) Mirror 15: 0x2348bded (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a122): Mirror 5: 0x2348bded (stale) Mirror 10: 0x2348bded (stale) Mirror 13: 0x2348bded (stale) Mirror 14: 0x2348bded (stale) Mirror 15: 0x2348bded (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a122: skipped 000798 1769074909.316139 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2a122): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a122): Mirror 5: 0x2348bded (stale) Mirror 10: 0x2348bded (stale) Mirror 13: 0x2348bded (stale) Mirror 14: 0x2348bded (stale) Mirror 15: 0x2348bded (stale) CRC-32 checksum value for chunk [0, 0x2a122): Mirror 1: 0x5f96ea55 Mirror 16: 0x5f96ea55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a122: skipped 000799 1769074910.622362 write 0x00f6d0 thru 0x01588d (0x61be bytes) 000800 1769074910.701143 trunc from 0x02a122 to 0x010481 (0x19ca0 bytes) 000801 1769074910.738861 fallocate 0x01888f thru 0x01eb84 (0x62f6 bytes) 000802 1769074910.744074 trunc from 0x010482 to 0x01b500 (0xb07f bytes) 000804 1769074910.758274 trunc from 0x015756 to 0x016d3c (0x15e7 bytes) 000805 1769074910.763616 fallocate 0x005594 thru 0x00805b (0x2ac8 bytes) 000806 1769074910.768245 write 0x029b5f thru 0x02c2f3 (0x2795 bytes) 000808 1769074910.774861 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c2f4): Mirror 5: 0x51dfaf48 (stale) Mirror 10: 0x51dfaf48 (stale) Mirror 13: 0x51dfaf48 (stale) Mirror 15: 0x51dfaf48 (stale) Mirror 16: 0xf2d1f59a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c2f4): Mirror 5: 0x51dfaf48 (stale) Mirror 10: 0x51dfaf48 (stale) Mirror 13: 0x51dfaf48 (stale) Mirror 15: 0x51dfaf48 (stale) Mirror 16: 0xf2d1f59a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c2f4: skipped 000809 1769074911.238175 mapwrite 0x03b075 thru 0x03e1a8 (0x3134 bytes) 000810 1769074911.306345 MIRROR_RESYNC 6 mirrors 000811 1769074913.269698 fallocate 0x02cd20 thru 0x039ee4 (0xd1c5 bytes) 000812 1769074913.333100 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e1a9): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1a9): Mirror 5: 0x6f7265ec (stale) Mirror 10: 0x6f7265ec (stale) Mirror 13: 0x6f7265ec (stale) Mirror 15: 0x6f7265ec (stale) Mirror 16: 0x6f7265ec (stale) CRC-32 checksum value for chunk [0, 0x3e1a9): Mirror 1: 0x6f7265ec Mirror 17: 0x6f7265ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1a9: skipped 000813 1769074914.670646 read 0x01ccb9 thru 0x022e20 (0x6168 bytes) 000814 1769074914.721217 punch from 0x004caa to 0x00ba64 (0x6dbb bytes) 000815 1769074914.788338 read 0x023895 thru 0x0331ac (0xf918 bytes) 000816 1769074914.814781 mapread 0x01075d thru 0x011d6b (0x160f bytes) 000817 1769074914.815701 punch from 0x035ab0 to 0x03c4ca (0x6a1b bytes) 000818 1769074914.824105 punch from 0x00dfd9 to 0x01dfd1 (0xfff9 bytes) 000819 1769074914.828855 trunc from 0x007406 to 0x00cdbb (0x59b6 bytes) 000820 1769074914.835359 punch from 0x035f34 to 0x03e1a8 (0x8275 bytes) 000821 1769074914.841310 trunc from 0x016514 to 0x021507 (0xaff4 bytes) 000822 1769074914.846771 mapread 0x018f62 thru 0x020d98 (0x7e37 bytes) 000823 1769074914.917089 write 0x026c96 thru 0x032b21 (0xbe8c bytes) 000824 1769074914.934360 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1a9): Mirror 10: 0x6f7265ec (stale) Mirror 13: 0x6f7265ec (stale) Mirror 15: 0x6f7265ec (stale) Mirror 16: 0x6f7265ec (stale) Mirror 17: 0x6f7265ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1a9): Mirror 10: 0x6f7265ec (stale) Mirror 13: 0x6f7265ec (stale) Mirror 15: 0x6f7265ec (stale) Mirror 16: 0x6f7265ec (stale) Mirror 17: 0x6f7265ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1a9: skipped 000825 1769074915.702229 mapread 0x029668 thru 0x031f4c (0x88e5 bytes) 000826 1769074915.724136 trunc from 0x03e1a9 to 0x03cd4c (0x145c bytes) 000827 1769074915.789887 fallocate 0x03c856 thru 0x03ffff (0x37aa bytes) 000828 1769074915.811013 mapread 0x0370e6 thru 0x03c8de (0x57f9 bytes) 000829 1769074915.830346 mapread 0x030a22 thru 0x03510a (0x46e9 bytes) 000830 1769074915.852272 punch from 0x00ed06 to 0x018cfa (0x9ff5 bytes) 000831 1769074915.867955 fallocate 0x03495f thru 0x03ffff (0xb6a1 bytes) 000832 1769074915.882229 trunc from 0x040000 to 0x01ba3f (0x245c0 bytes) 000833 1769074915.913947 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1ba40): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba40): Mirror 10: 0xec25b1a3 (stale) Mirror 13: 0xec25b1a3 (stale) Mirror 15: 0xec25b1a3 (stale) Mirror 16: 0xec25b1a3 (stale) Mirror 17: 0xec25b1a3 (stale) CRC-32 checksum value for chunk [0, 0x1ba40): Mirror 1: 0x1200d18c Mirror 18: 0x1200d18c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba40: skipped 000834 1769074917.519566 trunc from 0x01ba40 to 0x005890 (0x161af bytes) 000835 1769074917.638278 mapread 0x005692 thru 0x005890 (0x1ff bytes) 000836 1769074917.650100 mapread 0x001cad thru 0x005890 (0x3be4 bytes) 000837 1769074917.671990 trunc from 0x005891 to 0x00d64e (0x7dbe bytes) 000838 1769074917.696720 trunc from 0x00b8c4 to 0x00d64e (0x1d8b bytes) 000839 1769074917.700837 trunc from 0x00cae7 to 0x00d64e (0xb68 bytes) 000840 1769074917.705026 mapwrite 0x0300b3 thru 0x03112a (0x1078 bytes) 000841 1769074917.735758 punch from 0x0137af to 0x0190ac (0x58fe bytes) 000842 1769074917.744291 read 0x01ab5f thru 0x01da5b (0x2efd bytes) 000844 1769074917.752385 punch from 0x007a96 to 0x01645d (0xe9c8 bytes) 000845 1769074917.774922 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3112b): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3112b): Mirror 10: 0x17b01986 (stale) Mirror 13: 0x17b01986 (stale) Mirror 15: 0x17b01986 (stale) Mirror 16: 0x17b01986 (stale) Mirror 17: 0x17b01986 (stale) Mirror 18: 0x16e16755 (stale) CRC-32 checksum value for chunk [0, 0x3112b): Mirror 1: 0x96daaf2a Mirror 19: 0x96daaf2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3112b: skipped 000846 1769074918.979604 write 0x017458 thru 0x0222ad (0xae56 bytes) 000847 1769074919.018447 trunc from 0x03112b to 0x005837 (0x2b8f3 bytes) 000848 1769074919.031510 trunc from 0x005838 to 0x000289 (0x55ae bytes) 000850 1769074919.040366 write 0x02bd48 thru 0x02c58c (0x845 bytes) 000851 1769074919.041270 read 0x00f480 thru 0x018d9a (0x991b bytes) 000852 1769074919.047730 punch from 0x0012b2 to 0x006c98 (0x59e7 bytes) 000853 1769074919.050865 write 0x01b770 thru 0x0292f4 (0xdb85 bytes) 000854 1769074919.060895 read 0x0047dd thru 0x0078ca (0x30ee bytes) 000855 1769074919.075164 trunc from 0x02c58d to 0x008ab4 (0x23ad8 bytes) 000856 1769074919.093553 mapread 0x0011bb thru 0x0017b6 (0x5fc bytes) 000857 1769074919.094483 mapread 0x007885 thru 0x008ab4 (0x1230 bytes) 000858 1769074919.095980 trunc from 0x008ab5 to 0x01d4cf (0x14a1b bytes) 000860 1769074919.115147 trunc from 0x01d4d0 to 0x01cc04 (0x8cb bytes) 000861 1769074919.151267 write 0x0232dd thru 0x02e0c1 (0xade5 bytes) 000863 1769074919.153268 read 0x0139ed thru 0x014ea4 (0x14b8 bytes) 000864 1769074919.166546 mapread 0x0266c4 thru 0x02e0c1 (0x79fe bytes) 000866 1769074919.168230 read 0x00c626 thru 0x017d3d (0xb718 bytes) 000867 1769074919.177434 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2e0c2): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0c2): Mirror 10: 0xdcf47bfe (stale) Mirror 13: 0xdcf47bfe (stale) Mirror 15: 0xdcf47bfe (stale) Mirror 16: 0xdcf47bfe (stale) Mirror 17: 0xdcf47bfe (stale) Mirror 18: 0xedbbe38a (stale) Mirror 19: 0xedbbe38a (stale) CRC-32 checksum value for chunk [0, 0x2e0c2): Mirror 1: 0x12af06f0 Mirror 20: 0x12af06f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0c2: skipped 000868 1769074920.053584 read 0x0134c9 thru 0x01a2db (0x6e13 bytes) 000869 1769074920.090864 punch from 0x0260b0 to 0x02e0c1 (0x8012 bytes) 000871 1769074920.177403 mapread 0x010fdc thru 0x013d37 (0x2d5c bytes) 000872 1769074920.230367 write 0x00aab9 thru 0x00b9b9 (0xf01 bytes) 000873 1769074920.271159 MIRROR_RESYNC 9 mirrors 000874 1769074922.798699 trunc from 0x02e0c2 to 0x012a3e (0x1b683 bytes) 000875 1769074922.835513 mapread 0x009ee7 thru 0x010b85 (0x6c9f bytes) 000876 1769074922.853829 mapwrite 0x027c2b thru 0x030b6c (0x8f42 bytes) 000877 1769074922.901022 punch from 0x004acc to 0x0104db (0xba10 bytes) 000878 1769074922.906851 write 0x0207c0 thru 0x020da2 (0x5e3 bytes) 000879 1769074922.907947 punch from 0x02f66b to 0x030b6c (0x1502 bytes) 000880 1769074922.912689 punch from 0x022774 to 0x024443 (0x1cd0 bytes) 000882 1769074922.919275 mapread 0x007b41 thru 0x017111 (0xf5d1 bytes) 000883 1769074922.955184 mapread 0x006195 thru 0x00688a (0x6f6 bytes) 000884 1769074922.956155 fallocate 0x01f37d thru 0x026fda (0x7c5e bytes) 000886 1769074922.968505 mapread 0x01ab7c thru 0x01f535 (0x49ba bytes) 000887 1769074922.988474 punch from 0x01a1b4 to 0x028dd5 (0xec22 bytes) 000888 1769074923.025206 punch from 0x025b47 to 0x030b6c (0xb026 bytes) 000889 1769074923.039524 mapwrite 0x019137 thru 0x0214b3 (0x837d bytes) 000890 1769074923.128825 mapread 0x02e3ab thru 0x030b6c (0x27c2 bytes) 000891 1769074923.146953 fallocate 0x00ddfd thru 0x00f74d (0x1951 bytes) 000892 1769074923.155423 trunc from 0x030b6d to 0x0058e9 (0x2b283 bytes) 000894 1769074923.199591 read 0x0019ac thru 0x0058e9 (0x3f3e bytes) 000895 1769074923.217108 MIRROR_RESYNC 9 mirrors 000897 1769074924.343897 punch from 0x0016c2 to 0x00418a (0x2ac9 bytes) 000898 1769074924.377619 mapwrite 0x036282 thru 0x03cade (0x685d bytes) 000899 1769074924.437540 trunc from 0x03cadf to 0x021724 (0x1b3ba bytes) 000900 1769074924.462837 mapread 0x0155a4 thru 0x01f5ff (0xa05c bytes) 000901 1769074924.486992 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21725): Mirror 10: 0x522ba19d (stale) Mirror 13: 0x522ba19d (stale) Mirror 15: 0x522ba19d (stale) Mirror 16: 0x522ba19d (stale) Mirror 17: 0x522ba19d (stale) Mirror 18: 0x522ba19d (stale) Mirror 19: 0x522ba19d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21725): Mirror 10: 0x522ba19d (stale) Mirror 13: 0x522ba19d (stale) Mirror 15: 0x522ba19d (stale) Mirror 16: 0x522ba19d (stale) Mirror 17: 0x522ba19d (stale) Mirror 18: 0x522ba19d (stale) Mirror 19: 0x522ba19d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21725: skipped 000902 1769074924.907889 fallocate 0x02b2ec thru 0x036768 (0xb47d bytes) 000903 1769074924.914280 mapread 0x00b164 thru 0x014501 (0x939e bytes) 000904 1769074924.943731 trunc from 0x036769 to 0x017c87 (0x1eae1 bytes) 000905 1769074924.975962 write 0x03e4af thru 0x03ffff (0x1b51 bytes) 000906 1769074924.982733 punch from 0x015874 to 0x01e5bd (0x8d4a bytes) 000907 1769074924.992753 read 0x0110d5 thru 0x01388a (0x27b6 bytes) 000908 1769074925.009195 MIRROR_RESYNC 8 mirrors 000909 1769074925.838808 mapread 0x03325a thru 0x03ac19 (0x79c0 bytes) 000910 1769074925.850548 mapwrite 0x00a199 thru 0x00bf5d (0x1dc5 bytes) 000912 1769074925.902201 write 0x00498c thru 0x011875 (0xceea bytes) 000914 1769074925.906988 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5a5642f4 (stale) Mirror 13: 0x5a5642f4 (stale) Mirror 16: 0x5a5642f4 (stale) Mirror 17: 0x5a5642f4 (stale) Mirror 18: 0x5a5642f4 (stale) Mirror 19: 0x5a5642f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5a5642f4 (stale) Mirror 13: 0x5a5642f4 (stale) Mirror 16: 0x5a5642f4 (stale) Mirror 17: 0x5a5642f4 (stale) Mirror 18: 0x5a5642f4 (stale) Mirror 19: 0x5a5642f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000915 1769074926.334329 fallocate 0x030ab0 thru 0x031cc6 (0x1217 bytes) 000916 1769074926.364096 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5a5642f4 (stale) Mirror 13: 0x5a5642f4 (stale) Mirror 16: 0x5a5642f4 (stale) Mirror 17: 0x5a5642f4 (stale) Mirror 18: 0x5a5642f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5a5642f4 (stale) Mirror 13: 0x5a5642f4 (stale) Mirror 16: 0x5a5642f4 (stale) Mirror 17: 0x5a5642f4 (stale) Mirror 18: 0x5a5642f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1769074926.914548 punch from 0x0067c9 to 0x00fd24 (0x955c bytes) 000918 1769074926.921031 punch from 0x011701 to 0x01bb52 (0xa452 bytes) 000919 1769074926.926128 punch from 0x0392c1 to 0x03ffff (0x6d3f bytes) 000920 1769074926.932102 trunc from 0x0392d7 to 0x03a54b (0x1275 bytes) 000921 1769074926.936200 fallocate 0x02fad9 thru 0x03d9c7 (0xdeef bytes) 000922 1769074926.941128 mapwrite 0x02db7d thru 0x030422 (0x28a6 bytes) 000923 1769074926.980254 write 0x01f192 thru 0x027221 (0x8090 bytes) 000924 1769074926.998939 write 0x023de3 thru 0x028a83 (0x4ca1 bytes) 000927 1769074927.001539 trunc from 0x02349e to 0x026f07 (0x3a6a bytes) 000928 1769074927.016607 read 0x01b537 thru 0x02a75e (0xf228 bytes) 000929 1769074927.023513 read 0x035112 thru 0x03ffff (0xaeee bytes) 000930 1769074927.036757 trunc from 0x00e3c2 to 0x01ad63 (0xc9a2 bytes) 000931 1769074927.041884 mapread 0x018d15 thru 0x01e169 (0x5455 bytes) 000932 1769074927.055930 read 0x003f8e thru 0x008251 (0x42c4 bytes) 000933 1769074927.056922 fallocate 0x02cbbc thru 0x036b04 (0x9f49 bytes) 000934 1769074927.063865 trunc from 0x040000 to 0x0244f2 (0x1bb0d bytes) 000935 1769074927.078858 fallocate 0x0336dc thru 0x03adf9 (0x771e bytes) 000936 1769074927.084686 trunc from 0x00372c to 0x004204 (0xad9 bytes) 000937 1769074927.089288 write 0x01aea5 thru 0x01ecc1 (0x3e1d bytes) 000939 1769074927.090701 punch from 0x013aa7 to 0x01c440 (0x899a bytes) 000940 1769074927.106855 mapread 0x01a303 thru 0x027dc5 (0xdac3 bytes) 000941 1769074927.116469 punch from 0x031b8c to 0x03adf9 (0x926e bytes) 000942 1769074927.120483 fallocate 0x009637 thru 0x018ad6 (0xf4a0 bytes) 000943 1769074927.125456 trunc from 0x03adfa to 0x01e6fd (0x1c6fc bytes) 000944 1769074927.140947 trunc from 0x01e6fe to 0x021faa (0x38ad bytes) 000945 1769074927.158759 trunc from 0x021fab to 0x023696 (0x16ec bytes) 000947 1769074927.178148 mapread 0x01af8a thru 0x023696 (0x870d bytes) 000948 1769074927.200339 read 0x00bad9 thru 0x01b2b3 (0xf7db bytes) 000949 1769074927.201444 punch from 0x0197e2 to 0x023696 (0x9eb5 bytes) 000950 1769074927.205987 punch from 0x00c06d to 0x00d099 (0x102d bytes) 000951 1769074927.210168 punch from 0x01c850 to 0x023696 (0x6e47 bytes) 000952 1769074927.215027 mapread 0x01feb8 thru 0x023696 (0x37df bytes) 000953 1769074927.223982 trunc from 0x023697 to 0x0006b6 (0x22fe0 bytes) 000954 1769074927.247502 fallocate 0x001a8d thru 0x0033c4 (0x1938 bytes) 000955 1769074927.255233 mapwrite 0x02adde thru 0x02c121 (0x1344 bytes) 000956 1769074927.294515 mapwrite 0x00b362 thru 0x019ca0 (0xe93f bytes) 000957 1769074927.394824 read 0x014e24 thru 0x01bb48 (0x6d25 bytes) 000959 1769074927.410668 mapread 0x00f327 thru 0x01b825 (0xc4ff bytes) 000960 1769074927.412288 trunc from 0x001ca1 to 0x005618 (0x3978 bytes) 000961 1769074927.419017 write 0x02384a thru 0x02d0c4 (0x987b bytes) 000962 1769074927.421807 trunc from 0x02d0c5 to 0x01cf26 (0x1019e bytes) 000963 1769074927.443545 trunc from 0x0096a2 to 0x01721d (0xdb7c bytes) 000964 1769074927.459493 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cf27): Mirror 10: 0xc421045d (stale) Mirror 16: 0xc421045d (stale) Mirror 17: 0xc421045d (stale) Mirror 18: 0xc421045d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf27): Mirror 10: 0xc421045d (stale) Mirror 16: 0xc421045d (stale) Mirror 17: 0xc421045d (stale) Mirror 18: 0xc421045d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf27: skipped 000965 1769074927.942960 write 0x004fe6 thru 0x010b0c (0xbb27 bytes) 000966 1769074927.964417 write 0x017d2c thru 0x01d7de (0x5ab3 bytes) 000967 1769074927.973023 trunc from 0x01d7df to 0x0038c5 (0x19f19 bytes) 000968 1769074927.996847 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c6): Mirror 10: 0xe541c5a6 (stale) Mirror 16: 0xe541c5a6 (stale) Mirror 18: 0xe541c5a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c6): Mirror 10: 0xe541c5a6 (stale) Mirror 16: 0xe541c5a6 (stale) Mirror 18: 0xe541c5a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c6: skipped 000969 1769074928.319521 mapread 0x0033dd thru 0x0038c5 (0x4e9 bytes) 000970 1769074928.328026 punch from 0x0007ca to 0x0038c5 (0x30fc bytes) 000971 1769074928.335260 write 0x026efd thru 0x029f1a (0x301e bytes) 000972 1769074928.342945 mapwrite 0x01cdd5 thru 0x02b1e8 (0xe414 bytes) 000973 1769074928.398542 trunc from 0x01e423 to 0x026734 (0x8312 bytes) 000974 1769074928.403977 trunc from 0x02b1e9 to 0x02353a (0x7cae bytes) 000975 1769074928.419947 mapwrite 0x0105ca thru 0x012ab5 (0x24ec bytes) 000976 1769074928.446694 mapread 0x01c97e thru 0x02353a (0x6bbd bytes) 000977 1769074928.456549 write 0x0023fa thru 0x00fc73 (0xd87a bytes) 000978 1769074928.466970 trunc from 0x02353b to 0x026310 (0x2dd6 bytes) 000979 1769074928.493208 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26311): Mirror 10: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26311): Mirror 10: 0x3ab67bd (stale) Mirror 16: 0x3ab67bd (stale) Mirror 18: 0x3ab67bd (stale) CRC-32 checksum value for chunk [0, 0x26311): Mirror 1: 0x761c4cbf Mirror 19: 0x761c4cbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26311: skipped 000980 1769074929.317365 mapwrite 0x021a5a thru 0x025c82 (0x4229 bytes) 000981 1769074929.408134 read 0x019236 thru 0x01f2bf (0x608a bytes) 000982 1769074929.414334 trunc from 0x026311 to 0x010dab (0x15565 bytes) 000983 1769074929.435857 trunc from 0x010dac to 0x00218f (0xec1c bytes) 000984 1769074929.455857 mapwrite 0x002831 thru 0x00ab0c (0x82dc bytes) 000985 1769074929.490848 fallocate 0x008005 thru 0x009dba (0x1db6 bytes) 000986 1769074929.495781 mapread 0x000586 thru 0x001976 (0x13f1 bytes) 000987 1769074929.501184 read 0x0097c8 thru 0x00ab0c (0x1345 bytes) 000988 1769074929.501794 punch from 0x0010e9 to 0x00ab0c (0x9a24 bytes) 000989 1769074929.507561 read 0x001190 thru 0x00ab0c (0x997d bytes) 000990 1769074929.514764 mapwrite 0x0226fb thru 0x02c1e5 (0x9aeb bytes) 000991 1769074929.555283 mapwrite 0x02be18 thru 0x033e64 (0x804d bytes) 000992 1769074929.589743 write 0x019bc5 thru 0x020ccd (0x7109 bytes) 000993 1769074929.591512 fallocate 0x03952e thru 0x03e614 (0x50e7 bytes) 000995 1769074929.596908 punch from 0x031a17 to 0x03d201 (0xb7eb bytes) 000996 1769074929.602333 read 0x0158fb thru 0x01a07b (0x4781 bytes) 000997 1769074929.603110 write 0x009253 thru 0x016ebb (0xdc69 bytes) 000999 1769074929.605667 write 0x03dd0b thru 0x03e8fc (0xbf2 bytes) 001000 1769074929.612173 trunc from 0x03e8fd to 0x033a56 (0xaea6 bytes) All operations completed A-OK! PASS 70a (104s) == sanity-flr test 71: check mirror extend parent fid ==== 04:42:15 (1769074935) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x1ce:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 04:42:25 (1769074945) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261408 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099688 s, 10.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x76:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x77:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x96:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x95:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x78:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 04:42:34 (1769074954) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4806624 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7358112 .. Extending file size to 8443808 .. done Extending file size to 5575776 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9269696 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8471392 .. Extending file size to 8684320 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8816864 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8842336 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 59821 59823 59824 59827 59829 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:43:56 (1769075036) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7246240 .. Extending file size to 2350848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8934272 .. Extending file size to 3949888 .. failed Extending file size to 4101952 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6839680 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8645184 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8806912 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9322752 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61775 61777 61778 61781 61782 done 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:45:14 (1769075114) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2212 1269092 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19064 3587956 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25040 3581980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44104 7169936 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2276 1269028 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2212 1269092 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 19064 3587956 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25040 3581980 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44104 7169936 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161448 s, 13.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116235 s, 18.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x917341ba (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:45:35 (1769075135) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0693159 s, 30.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 04:45:43 (1769075143) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xac:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xad:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:45:51 (1769075151) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.062729 s, 33.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x233:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x235:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x236:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:45:58 (1769075158) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127361 s, 16.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x234:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x235:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x237:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x234:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x235:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x237:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x238:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:46:08 (1769075168) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110616 s, 19.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x236:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x239:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x236:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x239:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23a:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:46:42 (1769075202) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435495 s, 24.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x8e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xaf:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:47:35 (1769075255) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157867 s, 13.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x90:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:48:28 (1769075308) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg210-server: Pool lustre.test_204f created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130423 s, 16.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg210-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:49:46 (1769075386) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:49:54 (1769075394) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:50:03 (1769075403) oleg210-server: Pool lustre.test_206 created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg210-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:50:35 (1769075435) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130038 s, 8.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454753 s, 23.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x65a34976 Mirror 3: 0x65a34976 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:50:49 (1769075449) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:50:53 (1769075453) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:50:58 (1769075458) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.122766 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2270000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2270000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2270000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1630000 Read this (4 bytes): BBBB 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:51:40 (1769075500) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0413449 s, 198 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:51:48 (1769075508) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609572 s, 17.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x9b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 04:51:50.000000000 -0500 Modify: 2026-01-22 04:51:50.000000000 -0500 Change: 2026-01-22 04:51:51.000000000 -0500 Birth: 2026-01-22 04:51:50.000000000 -0500 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:51:57 (1769075517) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202399 s, 51.8 MB/s oleg210-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg210-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg210-server: O/280000401/d29/bd: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:52:18 (1769075538) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.164498 s, 249 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0179155 s, 229 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 22 04:52 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 22 04:52 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:52:27 (1769075547) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:52:35 (1769075555) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1874 sec =========== 04:52:44 (1769075564) === sanity-flr: start cleanup 04:52:46 (1769075566) === === sanity-flr: finish cleanup 04:52:56 (1769075576) ===