-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 07:07:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg241-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:08:01 (1787137681) === oleg241-client.virtnet: executing check_config_client /mnt/lustre oleg241-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg241-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f0608406000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f0608406000.idle_timeout=debug disable quota as required oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:08:42 (1787137722) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8250] == sanity-flr test 0a: lfs mirror create with -N option == 07:08:44 (1787137724) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:09:04 (1787137744) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:09:07 (1787137747) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:09:11 (1787137751) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:09:34 (1787137774) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:09:38 (1787137778) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:09:41 (1787137781) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f0608406000.stats=clear osc.lustre-OST0001-osc-ffff9f0608406000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:09:49 (1787137789) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:10:00 (1787137800) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:10:09 (1787137809) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:10:12 (1787137812) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:10:21 (1787137821) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 07:10:32 (1787137832) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:10:40 (1787137840) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:10:49 (1787137849) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:10:58 (1787137858) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:11:09 (1787137869) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:11:16 (1787137876) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.249234 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882268 s, 11.9 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:11:23 (1787137883) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.458388 s, 20.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:11:31 (1787137891) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:11:40 (1787137900) Stopping /mnt/lustre-ost1 (opts:) on oleg241-server oleg241-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg241-server: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg241-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg241-server oleg241-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg241-server: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg241-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:12:56 (1787137976) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.399446 s, 10.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg241-server oleg241-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg241-server: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg241-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:13:44 (1787138024) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg241-server oleg241-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg241-server: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg241-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg241-server oleg241-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg241-server: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg241-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:15:15 (1787138115) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.8682 s, 18.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 07:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.932s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg241-server oleg241-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0608406000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.214s user 0m0.000s sys 0m0.199s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg241-server: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg241-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg241-server oleg241-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.322s user 0m0.001s sys 0m0.192s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg241-server: oleg241-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg241-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg241-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0608406000.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:16:47 (1787138207) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:16:51 (1787138211) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:16:54 (1787138214) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 07:16:58 (1787138218) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 07:17:07 (1787138227) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.636102 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:17:20 (1787138240) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448456 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259968 s, 40.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:17:29 (1787138249) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:17:37 (1787138257) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000872642 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00169665 s, 604 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104218 s, 983 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00135135 s, 758 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 07:17:46 (1787138266) create test files with size 15734522 .. c19f43d271144b8c89bf33be8472b118 - 30be269ee4c62635328ad0f4ae0fd5a3 - b186e283fa25ba3c9e24b0a4d8723842 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg241-client.virtnet /mnt/lustre (opts:) Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 07:18:09 (1787138289) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:18:25 (1787138305) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:18:32 (1787138312) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115672 s, 9.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286617 s, 36.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 07:18:41 (1787138321) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.059276 s, 35.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.207428 s, 20.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105413 s, 19.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 07:18:56 (1787138336) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 07:18:59 (1787138339) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:19:03 (1787138343) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539995 s, 19.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 07:19:12 (1787138352) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.265919 s, 11.8 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:19:23 (1787138363) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:19:34 (1787138374) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.570526 s, 18.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg241-client.virtnet /mnt/lustre (opts:) Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:19:47 (1787138387) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f061974c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.321786 s, 32.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg241-client.virtnet /mnt/lustre (opts:) Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9f06040d2800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:20:04 (1787138404) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:20:44 (1787138444) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:21:23 (1787138483) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 07:21:26 (1787138486) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.221965 s, 14.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.235604 s, 13.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa5e26601 (nosync,stale) Mirror 2: 0xa5e26601 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5a457eb6c7d872944ad5fc2227b56d13 - mirror 1 checksum: 400babcfc9b611a098cd2f12fe206c8f - mirror 2 checksum: 5a457eb6c7d872944ad5fc2227b56d13 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T07:21:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T07:21:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5a457eb6c7d872944ad5fc2227b56d13 - mirror 1 checksum: 400babcfc9b611a098cd2f12fe206c8f - mirror 2 checksum: 400babcfc9b611a098cd2f12fe206c8f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T07:21:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T07:21:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 07:21:38 (1787138498) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725006 s, 14.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:21:46 (1787138506) Failing mds1 on oleg241-server Stopping /mnt/lustre-mds1 (opts:) on oleg241-server 07:21:58 (1787138518) shut down facet: mds1 facet_host: oleg241-server facet_failover_host: oleg241-server Failover mds1 to oleg241-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 07:22:23 (1787138543) targets are mounted 07:22:23 (1787138543) facet_failover done oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042244 s, 24.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:22:42 (1787138562) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0986578 s, 208 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0325385 s, 629 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:22:53 (1787138573) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.106874 s, 192 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0116228 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:23:04 (1787138584) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00808255 s, 2.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:23:12 (1787138592) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.107292 s, 191 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:23:21 (1787138601) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0103201 s, 397 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0331723 s, 123 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:23:32 (1787138612) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (31s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:24:03 (1787138643) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:24:30 (1787138670) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:24:52 (1787138692) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:25:15 (1787138715) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 07:25:22 (1787138722) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57589 mirror split pid 57590 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57589 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57590 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 07:26:30 (1787138790) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8793 fallocating to largest ever: 0x13adb 000001 1787138793.706877 fallocate 0x00fbe6 thru 0x013ada (0x3ef5 bytes) skipping zero length zero range skipping zero length zero range skipping zero length zero range 000005 1787138793.749692 trunc from 00000000 to 0x0016a3 (0x16a4 bytes) 000006 1787138793.778376 write 0x0352c2 thru 0x0386d2 (0x3411 bytes) fallocating to largest ever: 0x1b7e3 000007 1787138793.780072 fallocate 0x0122e0 thru 0x01b7e2 (0x9503 bytes) 000008 1787138793.797948 trunc from 0x012da2 to 0x015c40 (0x2e9f bytes) 000010 1787138793.821316 mapread 0x02db03 thru 0x0354a4 (0x79a2 bytes) 000011 1787138793.884760 mapread 0x016526 thru 0x017bdd (0x16b8 bytes) fallocating to largest ever: 0x3549d 000012 1787138793.907099 fallocate 0x030c1e thru 0x03549c (0x487f bytes) 000013 1787138793.931699 mapwrite 0x01e125 thru 0x01fb6c (0x1a48 bytes) 000014 1787138793.980304 mapread 0x02afb6 thru 0x037788 (0xc7d3 bytes) 000015 1787138793.986551 trunc from 0x000599 to 0x006e5f (0x68c7 bytes) truncating to largest ever: 0x3fd91 000016 1787138794.004374 trunc from 0x0386d3 to 0x03fd90 (0x76be bytes) 000017 1787138794.065212 read 0x01bc50 thru 0x027e9c (0xc24d bytes) 000018 1787138794.074774 write 0x03db20 thru 0x03ffff (0x24e0 bytes) 000019 1787138794.096400 mapwrite 0x0234d9 thru 0x02f8fd (0xc425 bytes) 000020 1787138794.169645 punch from 0x0051a1 to 0x010190 (0xaff0 bytes) 000021 1787138794.193553 MIRROR_EXTEND to 2 mirrors 000022 1787138795.164080 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) 000023 1787138795.422035 fallocate 0x027044 thru 0x02850a (0x14c7 bytes) 000024 1787138795.439174 trunc from 0x040000 to 0x035a88 (0xa577 bytes) 000025 1787138795.496605 mapwrite 0x00ef3e thru 0x012dcd (0x3e90 bytes) 000026 1787138795.540429 write 0x014096 thru 0x01c9fe (0x8969 bytes) 000027 1787138795.548296 mapwrite 0x014607 thru 0x01818d (0x3b87 bytes) 000028 1787138795.579338 write 0x00dbb0 thru 0x01cea9 (0xf2fa bytes) 000029 1787138795.581153 mapread 0x028e69 thru 0x02cb66 (0x3cfe bytes) 000030 1787138795.596279 mapread 0x02a7ff thru 0x034c9a (0xa49c bytes) 000031 1787138795.625419 read 0x01a15a thru 0x027943 (0xd7ea bytes) 000032 1787138795.640201 mapread 0x030932 thru 0x031b85 (0x1254 bytes) 000033 1787138795.645820 MIRROR_EXTEND to 2 mirrors 000034 1787138796.568966 mapwrite 0x038887 thru 0x03ffff (0x7779 bytes) 000035 1787138796.687384 read 0x0006ea thru 0x009f7d (0x9894 bytes) 000036 1787138796.699486 mapwrite 0x000ffb thru 0x00e532 (0xd538 bytes) 000037 1787138796.743505 punch from 0x035b50 to 0x03ffff (0xa4b0 bytes) 000038 1787138796.754076 mapread 0x02769d thru 0x02bca8 (0x460c bytes) 000039 1787138796.773965 write 0x02aeb0 thru 0x033978 (0x8ac9 bytes) 000040 1787138796.776777 fallocate 0x00800f thru 0x00ee25 (0x6e17 bytes) 000041 1787138796.788215 fallocate 0x026439 thru 0x02e677 (0x823f bytes) 000042 1787138796.795326 fallocate 0x01d5a4 thru 0x01ecec (0x1749 bytes) 000044 1787138796.804359 mapwrite 0x01170a thru 0x015bf3 (0x44ea bytes) 000045 1787138796.845681 trunc from 0x040000 to 0x0129f3 (0x2d60c bytes) 000046 1787138796.871913 read 0x00ad8d thru 0x00bc91 (0xf05 bytes) 000047 1787138796.875415 mapread 0x0006c3 thru 0x00bf22 (0xb860 bytes) 000048 1787138796.878387 trunc from 0x00ed7b to 0x0129f3 (0x3c79 bytes) 000049 1787138796.884808 read 0x00dabf thru 0x0129f3 (0x4f35 bytes) 000050 1787138796.893539 punch from 0x00e7ea to 0x0129f3 (0x420a bytes) 000051 1787138796.898807 trunc from 0x003c9c to 0x00a54a (0x68af bytes) 000052 1787138796.905601 trunc from 0x00c4da to 0x0129f3 (0x651a bytes) 000053 1787138796.911414 mapread 0x007e38 thru 0x00809e (0x267 bytes) 000054 1787138796.927419 punch from 0x01031e to 0x0129f3 (0x26d6 bytes) 000055 1787138796.933279 mapwrite 0x0105fd thru 0x0172a9 (0x6cad bytes) 000056 1787138797.038832 trunc from 0x002b2e to 0x007c6b (0x513e bytes) 000057 1787138797.068618 trunc from 0x016bfc to 0x0172a9 (0x6ae bytes) 000058 1787138797.085519 mapwrite 0x03747c thru 0x037ba2 (0x727 bytes) 000061 1787138797.169426 fallocate 0x0369ae thru 0x037e47 (0x149a bytes) 000062 1787138797.176112 trunc from 0x037e48 to 0x02595c (0x124eb bytes) 000063 1787138797.200773 fallocate 0x01f41b thru 0x02adb3 (0xb999 bytes) 000064 1787138797.206716 mapread 0x017374 thru 0x022bda (0xb867 bytes) 000065 1787138797.213194 trunc from 0x018d27 to 0x026729 (0xda03 bytes) 000066 1787138797.219438 punch from 0x000d63 to 0x0100e2 (0xf380 bytes) 000067 1787138797.225642 write 0x001097 thru 0x00f361 (0xe2cb bytes) 000069 1787138797.239207 write 0x0055da thru 0x01471a (0xf141 bytes) 000070 1787138797.242314 punch from 0x00aa42 to 0x012d43 (0x8302 bytes) 000071 1787138797.260498 read 0x001c84 thru 0x00d313 (0xb690 bytes) 000072 1787138797.271551 trunc from 0x007429 to 0x013219 (0xbdf1 bytes) 000073 1787138797.278138 punch from 0x017b69 to 0x01b7bd (0x3c55 bytes) 000074 1787138797.285266 trunc from 0x02adb4 to 0x01dd7a (0xd039 bytes) 000075 1787138797.310086 trunc from 0x01dd7b to 0x00b358 (0x12a22 bytes) 000076 1787138797.328600 trunc from 0x00b359 to 0x03298f (0x27637 bytes) 000077 1787138797.348274 write 0x020ac7 thru 0x026bab (0x60e5 bytes) 000078 1787138797.364794 punch from 0x01e10c to 0x02aaa7 (0xc99c bytes) 000079 1787138797.380524 punch from 0x007c8b to 0x015ccd (0xe043 bytes) 000080 1787138797.391242 mapwrite 0x03f347 thru 0x03ffff (0xcb9 bytes) 000081 1787138797.435491 fallocate 0x00bc66 thru 0x019e9b (0xe236 bytes) 000082 1787138797.441554 trunc from 0x040000 to 0x022ac3 (0x1d53c bytes) 000083 1787138797.468729 mapread 0x014462 thru 0x01f584 (0xb123 bytes) 000084 1787138797.534712 mapwrite 0x035f3a thru 0x038eb5 (0x2f7c bytes) 000085 1787138797.613134 punch from 0x018ba2 to 0x024202 (0xb661 bytes) 000086 1787138797.619503 mapwrite 0x01b0aa thru 0x0294fd (0xe454 bytes) 000087 1787138797.730425 MIRROR_RESYNC 2 mirrors 000088 1787138798.550163 read 0x01ac5b thru 0x01cfd0 (0x2376 bytes) 000089 1787138798.557538 mapwrite 0x039be3 thru 0x03ffff (0x641d bytes) 000090 1787138798.738131 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce6f9ac6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26992122 Mirror 3: 0x26992122 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000091 1787138799.926669 punch from 0x00a8ac to 0x018f1e (0xe673 bytes) 000092 1787138799.991761 read 0x0110b0 thru 0x020159 (0xf0aa bytes) 000093 1787138800.025373 mapread 0x02d09b thru 0x03756c (0xa4d2 bytes) 000094 1787138800.051459 mapwrite 0x033458 thru 0x03c2d0 (0x8e79 bytes) 000095 1787138800.083730 mapwrite 0x03fd13 thru 0x03ffff (0x2ed bytes) 000096 1787138800.099632 read 0x039c5e thru 0x03ffff (0x63a2 bytes) 000097 1787138800.100679 write 0x00d132 thru 0x018071 (0xaf40 bytes) 000098 1787138800.117427 read 0x03f008 thru 0x03ffff (0xff8 bytes) 000100 1787138800.118687 mapwrite 0x007f56 thru 0x01051c (0x85c7 bytes) 000101 1787138800.142468 trunc from 0x040000 to 0x034970 (0xb68f bytes) 000102 1787138800.169337 punch from 0x02d853 to 0x02f797 (0x1f45 bytes) 000103 1787138800.180106 punch from 0x00909c to 0x017c0d (0xeb72 bytes) 000104 1787138800.190438 fallocate 0x02f55f thru 0x03f07f (0xfb21 bytes) 000106 1787138800.202477 MIRROR_RESYNC 3 mirrors 000107 1787138801.440728 punch from 0x0292a8 to 0x033a0c (0xa765 bytes) 000108 1787138801.476754 trunc from 0x02abc0 to 0x02d62a (0x2a6b bytes) 000110 1787138801.485210 mapread 0x0360f5 thru 0x0382f8 (0x2204 bytes) 000111 1787138801.515269 trunc from 0x03f080 to 0x026250 (0x18e2f bytes) 000112 1787138801.536486 mapread 0x002995 thru 0x00a085 (0x76f1 bytes) 000113 1787138801.549650 fallocate 0x0157c4 thru 0x01ba7b (0x62b8 bytes) 000114 1787138801.555742 mapread 0x006015 thru 0x012883 (0xc86f bytes) 000116 1787138801.586451 write 0x00bc6d thru 0x00f5c4 (0x3958 bytes) 000117 1787138801.605413 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26251): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26251): Mirror 2: 0x908df633 (stale) Mirror 3: 0x908df633 (stale) CRC-32 checksum value for chunk [0, 0x26251): Mirror 1: 0xcec12f62 Mirror 4: 0xcec12f62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26251: skipped 000118 1787138803.213604 mapwrite 0x00bd69 thru 0x00d4c1 (0x1759 bytes) 000119 1787138803.585423 punch from 0x00b077 to 0x00f72f (0x46b9 bytes) 000120 1787138803.611825 fallocate 0x00d319 thru 0x00d48a (0x172 bytes) 000121 1787138803.667598 fallocate 0x034f2f thru 0x039568 (0x463a bytes) 000122 1787138803.688742 mapread 0x01c76d thru 0x022d93 (0x6627 bytes) 000123 1787138803.721339 mapwrite 0x03723b thru 0x03e70d (0x74d3 bytes) 000124 1787138803.860485 fallocate 0x02d211 thru 0x02f694 (0x2484 bytes) 000125 1787138803.866429 read 0x02929b thru 0x03874d (0xf4b3 bytes) 000126 1787138803.875070 fallocate 0x004c46 thru 0x00a866 (0x5c21 bytes) 000127 1787138803.879684 read 0x027e13 thru 0x0308ca (0x8ab8 bytes) 000128 1787138803.885768 read 0x00b22d thru 0x019413 (0xe1e7 bytes) 000129 1787138803.891544 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e70e): Mirror 3: 0x3d5f3bf4 (stale) Mirror 4: 0x2e1fbec8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e70e): Mirror 3: 0x3d5f3bf4 (stale) Mirror 4: 0x2e1fbec8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e70e: skipped 000130 1787138804.200473 write 0x03e50f thru 0x03ffff (0x1af1 bytes) 000131 1787138804.213792 punch from 0x03c831 to 0x03ffff (0x37cf bytes) 000133 1787138804.227967 punch from 0x028c1c to 0x031439 (0x881e bytes) 000135 1787138804.234979 mapread 0x0319c3 thru 0x034aa8 (0x30e6 bytes) 000136 1787138804.262551 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd3e07f74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd3e07f74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000137 1787138804.540253 mapread 0x011602 thru 0x017a2c (0x642b bytes) 000138 1787138804.572766 mapread 0x0390f3 thru 0x03b3a6 (0x22b4 bytes) 000139 1787138804.581904 mapwrite 0x037142 thru 0x03cf47 (0x5e06 bytes) 000140 1787138804.605764 read 0x00675f thru 0x0081c9 (0x1a6b bytes) 000141 1787138804.623404 mapwrite 0x006f06 thru 0x007611 (0x70c bytes) 000142 1787138804.675482 trunc from 0x040000 to 0x006deb (0x39214 bytes) 000143 1787138804.705441 mapwrite 0x019f78 thru 0x01ee82 (0x4f0b bytes) fallocating to largest ever: 0x40000 000145 1787138804.764385 fallocate 0x039b1b thru 0x03ffff (0x64e5 bytes) 000147 1787138804.774767 read 0x000e7b thru 0x00bcb5 (0xae3b bytes) 000148 1787138804.797989 read 0x005a39 thru 0x0159d4 (0xff9c bytes) 000149 1787138804.807545 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd3e07f74 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e476d9d Mirror 5: 0x9e476d9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1787138806.284240 fallocate 0x03d2c3 thru 0x03ffff (0x2d3d bytes) 000151 1787138806.354612 mapwrite 0x009a45 thru 0x0127e8 (0x8da4 bytes) 000152 1787138806.447471 mapwrite 0x032af7 thru 0x03f4c4 (0xc9ce bytes) 000153 1787138806.534408 trunc from 0x01e85c to 0x023022 (0x47c7 bytes) 000154 1787138806.546738 trunc from 0x02b16c to 0x0393b5 (0xe24a bytes) 000155 1787138806.556703 trunc from 0x00f39a to 0x01a7b1 (0xb418 bytes) 000156 1787138806.562056 punch from 0x0234bb to 0x0280ca (0x4c10 bytes) 000157 1787138806.572257 trunc from 0x03780b to 0x03ffff (0x87f5 bytes) 000158 1787138806.595036 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9e476d9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9e476d9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000159 1787138806.879503 read 0x00cc91 thru 0x0191ec (0xc55c bytes) 000160 1787138806.893379 read 0x0252d1 thru 0x02ee07 (0x9b37 bytes) 000161 1787138806.904191 write 0x03a033 thru 0x03a2b4 (0x282 bytes) 000162 1787138806.909222 read 0x03b3e8 thru 0x03ffff (0x4c18 bytes) 000163 1787138806.910026 punch from 0x03642d to 0x03d74b (0x731f bytes) 000164 1787138806.921572 mapread 0x01d94d thru 0x01f31c (0x19d0 bytes) 000165 1787138806.922641 mapwrite 0x00607d thru 0x00b8fd (0x5881 bytes) 000166 1787138806.938827 mapread 0x022166 thru 0x025f0a (0x3da5 bytes) 000167 1787138806.941472 write 0x01965c thru 0x02725a (0xdbff bytes) 000168 1787138806.943476 mapread 0x008cd5 thru 0x0105f9 (0x7925 bytes) 000169 1787138806.945201 mapread 0x015fc3 thru 0x016b1a (0xb58 bytes) 000170 1787138806.951435 MIRROR_RESYNC 2 mirrors 000171 1787138807.789263 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) 000172 1787138807.902625 write 0x0233fd thru 0x02aa05 (0x7609 bytes) 000173 1787138807.930476 fallocate 0x01b274 thru 0x01e199 (0x2f26 bytes) 000174 1787138807.934833 trunc from 0x039223 to 0x03ffff (0x6ddd bytes) 000175 1787138807.939696 fallocate 0x03bb64 thru 0x03df83 (0x2420 bytes) 000176 1787138807.943902 mapread 0x017ba8 thru 0x024aaa (0xcf03 bytes) 000178 1787138807.970990 write 0x036948 thru 0x03a8b7 (0x3f70 bytes) 000179 1787138807.978193 punch from 0x00b099 to 0x012094 (0x6ffc bytes) 000181 1787138807.984381 trunc from 0x040000 to 0x026945 (0x196ba bytes) 000182 1787138808.006642 punch from 0x003f1c to 0x0128c9 (0xe9ae bytes) 000183 1787138808.016779 MIRROR_EXTEND to 2 mirrors 000184 1787138808.494247 fallocate 0x022f7b thru 0x024f27 (0x1fad bytes) 000185 1787138808.536615 punch from 0x01e532 to 0x02085b (0x232a bytes) 000186 1787138808.541689 punch from 0x01840b to 0x01fefd (0x7af3 bytes) 000187 1787138808.553729 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26946): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26946): Mirror 2: 0x8c118bf7 (stale) CRC-32 checksum value for chunk [0, 0x26946): Mirror 1: 0x94e4e9e6 Mirror 3: 0x94e4e9e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26946: skipped 000188 1787138809.463059 read 0x024b64 thru 0x026945 (0x1de2 bytes) 000189 1787138809.476706 punch from 0x01c67a to 0x026945 (0xa2cc bytes) 000191 1787138809.546562 read 0x020630 thru 0x026945 (0x6316 bytes) 000192 1787138809.564307 write 0x014493 thru 0x01847f (0x3fed bytes) 000193 1787138809.569173 punch from 0x01d17f to 0x02317c (0x5ffe bytes) 000194 1787138809.575351 trunc from 0x00307a to 0x00e5a9 (0xb530 bytes) 000195 1787138809.582308 mapwrite 0x01defc thru 0x021ffc (0x4101 bytes) 000196 1787138809.680225 trunc from 0x0141fe to 0x01c745 (0x8548 bytes) 000197 1787138809.686727 trunc from 0x005330 to 0x00f61a (0xa2eb bytes) 000198 1787138809.693443 fallocate 0x01a13e thru 0x0284b3 (0xe376 bytes) 000199 1787138809.704753 fallocate 0x00c9c8 thru 0x01776b (0xada4 bytes) 000200 1787138809.710105 mapwrite 0x01c6ad thru 0x027dad (0xb701 bytes) 000201 1787138809.772893 mapwrite 0x039e99 thru 0x03ffff (0x6167 bytes) 000203 1787138809.827247 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x676e30de (stale) Mirror 3: 0x51a3418 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xea9ba9d Mirror 4: 0xea9ba9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000204 1787138811.028264 punch from 0x010275 to 0x01e547 (0xe2d3 bytes) 000205 1787138811.065636 trunc from 0x040000 to 0x013ee3 (0x2c11c bytes) 000206 1787138811.097345 read 0x00dc1c thru 0x013ee3 (0x62c8 bytes) 000207 1787138811.108203 fallocate 0x0095dd thru 0x013194 (0x9bb8 bytes) 000208 1787138811.122143 read 0x009c0f thru 0x00ae73 (0x1265 bytes) 000210 1787138811.154104 punch from 0x003935 to 0x00c968 (0x9034 bytes) 000211 1787138811.171242 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13ee4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ee4): Mirror 2: 0xa67e8a22 (stale) Mirror 3: 0xa67e8a22 (stale) Mirror 4: 0xf329d4fb (stale) CRC-32 checksum value for chunk [0, 0x13ee4): Mirror 1: 0xf329d4fb Mirror 5: 0xf329d4fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ee4: skipped 000212 1787138811.895240 fallocate 0x02ce3d thru 0x03201e (0x51e2 bytes) 000214 1787138811.937217 trunc from 0x0271f4 to 0x02919b (0x1fa8 bytes) 000215 1787138811.941902 trunc from 0x03201f to 0x024ef8 (0xd126 bytes) 000216 1787138811.976957 read 0x021bb5 thru 0x024ef8 (0x3344 bytes) 000217 1787138811.984672 fallocate 0x03b0bc thru 0x03fb87 (0x4acc bytes) 000218 1787138811.989151 fallocate 0x00f36e thru 0x017a9a (0x872d bytes) 000219 1787138811.993125 mapread 0x01e2e3 thru 0x024ef8 (0x6c16 bytes) 000220 1787138812.006960 punch from 0x02060b to 0x021bcc (0x15c2 bytes) 000221 1787138812.012677 trunc from 0x024ef9 to 0x02a52f (0x5637 bytes) 000222 1787138812.028669 trunc from 0x02a530 to 0x00a852 (0x1fcdd bytes) 000223 1787138812.050130 trunc from 0x005d48 to 0x00a852 (0x4b0b bytes) 000224 1787138812.055750 trunc from 0x00a853 to 0x03237c (0x27b2a bytes) 000225 1787138812.070668 trunc from 0x02cc94 to 0x02d50e (0x87b bytes) 000226 1787138812.081305 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3237d): Mirror 2: 0xc98df452 (stale) Mirror 3: 0x6d869821 (stale) Mirror 4: 0x9e4c55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3237d): Mirror 2: 0xc98df452 (stale) Mirror 3: 0x6d869821 (stale) Mirror 4: 0x9e4c55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3237d: skipped 000227 1787138812.401131 write 0x03e4c4 thru 0x03ffff (0x1b3c bytes) 000229 1787138812.407140 trunc from 0x00cb5a to 0x01363c (0x6ae3 bytes) 000230 1787138812.414419 punch from 0x007240 to 0x00df25 (0x6ce6 bytes) 000231 1787138812.420292 write 0x02519d thru 0x029f9f (0x4e03 bytes) 000232 1787138812.430618 punch from 0x0280dc to 0x034926 (0xc84b bytes) 000233 1787138812.459281 MIRROR_RESYNC 4 mirrors 000234 1787138813.749470 read 0x007915 thru 0x0178bf (0xffab bytes) 000236 1787138813.759743 mapwrite 0x00b659 thru 0x014bc5 (0x956d bytes) 000237 1787138813.836970 trunc from 0x02e40c to 0x03774e (0x9343 bytes) 000238 1787138813.841698 trunc from 0x012bcb to 0x0222b5 (0xf6eb bytes) 000239 1787138813.847337 mapwrite 0x01814f thru 0x023e10 (0xbcc2 bytes) 000241 1787138813.898051 punch from 0x02b007 to 0x036d22 (0xbd1c bytes) 000242 1787138813.905558 punch from 0x012c90 to 0x0161fd (0x356e bytes) 000243 1787138813.914332 fallocate 0x035038 thru 0x035952 (0x91b bytes) 000244 1787138813.920811 punch from 0x01cc08 to 0x025a66 (0x8e5f bytes) 000246 1787138813.935598 mapwrite 0x02d126 thru 0x037cdd (0xabb8 bytes) 000247 1787138814.016725 mapwrite 0x008ae9 thru 0x00ecd9 (0x61f1 bytes) 000251 1787138814.064646 mapread 0x010f04 thru 0x0121a6 (0x12a3 bytes) 000252 1787138814.090137 mapread 0x002f8e thru 0x003df7 (0xe6a bytes) 000253 1787138814.113451 mapread 0x004868 thru 0x012d47 (0xe4e0 bytes) 000254 1787138814.133345 MIRROR_RESYNC 4 mirrors 000255 1787138816.385664 fallocate 0x03b3c6 thru 0x03ffff (0x4c3a bytes) 000256 1787138816.433570 punch from 0x01ec13 to 0x02a28e (0xb67c bytes) 000257 1787138816.442475 read 0x02917b thru 0x035e38 (0xccbe bytes) 000258 1787138816.472515 trunc from 0x03d761 to 0x03f956 (0x21f6 bytes) 000259 1787138816.489552 fallocate 0x03b4c0 thru 0x03ffff (0x4b40 bytes) 000260 1787138816.510162 MIRROR_RESYNC 4 mirrors 000261 1787138818.479058 read 0x037cba thru 0x03ffff (0x8346 bytes) 000263 1787138818.493303 read 0x032e3e thru 0x03e69c (0xb85f bytes) 000264 1787138818.533461 write 0x0064e7 thru 0x009f5a (0x3a74 bytes) 000265 1787138818.615700 mapwrite 0x008b40 thru 0x00b962 (0x2e23 bytes) 000266 1787138818.659733 mapwrite 0x01109c thru 0x015cc0 (0x4c25 bytes) 000268 1787138818.735475 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x46eb69d6 (stale) Mirror 3: 0x46eb69d6 (stale) Mirror 4: 0x46eb69d6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe2e66ea5 Mirror 5: 0xe2e66ea5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000269 1787138820.233177 MIRROR_RESYNC 5 mirrors 000270 1787138822.035468 fallocate 0x02a48d thru 0x0345b7 (0xa12b bytes) 000271 1787138822.167918 read 0x007ec4 thru 0x0125e4 (0xa721 bytes) 000272 1787138822.194458 read 0x005159 thru 0x005898 (0x740 bytes) 000273 1787138822.297490 trunc from 0x02b3f3 to 0x02e553 (0x3161 bytes) 000274 1787138822.316667 mapwrite 0x012910 thru 0x01efc2 (0xc6b3 bytes) 000275 1787138822.411330 fallocate 0x00e2f3 thru 0x01132f (0x303d bytes) 000276 1787138822.422114 trunc from 0x01ed6e to 0x026da3 (0x8036 bytes) 000277 1787138822.431309 fallocate 0x039559 thru 0x03b28e (0x1d36 bytes) 000278 1787138822.445967 write 0x019314 thru 0x01d947 (0x4634 bytes) 000279 1787138822.451186 trunc from 0x040000 to 0x03ce15 (0x31ea bytes) 000280 1787138822.514278 write 0x02327b thru 0x02a443 (0x71c9 bytes) 000281 1787138822.531525 fallocate 0x02c286 thru 0x037414 (0xb18f bytes) 000284 1787138822.564716 mapread 0x022be6 thru 0x02d8c6 (0xace1 bytes) 000285 1787138822.604256 read 0x02a5bb thru 0x0327b5 (0x81fb bytes) 000286 1787138822.612733 mapread 0x01d77d thru 0x02697b (0x91ff bytes) 000287 1787138822.614214 fallocate 0x03d0c0 thru 0x03fdcb (0x2d0c bytes) 000288 1787138822.620461 fallocate 0x001b36 thru 0x00d0fa (0xb5c5 bytes) 000289 1787138822.630987 fallocate 0x003c65 thru 0x00ba96 (0x7e32 bytes) 000290 1787138822.643146 trunc from 0x017a02 to 0x0248b8 (0xceb7 bytes) 000291 1787138822.674123 mapwrite 0x0319eb thru 0x032ef1 (0x1507 bytes) 000292 1787138822.700087 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fdcc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdcc): Mirror 2: 0x225516ca (stale) Mirror 3: 0x225516ca (stale) Mirror 4: 0x225516ca (stale) Mirror 5: 0x225516ca (stale) CRC-32 checksum value for chunk [0, 0x3fdcc): Mirror 1: 0xdc27996d Mirror 6: 0xdc27996d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdcc: skipped 000293 1787138824.075562 trunc from 0x03fdcc to 0x00841d (0x379ae bytes) 000294 1787138824.131242 trunc from 0x00841e to 0x013315 (0xaef8 bytes) 000295 1787138824.152080 trunc from 0x00c941 to 0x013315 (0x69d5 bytes) 000296 1787138824.158169 fallocate 0x00c557 thru 0x00e737 (0x21e1 bytes) 000297 1787138824.162506 trunc from 0x00a06f to 0x00f06f (0x5001 bytes) 000298 1787138824.170152 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x13316): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13316): Mirror 2: 0xa1aa59be (stale) Mirror 3: 0xa1aa59be (stale) Mirror 4: 0xa1aa59be (stale) Mirror 5: 0xa1aa59be (stale) Mirror 6: 0x31b51692 (stale) CRC-32 checksum value for chunk [0, 0x13316): Mirror 1: 0x584c7303 Mirror 7: 0x584c7303 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13316: skipped 000299 1787138825.133999 mapread 0x003a19 thru 0x00a199 (0x6781 bytes) 000300 1787138825.165642 read 0x00bcd0 thru 0x00bfb9 (0x2ea bytes) 000301 1787138825.172792 mapread 0x0065e6 thru 0x00dd29 (0x7744 bytes) 000302 1787138825.196412 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13316): Mirror 2: 0xa1aa59be (stale) Mirror 3: 0xa1aa59be (stale) Mirror 4: 0xa1aa59be (stale) Mirror 6: 0x31b51692 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13316): Mirror 2: 0xa1aa59be (stale) Mirror 3: 0xa1aa59be (stale) Mirror 4: 0xa1aa59be (stale) Mirror 6: 0x31b51692 (stale) CRC-32 checksum value for chunk [0, 0x13316): Mirror 1: 0x584c7303 Mirror 7: 0x584c7303 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13316: skipped 000303 1787138825.959963 read 0x006bfc thru 0x00b166 (0x456b bytes) 000305 1787138825.999309 trunc from 0x008e3d to 0x00eda1 (0x5f65 bytes) 000306 1787138826.077134 mapwrite 0x014b28 thru 0x01de74 (0x934d bytes) 000307 1787138826.144705 write 0x00073a thru 0x00b89e (0xb165 bytes) 000308 1787138826.157420 mapread 0x0171e4 thru 0x01b2b5 (0x40d2 bytes) 000309 1787138826.161368 write 0x03cf35 thru 0x03ffff (0x30cb bytes) 000310 1787138826.164302 mapwrite 0x011602 thru 0x0165d2 (0x4fd1 bytes) 000311 1787138826.201110 mapread 0x01736c thru 0x01c7d6 (0x546b bytes) 000312 1787138826.202639 write 0x0172d0 thru 0x01c6ca (0x53fb bytes) 000313 1787138826.216522 MIRROR_RESYNC 6 mirrors 000314 1787138828.186250 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) 000315 1787138828.771749 fallocate 0x037c47 thru 0x03a49c (0x2856 bytes) 000316 1787138828.860830 read 0x00278f thru 0x006398 (0x3c0a bytes) 000317 1787138828.889355 punch from 0x00d66f to 0x01a30a (0xcc9c bytes) 000318 1787138828.911690 MIRROR_RESYNC 5 mirrors 000319 1787138831.019731 MIRROR_RESYNC 5 mirrors 000320 1787138831.862056 trunc from 0x040000 to 0x02c60d (0x139f2 bytes) 000321 1787138831.928447 trunc from 0x010871 to 0x01eb40 (0xe2d0 bytes) 000322 1787138831.936356 punch from 0x01d7be to 0x0260c2 (0x8905 bytes) 000323 1787138831.943058 mapwrite 0x0107a8 thru 0x010cab (0x504 bytes) 000324 1787138831.967154 read 0x027feb thru 0x02c60d (0x4623 bytes) 000325 1787138831.976751 trunc from 0x02c60e to 0x021492 (0xb17b bytes) 000326 1787138832.011612 mapwrite 0x00a8dd thru 0x01810f (0xd833 bytes) 000327 1787138832.080508 fallocate 0x00a35a thru 0x00f134 (0x4ddb bytes) 000328 1787138832.093813 MIRROR_RESYNC 5 mirrors 000329 1787138833.026141 trunc from 0x021493 to 0x036162 (0x14cd0 bytes) 000330 1787138833.098868 read 0x01c4b9 thru 0x026b94 (0xa6dc bytes) 000331 1787138833.108940 fallocate 0x00b9fe thru 0x016de3 (0xb3e6 bytes) 000332 1787138833.120691 read 0x00e357 thru 0x01800c (0x9cb6 bytes) 000333 1787138833.141959 trunc from 0x036163 to 0x02d911 (0x8851 bytes) 000334 1787138833.207507 punch from 0x00ef0c to 0x01309e (0x4193 bytes) 000335 1787138833.219743 mapread 0x007d0c thru 0x00d519 (0x580e bytes) 000336 1787138833.221935 mapread 0x01798e thru 0x026257 (0xe8ca bytes) 000337 1787138833.230377 write 0x0213fa thru 0x0310d3 (0xfcda bytes) 000339 1787138833.241385 fallocate 0x0149d0 thru 0x016949 (0x1f7a bytes) 000340 1787138833.251825 fallocate 0x013211 thru 0x0182fd (0x50ed bytes) 000341 1787138833.277887 mapwrite 0x02d136 thru 0x032cf8 (0x5bc3 bytes) 000342 1787138833.333884 mapread 0x00edd5 thru 0x01de0a (0xf036 bytes) 000343 1787138833.384992 punch from 0x0132fb to 0x013e02 (0xb08 bytes) 000345 1787138833.393314 fallocate 0x018cd5 thru 0x028bd9 (0xff05 bytes) 000346 1787138833.407391 read 0x004765 thru 0x00b7eb (0x7087 bytes) 000347 1787138833.417342 mapwrite 0x03f7ff thru 0x03ffff (0x801 bytes) 000348 1787138833.478006 read 0x004889 thru 0x0104e0 (0xbc58 bytes) 000350 1787138833.480194 trunc from 0x040000 to 0x01fa13 (0x205ec bytes) 000351 1787138833.500127 write 0x00e8c3 thru 0x01e160 (0xf89e bytes) 000352 1787138833.502228 trunc from 0x00cb69 to 0x01a914 (0xddac bytes) 000353 1787138833.526024 write 0x00032d thru 0x0018f6 (0x15ca bytes) 000354 1787138833.527267 mapwrite 0x0300d5 thru 0x0387d8 (0x8704 bytes) 000355 1787138833.599289 punch from 0x0223e3 to 0x02737b (0x4f99 bytes) 000356 1787138833.617409 write 0x03a138 thru 0x03ffff (0x5ec8 bytes) 000357 1787138833.622625 write 0x00af4f thru 0x017864 (0xc916 bytes) 000358 1787138833.626418 trunc from 0x012c47 to 0x022738 (0xfaf2 bytes) 000359 1787138833.647866 mapread 0x0258cd thru 0x0269a7 (0x10db bytes) 000360 1787138833.663120 mapwrite 0x02d97a thru 0x032685 (0x4d0c bytes) 000361 1787138833.688939 mapwrite 0x03fdbd thru 0x03ffff (0x243 bytes) 000362 1787138833.708793 mapwrite 0x01e722 thru 0x020c93 (0x2572 bytes) 000363 1787138833.760073 write 0x015cdf thru 0x020d55 (0xb077 bytes) 000365 1787138833.772445 trunc from 0x02f247 to 0x03c904 (0xd6be bytes) 000366 1787138833.778382 mapread 0x00e976 thru 0x017409 (0x8a94 bytes) 000367 1787138833.801395 MIRROR_RESYNC 5 mirrors 000368 1787138835.781424 mapwrite 0x01f372 thru 0x02cb2b (0xd7ba bytes) 000370 1787138835.944269 read 0x02df50 thru 0x0352eb (0x739c bytes) 000371 1787138835.961387 fallocate 0x03a4c3 thru 0x03d9e1 (0x351f bytes) 000372 1787138835.988120 mapread 0x0211ca thru 0x02215b (0xf92 bytes) 000374 1787138836.007067 mapread 0x008adc thru 0x0147f8 (0xbd1d bytes) 000375 1787138836.066831 trunc from 0x032808 to 0x0365aa (0x3da3 bytes) 000376 1787138836.103302 MIRROR_RESYNC 5 mirrors 000378 1787138837.392518 mapread 0x00c23a thru 0x01b452 (0xf219 bytes) 000379 1787138837.424005 punch from 0x02c6cd to 0x033e6f (0x77a3 bytes) 000380 1787138837.467954 trunc from 0x0318d1 to 0x03bfa0 (0xa6d0 bytes) 000381 1787138837.474237 write 0x033936 thru 0x03ed0a (0xb3d5 bytes) 000382 1787138837.484713 punch from 0x00f65a to 0x01dc61 (0xe608 bytes) 000383 1787138837.491134 read 0x03b5d2 thru 0x03ffff (0x4a2e bytes) 000384 1787138837.501537 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8133a527 (stale) Mirror 4: 0x8133a527 (stale) Mirror 7: 0x8133a527 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8133a527 (stale) Mirror 4: 0x8133a527 (stale) Mirror 7: 0x8133a527 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000385 1787138837.921175 punch from 0x020460 to 0x02181d (0x13be bytes) 000386 1787138837.931600 punch from 0x021727 to 0x02a649 (0x8f23 bytes) 000387 1787138837.937293 mapwrite 0x032c73 thru 0x03f014 (0xc3a2 bytes) 000389 1787138837.992754 fallocate 0x012f26 thru 0x01cec6 (0x9fa1 bytes) 000391 1787138838.000374 trunc from 0x040000 to 0x03b901 (0x46fe bytes) 000392 1787138838.033810 trunc from 0x03b902 to 0x010484 (0x2b47d bytes) 000393 1787138838.052261 read 0x009549 thru 0x010484 (0x6f3c bytes) 000394 1787138838.069565 mapwrite 0x01ca8b thru 0x026e1f (0xa395 bytes) 000395 1787138838.168988 write 0x03f749 thru 0x03ffff (0x8b7 bytes) 000396 1787138838.170324 fallocate 0x008d7c thru 0x00b4c5 (0x274a bytes) 000398 1787138838.189154 write 0x02d1d3 thru 0x03258a (0x53b8 bytes) 000399 1787138838.201778 trunc from 0x040000 to 0x00ac6d (0x35392 bytes) 000400 1787138838.227212 punch from 0x000571 to 0x00ac6d (0xa6fd bytes) 000401 1787138838.233984 write 0x034d2f thru 0x035675 (0x947 bytes) 000402 1787138838.241372 trunc from 0x01083e to 0x01eacb (0xe28e bytes) 000403 1787138838.257073 fallocate 0x011926 thru 0x019007 (0x76e2 bytes) 000404 1787138838.268272 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35676): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35676): Mirror 3: 0x4954dd9d (stale) Mirror 4: 0x4954dd9d (stale) Mirror 7: 0x4954dd9d (stale) CRC-32 checksum value for chunk [0, 0x35676): Mirror 1: 0xae52d8d1 Mirror 8: 0xae52d8d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35676: skipped 000405 1787138839.258324 mapread 0x01966f thru 0x020d71 (0x7703 bytes) 000406 1787138839.316262 trunc from 0x035676 to 0x0164a8 (0x1f1cd bytes) 000407 1787138839.395079 MIRROR_RESYNC 5 mirrors 000408 1787138840.806969 MIRROR_EXTEND to 6 mirrors 000409 1787138841.866778 mapread 0x004e94 thru 0x013ba6 (0xed13 bytes) 000410 1787138841.919367 read 0x013cc3 thru 0x0164a8 (0x27e6 bytes) 000412 1787138841.921063 punch from 0x007edc to 0x00b63c (0x3761 bytes) 000413 1787138841.989745 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x164a9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x164a9): Mirror 3: 0x9834acff (stale) Mirror 4: 0x9834acff (stale) Mirror 7: 0x9834acff (stale) Mirror 8: 0x9834acff (stale) Mirror 9: 0x9834acff (stale) CRC-32 checksum value for chunk [0, 0x164a9): Mirror 1: 0x9834acff Mirror 10: 0x9834acff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x164a9: skipped 000415 1787138843.288008 mapwrite 0x0072ea thru 0x009b93 (0x28aa bytes) 000418 1787138843.408528 trunc from 0x0040d2 to 0x010220 (0xc14f bytes) 000419 1787138843.414361 write 0x034b85 thru 0x03a17a (0x55f6 bytes) 000420 1787138843.421528 fallocate 0x03022c thru 0x03619e (0x5f73 bytes) 000421 1787138843.428684 punch from 0x035ded to 0x03a17a (0x438e bytes) 000422 1787138843.443266 read 0x0003cf thru 0x0085a8 (0x81da bytes) 000423 1787138843.458738 punch from 0x03160b to 0x03a17a (0x8b70 bytes) 000424 1787138843.465939 write 0x001cd7 thru 0x00c93c (0xac66 bytes) 000425 1787138843.476406 read 0x034be1 thru 0x03a17a (0x559a bytes) 000426 1787138843.513174 punch from 0x02058f to 0x02d2f8 (0xcd6a bytes) 000427 1787138843.522718 fallocate 0x0164ba thru 0x0249a3 (0xe4ea bytes) 000428 1787138843.549741 trunc from 0x03a17b to 0x0130b2 (0x270c8 bytes) 000429 1787138843.614349 mapread 0x00ef2c thru 0x0130b2 (0x4187 bytes) 000430 1787138843.617484 mapwrite 0x03e191 thru 0x03ffff (0x1e6f bytes) 000431 1787138843.729247 trunc from 0x040000 to 0x0271cb (0x18e34 bytes) 000432 1787138843.773038 mapwrite 0x03deff thru 0x03ffff (0x2101 bytes) 000433 1787138843.890089 trunc from 0x040000 to 0x02bb0d (0x144f2 bytes) 000434 1787138843.906908 fallocate 0x02f711 thru 0x0320dc (0x29cc bytes) 000435 1787138843.917110 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bb0e): Mirror 3: 0x615ac8e0 (stale) Mirror 7: 0x615ac8e0 (stale) Mirror 8: 0x615ac8e0 (stale) Mirror 9: 0x615ac8e0 (stale) Mirror 10: 0x615ac8e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bb0e): Mirror 3: 0x615ac8e0 (stale) Mirror 7: 0x615ac8e0 (stale) Mirror 8: 0x615ac8e0 (stale) Mirror 9: 0x615ac8e0 (stale) Mirror 10: 0x615ac8e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bb0e: skipped 000436 1787138844.449555 write 0x01205a thru 0x01f951 (0xd8f8 bytes) 000437 1787138844.468327 write 0x0233a1 thru 0x02a489 (0x70e9 bytes) 000439 1787138844.505894 trunc from 0x02bb0e to 0x00ac40 (0x20ecd bytes) 000440 1787138844.539051 fallocate 0x02fe8d thru 0x031606 (0x177a bytes) 000441 1787138844.546892 mapread 0x02f9e7 thru 0x02ff96 (0x5b0 bytes) 000442 1787138844.564685 read 0x027d82 thru 0x028f42 (0x11c1 bytes) 000443 1787138844.575763 write 0x03da9b thru 0x03ffff (0x2565 bytes) 000444 1787138844.584595 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa2fe87c2 (stale) Mirror 7: 0xa2fe87c2 (stale) Mirror 8: 0xa2fe87c2 (stale) Mirror 9: 0xa2fe87c2 (stale) Mirror 10: 0xa2fe87c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4a14dfc1 Mirror 11: 0x4a14dfc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000445 1787138845.585336 mapread 0x02e6c9 thru 0x033e00 (0x5738 bytes) 000446 1787138845.638044 trunc from 0x040000 to 0x03475d (0xb8a2 bytes) 000447 1787138845.793537 read 0x007aff thru 0x00b3ee (0x38f0 bytes) 000448 1787138845.810522 read 0x016579 thru 0x022e42 (0xc8ca bytes) 000449 1787138845.836807 write 0x01daf1 thru 0x029335 (0xb845 bytes) 000450 1787138845.844939 mapwrite 0x00e906 thru 0x01a2de (0xb9d9 bytes) 000451 1787138845.874611 fallocate 0x012513 thru 0x0160a0 (0x3b8e bytes) 000452 1787138845.881551 mapwrite 0x026bdc thru 0x028850 (0x1c75 bytes) 000453 1787138845.899907 write 0x026315 thru 0x032655 (0xc341 bytes) 000454 1787138845.901529 punch from 0x00e0aa to 0x01acd7 (0xcc2e bytes) 000455 1787138845.916767 read 0x02c77c thru 0x02eef8 (0x277d bytes) 000456 1787138845.917934 punch from 0x022299 to 0x02fe05 (0xdb6d bytes) 000457 1787138845.966037 trunc from 0x03475e to 0x0293d6 (0xb387 bytes) 000458 1787138846.012675 mapwrite 0x0226cf thru 0x030280 (0xdbb2 bytes) 000459 1787138846.096783 read 0x001ace thru 0x002f83 (0x14b6 bytes) 000460 1787138846.098073 trunc from 0x030281 to 0x0178f5 (0x1898b bytes) 000461 1787138846.118252 mapwrite 0x03328c thru 0x03e993 (0xb708 bytes) 000462 1787138846.179766 punch from 0x00569b to 0x009544 (0x3eaa bytes) 000463 1787138846.188394 write 0x0315f2 thru 0x03f8c7 (0xe2d6 bytes) 000464 1787138846.193191 mapread 0x00c2d2 thru 0x0173dd (0xb10c bytes) 000465 1787138846.195354 mapread 0x016d54 thru 0x020468 (0x9715 bytes) 000466 1787138846.197183 mapread 0x025643 thru 0x02c8a5 (0x7263 bytes) 000467 1787138846.199351 trunc from 0x03f8c8 to 0x0162f9 (0x295ce bytes) 000468 1787138846.222098 write 0x02e12f thru 0x03d656 (0xf528 bytes) 000469 1787138846.226561 punch from 0x02f0bd to 0x034611 (0x5555 bytes) 000470 1787138846.244958 punch from 0x01ba58 to 0x01c1f9 (0x7a2 bytes) 000471 1787138846.251447 mapwrite 0x03a44d thru 0x03f978 (0x552c bytes) 000472 1787138846.303791 fallocate 0x032486 thru 0x034499 (0x2014 bytes) 000473 1787138846.311097 trunc from 0x03f979 to 0x00e303 (0x31675 bytes) 000475 1787138846.342161 trunc from 0x0083b7 to 0x00e303 (0x5f4d bytes) 000476 1787138846.348300 mapread 0x006486 thru 0x00e303 (0x7e7e bytes) 000477 1787138846.453687 mapwrite 0x033c9c thru 0x0388f1 (0x4c56 bytes) 000478 1787138846.571417 mapread 0x023daa thru 0x027752 (0x39a9 bytes) 000479 1787138846.610030 mapread 0x01acd1 thru 0x026ce1 (0xc011 bytes) 000480 1787138846.675065 punch from 0x019f0a to 0x02272a (0x8821 bytes) 000481 1787138846.684303 mapread 0x025a63 thru 0x02b3ff (0x599d bytes) 000482 1787138846.723898 fallocate 0x0241da thru 0x0268c5 (0x26ec bytes) 000483 1787138846.732021 mapwrite 0x0163dd thru 0x01ce8e (0x6ab2 bytes) 000484 1787138846.831699 write 0x018723 thru 0x01ec5a (0x6538 bytes) 000486 1787138846.846041 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x388f2): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x388f2): Mirror 3: 0x707f9b72 (stale) Mirror 7: 0x707f9b72 (stale) Mirror 8: 0x707f9b72 (stale) Mirror 9: 0x707f9b72 (stale) Mirror 10: 0x707f9b72 (stale) Mirror 11: 0x64a7b615 (stale) CRC-32 checksum value for chunk [0, 0x388f2): Mirror 1: 0xbded8ff9 Mirror 12: 0xbded8ff9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x388f2: skipped 000488 1787138848.056477 fallocate 0x0342a9 thru 0x03ffff (0xbd57 bytes) 000489 1787138848.129220 read 0x01f3f7 thru 0x02978e (0xa398 bytes) 000491 1787138848.159602 read 0x03c3b8 thru 0x03ffff (0x3c48 bytes) 000492 1787138848.191884 fallocate 0x00e0ac thru 0x01d4b6 (0xf40b bytes) 000493 1787138848.197589 mapwrite 0x010f94 thru 0x014727 (0x3794 bytes) 000495 1787138848.255317 trunc from 0x040000 to 0x003ba3 (0x3c45c bytes) 000497 1787138848.341311 punch from 0x0005c8 to 0x003ba3 (0x35dc bytes) 000498 1787138848.350296 trunc from 0x003ba4 to 0x004bc2 (0x101f bytes) 000499 1787138848.391065 trunc from 0x004128 to 0x004bc2 (0xa9b bytes) 000500 1787138848.415657 write 0x01e1f1 thru 0x02976b (0xb57b bytes) 000501 1787138848.431438 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2976c): Mirror 3: 0x7f075015 (stale) Mirror 7: 0x7f075015 (stale) Mirror 8: 0x7f075015 (stale) Mirror 9: 0x7f075015 (stale) Mirror 10: 0x7f075015 (stale) Mirror 11: 0x8d6e25af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2976c): Mirror 3: 0x7f075015 (stale) Mirror 7: 0x7f075015 (stale) Mirror 8: 0x7f075015 (stale) Mirror 9: 0x7f075015 (stale) Mirror 10: 0x7f075015 (stale) Mirror 11: 0x8d6e25af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2976c: skipped 000502 1787138849.154311 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2976c): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2976c): Mirror 3: 0x7f075015 (stale) Mirror 7: 0x7f075015 (stale) Mirror 8: 0x7f075015 (stale) Mirror 9: 0x7f075015 (stale) Mirror 10: 0x7f075015 (stale) Mirror 11: 0x8d6e25af (stale) CRC-32 checksum value for chunk [0, 0x2976c): Mirror 1: 0x4404c0d4 Mirror 12: 0x4404c0d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2976c: skipped 000503 1787138850.355737 trunc from 0x02976c to 0x009b44 (0x1fc27 bytes) 000505 1787138850.423282 trunc from 0x009b45 to 0x0219d4 (0x17e90 bytes) 000506 1787138850.452122 punch from 0x012176 to 0x01301c (0xea7 bytes) 000508 1787138850.461232 trunc from 0x0219d5 to 0x030fca (0xf5f6 bytes) 000509 1787138850.483055 mapwrite 0x031eaf thru 0x039eaa (0x7ffc bytes) 000510 1787138850.577021 read 0x035290 thru 0x039eaa (0x4c1b bytes) 000511 1787138850.580218 fallocate 0x003775 thru 0x00597d (0x2209 bytes) 000512 1787138850.589897 read 0x00f567 thru 0x012cd7 (0x3771 bytes) 000514 1787138850.639204 punch from 0x035c4f to 0x039eaa (0x425c bytes) 000515 1787138850.651223 punch from 0x02edf7 to 0x036e54 (0x805e bytes) 000516 1787138850.659223 mapwrite 0x02c886 thru 0x039c88 (0xd403 bytes) 000517 1787138850.767917 write 0x008e83 thru 0x01350e (0xa68c bytes) 000518 1787138850.773078 MIRROR_RESYNC 8 mirrors 000519 1787138853.781957 MIRROR_RESYNC 8 mirrors 000520 1787138854.617243 write 0x007754 thru 0x00edee (0x769b bytes) 000521 1787138854.665299 fallocate 0x00425f thru 0x005255 (0xff7 bytes) 000522 1787138854.671975 write 0x01f6b7 thru 0x028468 (0x8db2 bytes) 000524 1787138854.685007 punch from 0x002741 to 0x00afb2 (0x8872 bytes) 000525 1787138854.704156 punch from 0x02cdc5 to 0x03759b (0xa7d7 bytes) 000526 1787138854.709709 trunc from 0x039eab to 0x00bf3c (0x2df6e bytes) 000527 1787138854.740813 mapread 0x0024ea thru 0x003ca3 (0x17ba bytes) 000528 1787138854.755238 trunc from 0x00bf3d to 0x02b370 (0x1f434 bytes) 000529 1787138854.770441 mapwrite 0x01e054 thru 0x02860a (0xa5b7 bytes) 000530 1787138854.864411 mapread 0x023618 thru 0x02b370 (0x7d59 bytes) 000531 1787138854.876022 trunc from 0x02b371 to 0x02126d (0xa103 bytes) 000532 1787138854.937142 write 0x024cdd thru 0x0317cc (0xcaf0 bytes) 000533 1787138854.948183 write 0x015940 thru 0x016274 (0x935 bytes) 000535 1787138854.973998 trunc from 0x013947 to 0x01e426 (0xaae0 bytes) 000536 1787138855.013254 punch from 0x014c00 to 0x0163be (0x17bf bytes) 000537 1787138855.028706 mapwrite 0x01d949 thru 0x021209 (0x38c1 bytes) 000538 1787138855.142130 punch from 0x017e22 to 0x01c27d (0x445c bytes) 000539 1787138855.147999 fallocate 0x02b336 thru 0x02c9e1 (0x16ac bytes) 000540 1787138855.158117 trunc from 0x0317cd to 0x0327d3 (0x1007 bytes) 000541 1787138855.201607 write 0x0108ec thru 0x014f00 (0x4615 bytes) 000542 1787138855.221104 punch from 0x0277e8 to 0x02b066 (0x387f bytes) 000543 1787138855.227762 read 0x01ed25 thru 0x0239de (0x4cba bytes) 000544 1787138855.230118 mapwrite 0x00f7bc thru 0x018c55 (0x949a bytes) 000545 1787138855.308532 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x327d4): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x327d4): Mirror 3: 0x4193bfb6 (stale) Mirror 7: 0x4193bfb6 (stale) Mirror 8: 0x4193bfb6 (stale) Mirror 9: 0x4193bfb6 (stale) Mirror 10: 0x4193bfb6 (stale) Mirror 11: 0x4193bfb6 (stale) Mirror 12: 0x4193bfb6 (stale) CRC-32 checksum value for chunk [0, 0x327d4): Mirror 1: 0x6b1c4a66 Mirror 13: 0x6b1c4a66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x327d4: skipped 000547 1787138857.047526 trunc from 0x020c64 to 0x026b1d (0x5eba bytes) 000548 1787138857.134003 fallocate 0x00583c thru 0x011d9a (0xc55f bytes) 000550 1787138857.190806 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x327d4): Mirror 3: 0x4193bfb6 (stale) Mirror 7: 0x4193bfb6 (stale) Mirror 8: 0x4193bfb6 (stale) Mirror 10: 0x4193bfb6 (stale) Mirror 11: 0x4193bfb6 (stale) Mirror 12: 0x4193bfb6 (stale) Mirror 13: 0x6b1c4a66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x327d4): Mirror 3: 0x4193bfb6 (stale) Mirror 7: 0x4193bfb6 (stale) Mirror 8: 0x4193bfb6 (stale) Mirror 10: 0x4193bfb6 (stale) Mirror 11: 0x4193bfb6 (stale) Mirror 12: 0x4193bfb6 (stale) Mirror 13: 0x6b1c4a66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x327d4: skipped 000551 1787138857.823110 punch from 0x03106b to 0x0327d3 (0x1769 bytes) 000552 1787138857.839515 mapwrite 0x001a93 thru 0x0056df (0x3c4d bytes) 000553 1787138857.880713 mapwrite 0x00181b thru 0x005697 (0x3e7d bytes) 000554 1787138857.911106 read 0x005db7 thru 0x0157ab (0xf9f5 bytes) 000555 1787138857.926061 read 0x010caa thru 0x017db4 (0x710b bytes) 000556 1787138857.943995 mapread 0x00a050 thru 0x017118 (0xd0c9 bytes) 000557 1787138857.948076 mapwrite 0x036586 thru 0x03b285 (0x4d00 bytes) 000558 1787138858.025689 mapread 0x00e83f thru 0x010ba8 (0x236a bytes) 000559 1787138858.028009 trunc from 0x021806 to 0x0230f1 (0x18ec bytes) 000560 1787138858.038947 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b286): Mirror 3: 0xf7d78d6f (stale) Mirror 7: 0xf7d78d6f (stale) Mirror 10: 0xf7d78d6f (stale) Mirror 11: 0xf7d78d6f (stale) Mirror 12: 0xf7d78d6f (stale) Mirror 13: 0x1e7984da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b286): Mirror 3: 0xf7d78d6f (stale) Mirror 7: 0xf7d78d6f (stale) Mirror 10: 0xf7d78d6f (stale) Mirror 11: 0xf7d78d6f (stale) Mirror 12: 0xf7d78d6f (stale) Mirror 13: 0x1e7984da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b286: skipped 000561 1787138858.519957 fallocate 0x00e279 thru 0x013304 (0x508c bytes) 000562 1787138858.530646 MIRROR_RESYNC 7 mirrors 000563 1787138861.899220 read 0x029399 thru 0x02db42 (0x47aa bytes) 000564 1787138861.909952 punch from 0x0199f1 to 0x021b25 (0x8135 bytes) 000565 1787138861.943804 trunc from 0x03b286 to 0x02d5b7 (0xdcce bytes) 000566 1787138861.970312 trunc from 0x0243a3 to 0x02b51f (0x717d bytes) 000567 1787138861.976934 mapread 0x0249e6 thru 0x02c13c (0x7757 bytes) 000568 1787138862.004286 punch from 0x01df38 to 0x029b56 (0xbc1f bytes) 000569 1787138862.009724 trunc from 0x018348 to 0x01e199 (0x5e52 bytes) 000570 1787138862.016924 punch from 0x013759 to 0x020997 (0xd23f bytes) 000571 1787138862.024220 mapwrite 0x03906b thru 0x03ffff (0x6f95 bytes) 000573 1787138862.081922 read 0x027320 thru 0x02da1a (0x66fb bytes) 000574 1787138862.093887 fallocate 0x02709a thru 0x03461d (0xd584 bytes) 000576 1787138862.105895 mapread 0x011343 thru 0x012946 (0x1604 bytes) 000578 1787138862.110574 read 0x029abd thru 0x02cb0c (0x3050 bytes) 000579 1787138862.122685 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5a7ccc37 (stale) Mirror 7: 0x5a7ccc37 (stale) Mirror 10: 0x5a7ccc37 (stale) Mirror 11: 0x5a7ccc37 (stale) Mirror 12: 0x5a7ccc37 (stale) Mirror 13: 0x5a7ccc37 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x813bf6ee Mirror 14: 0x813bf6ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000580 1787138863.297490 mapwrite 0x022148 thru 0x027fae (0x5e67 bytes) 000581 1787138863.591256 mapread 0x018735 thru 0x01a908 (0x21d4 bytes) 000582 1787138863.624607 MIRROR_RESYNC 8 mirrors 000583 1787138867.023320 mapread 0x003217 thru 0x00daee (0xa8d8 bytes) 000584 1787138867.041723 mapwrite 0x01242e thru 0x014c28 (0x27fb bytes) 000585 1787138867.164605 punch from 0x01bc83 to 0x02169a (0x5a18 bytes) 000586 1787138867.201119 trunc from 0x027c05 to 0x036ed5 (0xf2d1 bytes) 000587 1787138867.232508 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc9944ae0 (stale) Mirror 7: 0xc9944ae0 (stale) Mirror 10: 0xc9944ae0 (stale) Mirror 11: 0xc9944ae0 (stale) Mirror 12: 0xc9944ae0 (stale) Mirror 13: 0xc9944ae0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc9944ae0 (stale) Mirror 7: 0xc9944ae0 (stale) Mirror 10: 0xc9944ae0 (stale) Mirror 11: 0xc9944ae0 (stale) Mirror 12: 0xc9944ae0 (stale) Mirror 13: 0xc9944ae0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000588 1787138867.988450 write 0x02831e thru 0x02f155 (0x6e38 bytes) 000589 1787138868.005299 mapread 0x0049bd thru 0x008018 (0x365c bytes) 000590 1787138868.029220 mapwrite 0x0039c2 thru 0x0112ce (0xd90d bytes) 000591 1787138868.063137 fallocate 0x025491 thru 0x02bacc (0x663c bytes) 000594 1787138868.071057 read 0x029b73 thru 0x02bbfd (0x208b bytes) 000595 1787138868.076939 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc9944ae0 (stale) Mirror 7: 0xc9944ae0 (stale) Mirror 11: 0xc9944ae0 (stale) Mirror 12: 0xc9944ae0 (stale) Mirror 13: 0xc9944ae0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc9944ae0 (stale) Mirror 7: 0xc9944ae0 (stale) Mirror 11: 0xc9944ae0 (stale) Mirror 12: 0xc9944ae0 (stale) Mirror 13: 0xc9944ae0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000596 1787138868.585205 write 0x0121bc thru 0x01bd01 (0x9b46 bytes) 000597 1787138868.606824 read 0x01915d thru 0x01ab45 (0x19e9 bytes) 000598 1787138868.607914 fallocate 0x031f5a thru 0x03708a (0x5131 bytes) 000599 1787138868.620274 trunc from 0x03ba63 to 0x03ffff (0x459d bytes) 000600 1787138868.626582 trunc from 0x02634d to 0x02a3c7 (0x407b bytes) 000601 1787138868.634023 trunc from 0x003924 to 0x007d3f (0x441c bytes) 000602 1787138868.640767 trunc from 0x040000 to 0x02900a (0x16ff5 bytes) 000603 1787138868.661125 read 0x02019f thru 0x02900a (0x8e6c bytes) 000604 1787138868.670778 mapwrite 0x036967 thru 0x03ffff (0x9699 bytes) 000605 1787138868.717246 mapread 0x034f6e thru 0x03ffff (0xb092 bytes) 000606 1787138868.725245 MIRROR_RESYNC 6 mirrors 000607 1787138870.479958 read 0x00e713 thru 0x01c2a5 (0xdb93 bytes) 000608 1787138870.491769 write 0x001fdf thru 0x00f48f (0xd4b1 bytes) 000609 1787138870.567633 trunc from 0x039d7e to 0x03a8fb (0xb7e bytes) 000610 1787138870.574763 write 0x03c4db thru 0x03ffff (0x3b25 bytes) 000611 1787138870.581273 write 0x026139 thru 0x033bd4 (0xda9c bytes) 000612 1787138870.593795 mapread 0x01faab thru 0x026cea (0x7240 bytes) 000613 1787138870.631254 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x17cb4f53 (stale) Mirror 7: 0x17cb4f53 (stale) Mirror 11: 0x17cb4f53 (stale) Mirror 12: 0x17cb4f53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x17cb4f53 (stale) Mirror 7: 0x17cb4f53 (stale) Mirror 11: 0x17cb4f53 (stale) Mirror 12: 0x17cb4f53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000614 1787138871.054227 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x17cb4f53 (stale) Mirror 7: 0x17cb4f53 (stale) Mirror 11: 0x17cb4f53 (stale) Mirror 12: 0x17cb4f53 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x517a12e7 Mirror 13: 0x517a12e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000615 1787138872.263939 read 0x0056e4 thru 0x00df6c (0x8889 bytes) 000616 1787138872.288692 mapwrite 0x02976b thru 0x036412 (0xcca8 bytes) 000617 1787138872.425747 trunc from 0x00313d to 0x0049c5 (0x1889 bytes) 000618 1787138872.433723 trunc from 0x03dcd0 to 0x03ffff (0x2330 bytes) 000619 1787138872.451817 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x17cb4f53 (stale) Mirror 11: 0x17cb4f53 (stale) Mirror 12: 0x17cb4f53 (stale) Mirror 13: 0x517a12e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x17cb4f53 (stale) Mirror 11: 0x17cb4f53 (stale) Mirror 12: 0x17cb4f53 (stale) Mirror 13: 0x517a12e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000621 1787138872.905985 read 0x0012d4 thru 0x00b773 (0xa4a0 bytes) 000622 1787138872.917613 read 0x012e6b thru 0x016c14 (0x3daa bytes) 000623 1787138872.939139 trunc from 0x03549b to 0x038374 (0x2eda bytes) 000624 1787138872.944216 write 0x00af75 thru 0x019541 (0xe5cd bytes) 000625 1787138872.949776 write 0x0078d9 thru 0x012087 (0xa7af bytes) 000626 1787138872.955978 trunc from 0x03bb9b to 0x03ffff (0x4465 bytes) 000628 1787138872.963143 trunc from 0x040000 to 0x011519 (0x2eae6 bytes) 000629 1787138872.983450 mapread 0x00462e thru 0x005661 (0x1034 bytes) 000630 1787138872.984930 read 0x007330 thru 0x00b69a (0x436b bytes) 000631 1787138872.992244 MIRROR_RESYNC 5 mirrors 000632 1787138873.710649 read 0x0046a1 thru 0x0054fb (0xe5b bytes) 000633 1787138873.715972 read 0x0048da thru 0x011519 (0xcc40 bytes) 000634 1787138873.725243 trunc from 0x01151a to 0x031e46 (0x2092d bytes) 000635 1787138873.768898 read 0x0273b8 thru 0x031e46 (0xaa8f bytes) 000636 1787138873.785475 write 0x00bb95 thru 0x0122bd (0x6729 bytes) 000637 1787138873.788059 fallocate 0x00514d thru 0x008c43 (0x3af7 bytes) 000639 1787138873.794710 fallocate 0x00cd72 thru 0x011e5e (0x50ed bytes) 000641 1787138873.801035 trunc from 0x018906 to 0x0255a4 (0xcc9f bytes) 000642 1787138873.806269 write 0x024c18 thru 0x0325bf (0xd9a8 bytes) 000643 1787138873.812556 trunc from 0x0325c0 to 0x03e3e5 (0xbe26 bytes) 000644 1787138873.834411 trunc from 0x011944 to 0x01a72c (0x8de9 bytes) 000645 1787138873.852016 trunc from 0x03e3e6 to 0x032a39 (0xb9ac bytes) 000647 1787138873.871449 mapwrite 0x033ae5 thru 0x03ffff (0xc51b bytes) 000648 1787138873.926094 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xd1990e76 (stale) Mirror 12: 0xd1990e76 (stale) Mirror 13: 0xd1990e76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xd1990e76 (stale) Mirror 12: 0xd1990e76 (stale) Mirror 13: 0xd1990e76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000649 1787138874.282825 fallocate 0x017b4b thru 0x0257ce (0xdc84 bytes) 000650 1787138874.293811 read 0x02af70 thru 0x02e0af (0x3140 bytes) 000651 1787138874.306652 trunc from 0x01d772 to 0x027fdf (0xa86e bytes) 000652 1787138874.313120 read 0x034645 thru 0x03b43f (0x6dfb bytes) 000653 1787138874.333033 trunc from 0x03daf6 to 0x03ffff (0x250a bytes) 000654 1787138874.339686 read 0x03a21a thru 0x03ffff (0x5de6 bytes) 000655 1787138874.345523 mapwrite 0x0044ee thru 0x00a31a (0x5e2d bytes) 000656 1787138874.368204 write 0x00947e thru 0x009fc9 (0xb4c bytes) 000657 1787138874.375266 write 0x018cdf thru 0x024e91 (0xc1b3 bytes) 000658 1787138874.378652 trunc from 0x040000 to 0x02a917 (0x156e8 bytes) 000659 1787138874.399402 punch from 0x00475c to 0x009238 (0x4add bytes) 000660 1787138874.419734 punch from 0x00c4be to 0x0147f4 (0x8337 bytes) 000661 1787138874.425619 fallocate 0x038409 thru 0x03fe25 (0x7a1d bytes) 000662 1787138874.430903 mapread 0x01110b thru 0x013f68 (0x2e5e bytes) 000663 1787138874.446406 trunc from 0x010ccb to 0x015b5b (0x4e91 bytes) 000664 1787138874.452565 trunc from 0x012769 to 0x01e047 (0xb8df bytes) 000665 1787138874.457930 punch from 0x00099f to 0x0065d0 (0x5c32 bytes) 000666 1787138874.463742 punch from 0x01268f to 0x01bb2e (0x94a0 bytes) 000667 1787138874.468949 fallocate 0x019b37 thru 0x02080e (0x6cd8 bytes) 000668 1787138874.473817 mapwrite 0x00cd32 thru 0x00d34a (0x619 bytes) 000669 1787138874.495950 fallocate 0x00574a thru 0x00e378 (0x8c2f bytes) 000670 1787138874.501242 trunc from 0x02a918 to 0x02865d (0x22ba bytes) 000672 1787138874.523671 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2865e): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2865e): Mirror 11: 0xba84b0fa (stale) Mirror 12: 0xba84b0fa (stale) Mirror 13: 0xba84b0fa (stale) CRC-32 checksum value for chunk [0, 0x2865e): Mirror 1: 0x74800410 Mirror 14: 0x74800410 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2865e: skipped 000673 1787138875.297815 punch from 0x00fa65 to 0x01076b (0xd07 bytes) 000674 1787138875.339691 mapread 0x008c5f thru 0x0109d1 (0x7d73 bytes) 000675 1787138875.358522 mapread 0x013550 thru 0x020bf6 (0xd6a7 bytes) 000676 1787138875.381748 fallocate 0x03cc32 thru 0x03ffff (0x33ce bytes) 000677 1787138875.386784 mapread 0x023f46 thru 0x02865d (0x4718 bytes) 000678 1787138875.394211 mapwrite 0x03b40f thru 0x03cd8d (0x197f bytes) 000679 1787138875.467233 punch from 0x002aa1 to 0x0037f5 (0xd55 bytes) 000680 1787138875.473393 mapread 0x0052c3 thru 0x0084ee (0x322c bytes) 000681 1787138875.479906 MIRROR_RESYNC 5 mirrors 000682 1787138877.605626 read 0x02210c thru 0x0236dc (0x15d1 bytes) 000683 1787138877.618667 trunc from 0x027904 to 0x02f526 (0x7c23 bytes) 000684 1787138877.678734 MIRROR_RESYNC 5 mirrors 000685 1787138879.525262 mapwrite 0x019fa0 thru 0x026a60 (0xcac1 bytes) 000686 1787138879.661778 mapwrite 0x02885a thru 0x03340e (0xabb5 bytes) 000687 1787138879.715524 read 0x0079c3 thru 0x00c61c (0x4c5a bytes) 000688 1787138879.736409 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3cd8e): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd8e): Mirror 11: 0x29c15e40 (stale) Mirror 12: 0x29c15e40 (stale) Mirror 13: 0x29c15e40 (stale) Mirror 14: 0x29c15e40 (stale) CRC-32 checksum value for chunk [0, 0x3cd8e): Mirror 1: 0x4ea0b936 Mirror 15: 0x4ea0b936 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd8e: skipped 000689 1787138880.965379 trunc from 0x039d84 to 0x03cd8d (0x300a bytes) 000691 1787138881.039187 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3cd8e): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd8e): Mirror 11: 0x29c15e40 (stale) Mirror 12: 0x29c15e40 (stale) Mirror 13: 0x29c15e40 (stale) Mirror 14: 0x29c15e40 (stale) Mirror 15: 0x4ea0b936 (stale) CRC-32 checksum value for chunk [0, 0x3cd8e): Mirror 1: 0xeff18ed7 Mirror 16: 0xeff18ed7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd8e: skipped 000693 1787138882.276358 MIRROR_RESYNC 7 mirrors 000694 1787138884.063443 trunc from 0x03cd8e to 0x02c8f4 (0x10499 bytes) 000695 1787138884.122638 mapwrite 0x01e62d thru 0x0225fa (0x3fce bytes) 000696 1787138884.178319 fallocate 0x028ff2 thru 0x0384c5 (0xf4d4 bytes) 000697 1787138884.184947 trunc from 0x0384c6 to 0x01a9fc (0x1dac9 bytes) 000698 1787138884.211292 punch from 0x00f764 to 0x01a9fc (0xb299 bytes) 000699 1787138884.220951 trunc from 0x002d0f to 0x01191b (0xec0d bytes) 000700 1787138884.228650 mapread 0x0189a5 thru 0x01a9fc (0x2058 bytes) 000701 1787138884.245938 read 0x003650 thru 0x009543 (0x5ef4 bytes) 000702 1787138884.274488 mapwrite 0x03cd87 thru 0x03f572 (0x27ec bytes) 000703 1787138884.361045 trunc from 0x014aec to 0x022369 (0xd87e bytes) 000704 1787138884.373664 mapwrite 0x00c0c2 thru 0x014aaf (0x89ee bytes) 000705 1787138884.466163 trunc from 0x03f573 to 0x021aab (0x1dac7 bytes) 000709 1787138884.504453 write 0x03021c thru 0x0376f2 (0x74d7 bytes) 000710 1787138884.511006 punch from 0x00e8e8 to 0x00fb83 (0x129c bytes) 000711 1787138884.519344 trunc from 0x0376f3 to 0x023768 (0x13f8a bytes) 000712 1787138884.553320 trunc from 0x023769 to 0x03d895 (0x1a12d bytes) 000713 1787138884.604494 write 0x007df5 thru 0x008ea5 (0x10b1 bytes) 000714 1787138884.606819 fallocate 0x02b66c thru 0x032323 (0x6cb8 bytes) 000715 1787138884.616326 write 0x033348 thru 0x03d30f (0x9fc8 bytes) 000716 1787138884.641313 punch from 0x03847f to 0x03ab9d (0x271f bytes) 000717 1787138884.661711 write 0x02ecdf thru 0x0381ff (0x9521 bytes) 000719 1787138884.676928 fallocate 0x005b23 thru 0x00e165 (0x8643 bytes) 000720 1787138884.685913 mapwrite 0x03e7b1 thru 0x03ffff (0x184f bytes) 000721 1787138884.742505 write 0x0121f0 thru 0x012e47 (0xc58 bytes) 000722 1787138884.744718 trunc from 0x02fa39 to 0x035c94 (0x625c bytes) 000723 1787138884.756253 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdaec89cc (stale) Mirror 12: 0xdaec89cc (stale) Mirror 13: 0xdaec89cc (stale) Mirror 14: 0xdaec89cc (stale) Mirror 15: 0xdaec89cc (stale) Mirror 16: 0xdaec89cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6481710e Mirror 17: 0x6481710e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1787138885.955987 fallocate 0x01dc36 thru 0x029309 (0xb6d4 bytes) 000725 1787138886.024480 read 0x037355 thru 0x03f779 (0x8425 bytes) 000726 1787138886.055346 mapread 0x0269e1 thru 0x0271ce (0x7ee bytes) 000728 1787138886.090018 read 0x0297e7 thru 0x02fb5e (0x6378 bytes) 000729 1787138886.101434 write 0x023c3a thru 0x02648d (0x2854 bytes) 000730 1787138886.110945 MIRROR_RESYNC 8 mirrors 000731 1787138889.052085 mapread 0x0010e9 thru 0x00bcd4 (0xabec bytes) 000732 1787138889.080433 punch from 0x021c73 to 0x0275d8 (0x5966 bytes) 000734 1787138889.143441 punch from 0x02a31f to 0x0355d4 (0xb2b6 bytes) 000735 1787138889.148817 punch from 0x01ab74 to 0x01b012 (0x49f bytes) 000736 1787138889.157526 mapread 0x0155b1 thru 0x0238e2 (0xe332 bytes) 000738 1787138889.193910 read 0x02259f thru 0x029b12 (0x7574 bytes) 000739 1787138889.207558 trunc from 0x0026a0 to 0x005f5b (0x38bc bytes) 000740 1787138889.214667 mapread 0x033d71 thru 0x038396 (0x4626 bytes) 000741 1787138889.222661 punch from 0x0076f1 to 0x00e53f (0x6e4f bytes) 000742 1787138889.230333 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x9adf2295 (stale) Mirror 12: 0x9adf2295 (stale) Mirror 13: 0x9adf2295 (stale) Mirror 14: 0x9adf2295 (stale) Mirror 15: 0x9adf2295 (stale) Mirror 17: 0x9adf2295 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x9adf2295 (stale) Mirror 12: 0x9adf2295 (stale) Mirror 13: 0x9adf2295 (stale) Mirror 14: 0x9adf2295 (stale) Mirror 15: 0x9adf2295 (stale) Mirror 17: 0x9adf2295 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000743 1787138889.735929 MIRROR_RESYNC 7 mirrors 000744 1787138891.970250 mapwrite 0x02174e thru 0x025c7b (0x452e bytes) 000745 1787138892.067623 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xcddd2ea7 (stale) Mirror 12: 0xcddd2ea7 (stale) Mirror 13: 0xcddd2ea7 (stale) Mirror 15: 0xcddd2ea7 (stale) Mirror 17: 0xcddd2ea7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xcddd2ea7 (stale) Mirror 12: 0xcddd2ea7 (stale) Mirror 13: 0xcddd2ea7 (stale) Mirror 15: 0xcddd2ea7 (stale) Mirror 17: 0xcddd2ea7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000746 1787138892.431637 mapwrite 0x00ad93 thru 0x00fe6f (0x50dd bytes) 000747 1787138892.466608 trunc from 0x040000 to 0x0357e6 (0xa819 bytes) 000748 1787138892.497810 write 0x03178a thru 0x039b1d (0x8394 bytes) 000749 1787138892.507238 write 0x03bfc3 thru 0x03ffff (0x403d bytes) 000750 1787138892.516612 trunc from 0x040000 to 0x01bda6 (0x24259 bytes) 000751 1787138892.542338 write 0x005fa9 thru 0x015de5 (0xfe3d bytes) 000753 1787138892.553109 mapwrite 0x00bb21 thru 0x00f8a3 (0x3d83 bytes) 000754 1787138892.586098 mapwrite 0x001864 thru 0x011624 (0xfdc1 bytes) 000755 1787138892.631476 mapread 0x00c32b thru 0x016cab (0xa981 bytes) 000756 1787138892.640893 punch from 0x008880 to 0x0097eb (0xf6c bytes) 000757 1787138892.646783 mapread 0x002088 thru 0x00e72e (0xc6a7 bytes) 000758 1787138892.654707 mapwrite 0x025cca thru 0x0271fb (0x1532 bytes) 000760 1787138892.701525 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x271fc): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x271fc): Mirror 11: 0x335ffdfc (stale) Mirror 12: 0x335ffdfc (stale) Mirror 13: 0x335ffdfc (stale) Mirror 15: 0x335ffdfc (stale) Mirror 17: 0x335ffdfc (stale) CRC-32 checksum value for chunk [0, 0x271fc): Mirror 1: 0x86b94ea2 Mirror 18: 0x86b94ea2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x271fc: skipped 000761 1787138893.690948 MIRROR_RESYNC 7 mirrors 000762 1787138894.973460 trunc from 0x0005a2 to 0x00670f (0x616e bytes) 000763 1787138895.010345 trunc from 0x0271fc to 0x019b1b (0xd6e0 bytes) 000764 1787138895.032489 punch from 0x0007e1 to 0x00e132 (0xd952 bytes) 000765 1787138895.038007 fallocate 0x018276 thru 0x018946 (0x6d1 bytes) 000766 1787138895.042777 mapwrite 0x0180dc thru 0x020e62 (0x8d87 bytes) 000767 1787138895.127216 mapread 0x0112a1 thru 0x011f15 (0xc75 bytes) 000768 1787138895.146985 MIRROR_RESYNC 7 mirrors 000769 1787138896.545414 trunc from 0x01b3da to 0x020e62 (0x5a89 bytes) 000770 1787138896.619628 fallocate 0x00b3cd thru 0x00f827 (0x445b bytes) 000771 1787138896.643820 MIRROR_RESYNC 7 mirrors 000773 1787138898.091584 trunc from 0x020e63 to 0x008a60 (0x18402 bytes) 000774 1787138898.140803 fallocate 0x03030d thru 0x037ce3 (0x79d7 bytes) 000775 1787138898.153147 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x8a61): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8a61): Mirror 11: 0x7f8d94c2 (stale) Mirror 12: 0x7f8d94c2 (stale) Mirror 13: 0x7f8d94c2 (stale) Mirror 15: 0x7f8d94c2 (stale) Mirror 17: 0x7f8d94c2 (stale) Mirror 18: 0x7f8d94c2 (stale) CRC-32 checksum value for chunk [0, 0x8a61): Mirror 1: 0x7f8d94c2 Mirror 19: 0x7f8d94c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8a61: skipped 000776 1787138899.112454 punch from 0x000b3d to 0x0073b1 (0x6875 bytes) 000777 1787138899.165352 fallocate 0x032b72 thru 0x03ffff (0xd48e bytes) 000778 1787138899.172318 read 0x029880 thru 0x02fbec (0x636d bytes) 000779 1787138899.187032 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xc610a0b9 (stale) Mirror 12: 0xc610a0b9 (stale) Mirror 13: 0xc610a0b9 (stale) Mirror 15: 0xc610a0b9 (stale) Mirror 17: 0xc610a0b9 (stale) Mirror 19: 0xa2fe87c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xc610a0b9 (stale) Mirror 12: 0xc610a0b9 (stale) Mirror 13: 0xc610a0b9 (stale) Mirror 15: 0xc610a0b9 (stale) Mirror 17: 0xc610a0b9 (stale) Mirror 19: 0xa2fe87c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000780 1787138899.679540 punch from 0x00fcc2 to 0x00fd0d (0x4c bytes) 000781 1787138899.689252 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xc610a0b9 (stale) Mirror 12: 0xc610a0b9 (stale) Mirror 13: 0xc610a0b9 (stale) Mirror 15: 0xc610a0b9 (stale) Mirror 17: 0xc610a0b9 (stale) Mirror 19: 0xa2fe87c2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa2fe87c2 Mirror 20: 0xa2fe87c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1787138900.645406 trunc from 0x040000 to 0x030c1f (0xf3e0 bytes) 000783 1787138900.685443 write 0x00f22b thru 0x0169ef (0x77c5 bytes) 000785 1787138900.697190 trunc from 0x030c20 to 0x016fc3 (0x19c5c bytes) 000786 1787138900.734086 mapwrite 0x00100b thru 0x00b602 (0xa5f8 bytes) 000788 1787138900.816507 trunc from 0x016fc4 to 0x03927a (0x222b7 bytes) 000789 1787138900.869524 mapread 0x033b01 thru 0x038fc3 (0x54c3 bytes) 000790 1787138900.889403 read 0x008c39 thru 0x00f477 (0x683f bytes) 000792 1787138900.890875 mapread 0x0114e8 thru 0x016d9c (0x58b5 bytes) 000793 1787138900.896232 mapwrite 0x03a3c0 thru 0x03f869 (0x54aa bytes) 000794 1787138900.974816 trunc from 0x03f86a to 0x01a8a6 (0x24fc3 bytes) 000796 1787138901.024267 trunc from 0x006ec9 to 0x011810 (0xa948 bytes) 000797 1787138901.037235 MIRROR_RESYNC 8 mirrors 000799 1787138903.050296 trunc from 0x004f89 to 0x00caed (0x7b65 bytes) 000800 1787138903.098515 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a8a7): Mirror 11: 0x9a4d0d3b (stale) Mirror 12: 0x9a4d0d3b (stale) Mirror 13: 0x9a4d0d3b (stale) Mirror 15: 0x9a4d0d3b (stale) Mirror 17: 0x9a4d0d3b (stale) Mirror 20: 0x9a4d0d3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a8a7): Mirror 11: 0x9a4d0d3b (stale) Mirror 12: 0x9a4d0d3b (stale) Mirror 13: 0x9a4d0d3b (stale) Mirror 15: 0x9a4d0d3b (stale) Mirror 17: 0x9a4d0d3b (stale) Mirror 20: 0x9a4d0d3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a8a7: skipped 000801 1787138903.594461 read 0x003157 thru 0x005bd5 (0x2a7f bytes) 000802 1787138903.600825 read 0x010f4e thru 0x01a8a6 (0x9959 bytes) 000803 1787138903.619802 MIRROR_RESYNC 7 mirrors 000804 1787138905.009991 punch from 0x0094ea to 0x011191 (0x7ca8 bytes) 000805 1787138905.044692 mapwrite 0x03a551 thru 0x03afb5 (0xa65 bytes) 000806 1787138905.090811 mapwrite 0x00b04a thru 0x0112aa (0x6261 bytes) 000807 1787138905.152229 write 0x0243fe thru 0x032091 (0xdc94 bytes) 000809 1787138905.176615 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3afb6): Mirror 11: 0xedf9976e (stale) Mirror 12: 0xedf9976e (stale) Mirror 13: 0xedf9976e (stale) Mirror 15: 0xedf9976e (stale) Mirror 20: 0xedf9976e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afb6): Mirror 11: 0xedf9976e (stale) Mirror 12: 0xedf9976e (stale) Mirror 13: 0xedf9976e (stale) Mirror 15: 0xedf9976e (stale) Mirror 20: 0xedf9976e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afb6: skipped 000810 1787138905.673703 punch from 0x02758b to 0x031c71 (0xa6e7 bytes) 000811 1787138905.679598 mapread 0x00a0ec thru 0x0103d1 (0x62e6 bytes) 000812 1787138905.711828 read 0x038510 thru 0x03afb5 (0x2aa6 bytes) 000813 1787138905.722519 mapwrite 0x03956d thru 0x03b265 (0x1cf9 bytes) 000814 1787138905.784618 punch from 0x01aab5 to 0x022b1f (0x806b bytes) 000815 1787138905.791707 trunc from 0x03b266 to 0x02b862 (0xfa03 bytes) 000816 1787138905.818965 mapwrite 0x029676 thru 0x02a29c (0xc27 bytes) 000818 1787138905.845701 punch from 0x025521 to 0x0278cd (0x23ad bytes) 000819 1787138905.859815 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b863): Mirror 12: 0xeae9aa05 (stale) Mirror 13: 0xeae9aa05 (stale) Mirror 15: 0xeae9aa05 (stale) Mirror 20: 0xeae9aa05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b863): Mirror 12: 0xeae9aa05 (stale) Mirror 13: 0xeae9aa05 (stale) Mirror 15: 0xeae9aa05 (stale) Mirror 20: 0xeae9aa05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b863: skipped 000820 1787138906.372895 fallocate 0x023cae thru 0x02a9ec (0x6d3f bytes) 000822 1787138906.381180 mapread 0x00ee91 thru 0x00f3e9 (0x559 bytes) 000823 1787138906.401551 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b863): Mirror 13: 0xeae9aa05 (stale) Mirror 15: 0xeae9aa05 (stale) Mirror 20: 0xeae9aa05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b863): Mirror 13: 0xeae9aa05 (stale) Mirror 15: 0xeae9aa05 (stale) Mirror 20: 0xeae9aa05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b863: skipped 000824 1787138906.755246 trunc from 0x027a30 to 0x02b862 (0x3e33 bytes) 000825 1787138906.760353 mapwrite 0x0287bb thru 0x02e216 (0x5a5c bytes) 000826 1787138906.812177 trunc from 0x02e217 to 0x03df01 (0xfceb bytes) 000827 1787138906.825314 fallocate 0x009bad thru 0x017127 (0xd57b bytes) 000828 1787138906.828725 mapwrite 0x02391a thru 0x02f868 (0xbf4f bytes) 000829 1787138906.864784 write 0x013163 thru 0x01dbda (0xaa78 bytes) 000830 1787138906.879175 fallocate 0x023252 thru 0x02ea36 (0xb7e5 bytes) 000831 1787138906.890816 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3df02): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df02): Mirror 13: 0x13fc6466 (stale) Mirror 15: 0x13fc6466 (stale) Mirror 20: 0x13fc6466 (stale) CRC-32 checksum value for chunk [0, 0x3df02): Mirror 1: 0x2c8f43e4 Mirror 21: 0x2c8f43e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df02: skipped 000832 1787138907.859350 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3df02): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df02): Mirror 13: 0x13fc6466 (stale) Mirror 15: 0x13fc6466 (stale) Mirror 20: 0x13fc6466 (stale) CRC-32 checksum value for chunk [0, 0x3df02): Mirror 1: 0x2c8f43e4 Mirror 21: 0x2c8f43e4 Mirror 22: 0x2c8f43e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df02: skipped 000833 1787138909.232628 fallocate 0x00b4d6 thru 0x00f03e (0x3b69 bytes) 000834 1787138909.302373 trunc from 0x03df02 to 0x030a2b (0xd4d6 bytes) 000835 1787138909.381927 mapread 0x006e26 thru 0x012972 (0xbb4d bytes) 000836 1787138909.438772 mapread 0x010a3e thru 0x01b4f2 (0xaab5 bytes) 000838 1787138909.458068 write 0x024913 thru 0x0267d5 (0x1ec3 bytes) 000839 1787138909.482792 trunc from 0x030a2c to 0x0125ad (0x1e47e bytes) 000840 1787138909.520710 mapwrite 0x0060bf thru 0x00caf6 (0x6a38 bytes) 000841 1787138909.552518 write 0x0328de thru 0x03ffff (0xd722 bytes) 000843 1787138909.565266 fallocate 0x01f079 thru 0x0235b0 (0x4538 bytes) 000844 1787138909.574234 punch from 0x0244dd to 0x032619 (0xe13d bytes) 000845 1787138909.600925 trunc from 0x03bb9f to 0x03bc82 (0xe4 bytes) 000846 1787138909.607538 punch from 0x023c0b to 0x024fec (0x13e2 bytes) 000847 1787138909.618952 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x9715b9e5 (stale) Mirror 15: 0x9715b9e5 (stale) Mirror 20: 0x9715b9e5 (stale) Mirror 21: 0xc40bd88e (stale) Mirror 22: 0xc40bd88e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x795994c8 Mirror 23: 0x795994c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000848 1787138910.757691 write 0x02050d thru 0x026f81 (0x6a75 bytes) 000849 1787138910.836684 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x9715b9e5 (stale) Mirror 15: 0x9715b9e5 (stale) Mirror 20: 0x9715b9e5 (stale) Mirror 21: 0xc40bd88e (stale) Mirror 22: 0xc40bd88e (stale) Mirror 23: 0x795994c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5d5ddc35 Mirror 24: 0x5d5ddc35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000850 1787138912.245929 fallocate 0x0198a0 thru 0x01bc27 (0x2388 bytes) 000851 1787138912.294376 fallocate 0x0129c6 thru 0x01cfa5 (0xa5e0 bytes) 000852 1787138912.309638 trunc from 0x040000 to 0x01d9af (0x22650 bytes) 000853 1787138912.375463 mapread 0x0035d5 thru 0x01213d (0xeb69 bytes) 000854 1787138912.437142 fallocate 0x0353a9 thru 0x0353e0 (0x38 bytes) 000855 1787138912.451015 punch from 0x01b00f to 0x01cd4a (0x1d3c bytes) 000856 1787138912.460152 trunc from 0x001cac to 0x009433 (0x7788 bytes) 000857 1787138912.467084 mapread 0x032752 thru 0x0353e0 (0x2c8f bytes) 000858 1787138912.488362 trunc from 0x0353e1 to 0x03869b (0x32bb bytes) 000859 1787138912.525592 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3869c): Mirror 13: 0x55e98154 (stale) Mirror 15: 0x55e98154 (stale) Mirror 21: 0x225139bf (stale) Mirror 22: 0x225139bf (stale) Mirror 23: 0x62f2e812 (stale) Mirror 24: 0x2d722bb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3869c): Mirror 13: 0x55e98154 (stale) Mirror 15: 0x55e98154 (stale) Mirror 21: 0x225139bf (stale) Mirror 22: 0x225139bf (stale) Mirror 23: 0x62f2e812 (stale) Mirror 24: 0x2d722bb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3869c: skipped 000860 1787138913.225371 write 0x0073d0 thru 0x011327 (0x9f58 bytes) 000861 1787138913.242652 read 0x00897a thru 0x011ed9 (0x9560 bytes) 000862 1787138913.244535 trunc from 0x015b4c to 0x023121 (0xd5d6 bytes) 000863 1787138913.251161 mapwrite 0x01fa3c thru 0x02d40f (0xd9d4 bytes) 000864 1787138913.322174 fallocate 0x021e37 thru 0x02eba2 (0xcd6c bytes) 000865 1787138913.333143 trunc from 0x03869c to 0x031c8d (0x6a0e bytes) 000866 1787138913.351749 punch from 0x015611 to 0x017311 (0x1d01 bytes) 000867 1787138913.356502 mapwrite 0x00a385 thru 0x00b887 (0x1503 bytes) 000868 1787138913.371148 trunc from 0x01fda9 to 0x01fedf (0x137 bytes) 000869 1787138913.377498 read 0x01b652 thru 0x027f6a (0xc919 bytes) 000870 1787138913.397271 trunc from 0x031c8e to 0x000ab3 (0x311da bytes) 000871 1787138913.421937 fallocate 0x02badd thru 0x0372ac (0xb7d0 bytes) 000872 1787138913.427582 trunc from 0x0003cf to 0x000ab3 (0x6e5 bytes) 000873 1787138913.433687 trunc from 0x000ab4 to 0x03715d (0x366aa bytes) 000875 1787138913.456224 punch from 0x02db56 to 0x02ff8c (0x2437 bytes) 000876 1787138913.462046 write 0x02f5a7 thru 0x036c87 (0x76e1 bytes) 000877 1787138913.473167 read 0x013182 thru 0x01c42e (0x92ad bytes) 000879 1787138913.490384 trunc from 0x03715e to 0x023ba3 (0x135ba bytes) 000880 1787138913.510701 fallocate 0x03b028 thru 0x03ffff (0x4fd8 bytes) 000881 1787138913.517232 write 0x03d741 thru 0x03ffff (0x28bf bytes) 000882 1787138913.520157 punch from 0x02d72d to 0x037842 (0xa116 bytes) 000883 1787138913.530054 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x9715b9e5 (stale) Mirror 15: 0x9715b9e5 (stale) Mirror 21: 0xc40bd88e (stale) Mirror 22: 0xc40bd88e (stale) Mirror 23: 0x795994c8 (stale) Mirror 24: 0x5d5ddc35 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5b5ce0b7 Mirror 25: 0x5b5ce0b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000884 1787138914.928242 read 0x038c72 thru 0x03ffff (0x738e bytes) 000885 1787138914.951111 write 0x01d59c thru 0x02035e (0x2dc3 bytes) 000886 1787138915.040584 MIRROR_RESYNC 8 mirrors 000887 1787138917.456190 punch from 0x03e225 to 0x03f391 (0x116d bytes) 000888 1787138917.532452 mapwrite 0x015da7 thru 0x01ec8f (0x8ee9 bytes) 000889 1787138917.609700 trunc from 0x01dc5c to 0x02d760 (0xfb05 bytes) 000890 1787138917.618044 trunc from 0x040000 to 0x02cdd7 (0x13228 bytes) 000892 1787138917.659377 write 0x02f851 thru 0x034ab1 (0x5261 bytes) 000893 1787138917.671258 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34ab2): Mirror 13: 0x2d20ec59 (stale) Mirror 15: 0x2d20ec59 (stale) Mirror 21: 0x2d20ec59 (stale) Mirror 23: 0x2d20ec59 (stale) Mirror 24: 0x2d20ec59 (stale) Mirror 25: 0x2d20ec59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ab2): Mirror 13: 0x2d20ec59 (stale) Mirror 15: 0x2d20ec59 (stale) Mirror 21: 0x2d20ec59 (stale) Mirror 23: 0x2d20ec59 (stale) Mirror 24: 0x2d20ec59 (stale) Mirror 25: 0x2d20ec59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ab2: skipped 000894 1787138918.351468 punch from 0x030347 to 0x034ab1 (0x476b bytes) 000895 1787138918.358566 write 0x014b43 thru 0x016c1b (0x20d9 bytes) 000896 1787138918.378581 fallocate 0x0250a9 thru 0x02b10b (0x6063 bytes) 000897 1787138918.387621 punch from 0x027f1d to 0x034ab1 (0xcb95 bytes) 000898 1787138918.393614 punch from 0x029bc0 to 0x034ab1 (0xaef2 bytes) 000899 1787138918.398165 fallocate 0x0316c5 thru 0x03ffff (0xe93b bytes) 000900 1787138918.404092 write 0x019189 thru 0x01d7f6 (0x466e bytes) 000901 1787138918.413636 mapwrite 0x0084cd thru 0x009196 (0xcca bytes) 000902 1787138918.450380 mapwrite 0x019ebc thru 0x0237ee (0x9933 bytes) 000903 1787138918.489202 read 0x02476f thru 0x02af9f (0x6831 bytes) 000904 1787138918.501062 fallocate 0x021dfc thru 0x02b5a4 (0x97a9 bytes) 000905 1787138918.507947 punch from 0x021f71 to 0x0295a6 (0x7636 bytes) 000907 1787138918.517686 mapread 0x00ab73 thru 0x00d033 (0x24c1 bytes) 000908 1787138918.528577 trunc from 0x0201f4 to 0x028329 (0x8136 bytes) 000909 1787138918.541561 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34ab2): Mirror 13: 0x2d20ec59 (stale) Mirror 21: 0x2d20ec59 (stale) Mirror 23: 0x2d20ec59 (stale) Mirror 24: 0x2d20ec59 (stale) Mirror 25: 0x2d20ec59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ab2): Mirror 13: 0x2d20ec59 (stale) Mirror 21: 0x2d20ec59 (stale) Mirror 23: 0x2d20ec59 (stale) Mirror 24: 0x2d20ec59 (stale) Mirror 25: 0x2d20ec59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ab2: skipped 000910 1787138919.004618 mapwrite 0x021493 thru 0x02b9da (0xa548 bytes) 000911 1787138919.054351 mapread 0x00354b thru 0x0104e5 (0xcf9b bytes) 000912 1787138919.068996 read 0x01c336 thru 0x02b0a9 (0xed74 bytes) 000913 1787138919.070216 write 0x011e14 thru 0x01e943 (0xcb30 bytes) 000914 1787138919.084311 write 0x0033df thru 0x00b176 (0x7d98 bytes) 000915 1787138919.086338 trunc from 0x034ab2 to 0x009b9e (0x2af13 bytes) 000916 1787138919.109580 mapwrite 0x03715e thru 0x03868c (0x152f bytes) 000917 1787138919.173091 mapwrite 0x035765 thru 0x03ea0f (0x92ab bytes) 000918 1787138919.225405 trunc from 0x03ea10 to 0x008d55 (0x35cba bytes) 000919 1787138919.253330 punch from 0x001c39 to 0x003883 (0x1c4b bytes) 000920 1787138919.261214 write 0x03fb7e thru 0x03ffff (0x482 bytes) 000921 1787138919.262414 read 0x014d4b thru 0x0156da (0x990 bytes) 000922 1787138919.274633 punch from 0x01eeb1 to 0x0265ed (0x773d bytes) 000923 1787138919.280497 fallocate 0x0180c7 thru 0x018bad (0xae7 bytes) 000924 1787138919.292969 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x38b2b00 (stale) Mirror 23: 0x38b2b00 (stale) Mirror 24: 0x38b2b00 (stale) Mirror 25: 0x38b2b00 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x38b2b00 (stale) Mirror 23: 0x38b2b00 (stale) Mirror 24: 0x38b2b00 (stale) Mirror 25: 0x38b2b00 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000925 1787138919.763292 read 0x03e89c thru 0x03ffff (0x1764 bytes) 000926 1787138919.774291 punch from 0x03afad to 0x03ffff (0x5053 bytes) 000927 1787138919.789928 fallocate 0x021a22 thru 0x0314e0 (0xfabf bytes) 000928 1787138919.798577 mapwrite 0x028369 thru 0x02ff94 (0x7c2c bytes) 000929 1787138919.844318 mapread 0x008c93 thru 0x00f504 (0x6872 bytes) 000930 1787138919.866060 fallocate 0x019289 thru 0x0250ae (0xbe26 bytes) 000931 1787138919.874322 write 0x03434e thru 0x03ffff (0xbcb2 bytes) 000932 1787138919.888612 trunc from 0x019331 to 0x01c1d1 (0x2ea1 bytes) 000933 1787138919.896396 write 0x02a8c1 thru 0x03948a (0xebca bytes) 000934 1787138919.905375 MIRROR_RESYNC 5 mirrors 000935 1787138921.502483 trunc from 0x040000 to 0x03571c (0xa8e3 bytes) 000936 1787138921.600319 mapread 0x0068b3 thru 0x00a768 (0x3eb6 bytes) 000937 1787138921.639870 fallocate 0x0392e7 thru 0x03ffff (0x6d19 bytes) 000938 1787138921.647824 read 0x00beba thru 0x014235 (0x837c bytes) 000939 1787138921.667539 MIRROR_RESYNC 5 mirrors 000940 1787138923.273883 MIRROR_RESYNC 5 mirrors 000941 1787138923.837053 mapread 0x01c560 thru 0x02503c (0x8add bytes) 000942 1787138923.859785 trunc from 0x03571d to 0x03d4c6 (0x7daa bytes) 000943 1787138923.908488 trunc from 0x028ddc to 0x02a178 (0x139d bytes) 000944 1787138923.919424 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4c7): Mirror 23: 0x626cf510 (stale) Mirror 24: 0x626cf510 (stale) Mirror 25: 0x626cf510 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4c7): Mirror 23: 0x626cf510 (stale) Mirror 24: 0x626cf510 (stale) Mirror 25: 0x626cf510 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4c7: skipped 000945 1787138924.409068 fallocate 0x0171d4 thru 0x01e8e1 (0x770e bytes) 000946 1787138924.418589 read 0x00d92d thru 0x00ed7f (0x1453 bytes) 000948 1787138924.427515 write 0x034b50 thru 0x03b69b (0x6b4c bytes) 000949 1787138924.446167 write 0x023560 thru 0x02c366 (0x8e07 bytes) 000950 1787138924.463039 MIRROR_RESYNC 4 mirrors 000951 1787138925.581151 trunc from 0x006a93 to 0x010992 (0x9f00 bytes) 000952 1787138925.621424 punch from 0x015f34 to 0x01c623 (0x66f0 bytes) 000954 1787138925.630453 mapwrite 0x031242 thru 0x03ffff (0xedbe bytes) 000955 1787138925.725615 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x27b12f38 (stale) Mirror 24: 0x27b12f38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 23: 0x27b12f38 (stale) Mirror 24: 0x27b12f38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000956 1787138926.033414 fallocate 0x011002 thru 0x01946f (0x846e bytes) 000957 1787138926.040757 fallocate 0x012d96 thru 0x0183a0 (0x560b bytes) 000958 1787138926.046509 write 0x007ed5 thru 0x00fa9c (0x7bc8 bytes) 000959 1787138926.060865 mapwrite 0x019d08 thru 0x01d5e0 (0x38d9 bytes) 000960 1787138926.096509 punch from 0x01d6d7 to 0x021b56 (0x4480 bytes) 000961 1787138926.102137 mapread 0x034455 thru 0x03e86b (0xa417 bytes) 000962 1787138926.115739 punch from 0x007eb7 to 0x00aca1 (0x2deb bytes) 000963 1787138926.121385 trunc from 0x0094f2 to 0x00dfc0 (0x4acf bytes) 000964 1787138926.126511 fallocate 0x006d9d thru 0x00d5bc (0x6820 bytes) 000967 1787138926.132155 write 0x03183c thru 0x033717 (0x1edc bytes) 000968 1787138926.134383 write 0x015a8e thru 0x018c1c (0x318f bytes) 000969 1787138926.135584 write 0x03b6c6 thru 0x03ffff (0x493a bytes) 000970 1787138926.137451 mapread 0x0346c4 thru 0x03c81f (0x815c bytes) 000971 1787138926.138963 mapread 0x017587 thru 0x0226bc (0xb136 bytes) 000972 1787138926.147742 mapwrite 0x01b173 thru 0x01e770 (0x35fe bytes) 000973 1787138926.167577 trunc from 0x01cb91 to 0x02adaf (0xe21f bytes) 000974 1787138926.173312 mapwrite 0x008083 thru 0x009824 (0x17a2 bytes) 000975 1787138926.191328 read 0x03e0e7 thru 0x03ffff (0x1f19 bytes) 000977 1787138926.192454 trunc from 0x040000 to 0x01eca9 (0x21356 bytes) 000978 1787138926.206667 trunc from 0x01ecaa to 0x013b2f (0xb17a bytes) 000979 1787138926.224709 punch from 0x00310c to 0x0076b7 (0x45ac bytes) 000980 1787138926.234831 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13b30): Mirror 24: 0xe33f464c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b30): Mirror 24: 0xe33f464c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b30: skipped 000981 1787138926.443796 write 0x019bd0 thru 0x0258a9 (0xbcda bytes) 000982 1787138926.452949 mapwrite 0x02eb40 thru 0x039873 (0xad34 bytes) 000983 1787138926.549318 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39874): Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39874): Mirror 24: 0x236e7158 (stale) CRC-32 checksum value for chunk [0, 0x39874): Mirror 1: 0xc1d4186b Mirror 25: 0xc1d4186b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39874: skipped 000984 1787138927.926222 fallocate 0x007502 thru 0x00eebe (0x79bd bytes) 000985 1787138927.991485 punch from 0x0241f0 to 0x02533b (0x114c bytes) 000987 1787138928.015281 punch from 0x00d197 to 0x01a8f5 (0xd75f bytes) 000989 1787138928.027510 trunc from 0x039874 to 0x011347 (0x2852c bytes) 000990 1787138928.082407 trunc from 0x00248a to 0x00fd3c (0xd8b3 bytes) 000991 1787138928.089776 mapread 0x009e0a thru 0x011347 (0x753e bytes) 000993 1787138928.116969 trunc from 0x011348 to 0x038130 (0x26de9 bytes) 000994 1787138928.137536 read 0x01ba6b thru 0x01df8d (0x2523 bytes) 000995 1787138928.146954 trunc from 0x017877 to 0x017b4e (0x2d8 bytes) 000996 1787138928.153405 mapwrite 0x006e58 thru 0x01578c (0xe935 bytes) 000997 1787138928.188155 fallocate 0x01c71d thru 0x01db36 (0x141a bytes) 000998 1787138928.194139 fallocate 0x0155a6 thru 0x01a76d (0x51c8 bytes) 000999 1787138928.200477 fallocate 0x00664e thru 0x0148c6 (0xe279 bytes) 001000 1787138928.206126 write 0x014bcc thru 0x01fda6 (0xb1db bytes) All operations completed A-OK! PASS 70a (143s) == sanity-flr test 71: check mirror extend parent fid ==== 07:28:54 (1787138934) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 07:29:06 (1787138946) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.42123 s, 5.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886399 s, 11.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x358:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x359:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x358:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35b:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 07:29:18 (1787138958) Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre2 Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5376224 .. Extending file size to 238080 .. Extending file size to 2980000 .. Extending file size to 8066528 .. Extending file size to 7358304 .. Extending file size to 9227104 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8268960 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8587968 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61530 61531 61534 61535 61536 192.168.202.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg241-client.virtnet /mnt/lustre2 (opts:) 192.168.202.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg241-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:30:46 (1787139046) Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre2 Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1731680 .. Extending file size to 7916864 .. failed Extending file size to 6034656 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8765280 .. Extending file size to 6379008 .. Extending file size to 6902880 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8605216 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8791360 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63041 63042 63044 63046 63048 done 192.168.202.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg241-client.virtnet /mnt/lustre2 (opts:) 192.168.202.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg241-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:32:10 (1787139130) Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2684 1268620 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18428 3588592 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23748 3583272 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42176 7171864 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2684 1268620 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18428 3588592 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23748 3583272 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42176 7171864 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195118 s, 10.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225721 s, 9.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4fdf53cb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg241-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:32:32 (1787139152) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186779 s, 11.2 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 07:32:40 (1787139160) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x24a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x36e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x36e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x36f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x370:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:32:49 (1787139169) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0904438 s, 23.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:32:57 (1787139177) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0967288 s, 21.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x371:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x371:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x371:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x371:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x372:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:33:06 (1787139186) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133202 s, 15.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x373:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x373:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x374:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:33:41 (1787139221) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg241-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948933 s, 11.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:34:33 (1787139273) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg241-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x373:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x374:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19852 s, 10.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x373:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x374:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x375:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x373:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x374:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x375:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:35:29 (1787139329) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg241-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg241-server: Pool lustre.test_204f created Adding targets to pool oleg241-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg241-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123492 s, 17.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x375:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x376:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x375:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x376:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg241-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg241-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg241-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:36:42 (1787139402) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:36:51 (1787139411) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:36:59 (1787139419) oleg241-server: Pool lustre.test_206 created Adding targets to pool oleg241-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg241-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg241-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg241-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg241-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:37:28 (1787139448) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145288 s, 7.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723511 s, 14.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x98ad84c3 Mirror 3: 0x98ad84c3 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:37:40 (1787139460) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:37:44 (1787139464) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:37:48 (1787139468) Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0954807 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2540000 Read this (4 bytes): BBBB 192.168.202.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg241-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:38:29 (1787139509) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0421749 s, 194 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:38:36 (1787139516) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683371 s, 15.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x26b lmm_fid: [0x200000405:0x26b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x26b lmm_fid: [0x200000405:0x26b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x380:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611755 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 07:38:38.000000000 -0400 Modify: 2026-08-19 07:38:38.000000000 -0400 Change: 2026-08-19 07:38:39.000000000 -0400 Birth: 2026-08-19 07:38:37.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:38:44 (1787139524) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366769 s, 28.6 MB/s oleg241-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg241-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg241-server: O/280000401/d0/380: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:39:10 (1787139550) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.308505 s, 133 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0297319 s, 138 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 07:39 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x270:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 07:39 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:39:19 (1787139559) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:39:28 (1787139568) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1916 sec =========== 07:39:38 (1787139578) === sanity-flr: start cleanup 07:39:39 (1787139579) === === sanity-flr: finish cleanup 07:39:50 (1787139590) ===