-----============= acceptance-small: sanity-flr ============----- Tue Jun 23 08:19:34 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg114-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:20:02 (1782217202) === oleg114-client.virtnet: executing check_config_client /mnt/lustre oleg114-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg114-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9debc3fad000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9debc3fad000.idle_timeout=debug disable quota as required oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:21:00 (1782217260) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8221] == sanity-flr test 0a: lfs mirror create with -N option == 08:21:04 (1782217264) PASS 0a (25s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:21:29 (1782217289) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:21:35 (1782217295) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:21:40 (1782217300) PASS 0d (43s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:22:23 (1782217343) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:22:28 (1782217348) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:22:33 (1782217353) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9debc3fad000.stats=clear osc.lustre-OST0001-osc-ffff9debc3fad000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:22:42 (1782217362) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:22:56 (1782217376) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:23:07 (1782217387) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:23:13 (1782217393) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (13s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:23:25 (1782217405) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (14s) == sanity-flr test 2: create components from existing files ========================================================== 08:23:39 (1782217419) PASS 2 (11s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:23:50 (1782217430) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:24:01 (1782217441) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:24:13 (1782217453) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (11s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:24:28 (1782217468) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:24:38 (1782217478) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.940541 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263332 s, 4.0 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:24:51 (1782217491) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.736665 s, 18.5 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:25:03 (1782217503) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (16s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:25:19 (1782217519) Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (115s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:27:15 (1782217635) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.226329 s, 13.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (54s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:28:10 (1782217690) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (112s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:30:02 (1782217802) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.62496 s, 11.3 MB/s -rw-r--r-- 1 root root 52428800 Jun 23 08:30 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.646s user 0m0.000s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9debc3fad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m2.054s user 0m0.004s sys 0m0.238s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg114-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg114-server oleg114-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.494s user 0m0.001s sys 0m0.246s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg114-server: oleg114-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg114-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg114-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9debc3fad000.ost_server_uuid in FULL state after 0 sec PASS 33b (139s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:32:21 (1782217941) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:32:27 (1782217947) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (6s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:32:32 (1782217952) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 08:32:37 (1782217957) PASS 35 (12s) == sanity-flr test 36a: write to mirrored files ========== 08:32:49 (1782217969) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.65862 s, 19.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (22s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:33:11 (1782217991) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481953 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119201 s, 8.8 MB/s PASS 36b (12s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:33:23 (1782218003) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:33:35 (1782218015) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0021513 s, 476 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00203344 s, 504 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00672949 s, 152 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00464154 s, 221 kB/s flv=10 olv=10 PASS 36d (17s) == sanity-flr test 37: mirror I/O API verification ======= 08:33:52 (1782218032) create test files with size 15749229 .. 0fa66c1cb166c36db1ee4ab3639aecc0 - b5f34acf65c50a9a5e97a4a0602f34b9 - edad23c21df320512fe91a1ec5434249 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre PASS 37 (37s) == sanity-flr test 38: resync ============================ 08:34:29 (1782218069) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (24s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:34:54 (1782218094) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:35:04 (1782218104) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111492 s, 9.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293597 s, 35.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (14s) == sanity-flr test 41: lfs mirror resync check =========== 08:35:18 (1782218118) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204571 s, 10.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.589631 s, 7.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.325139 s, 6.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (26s) == sanity-flr test 42: lfs mirror verify ================= 08:35:44 (1782218144) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 08:35:50 (1782218150) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:35:55 (1782218155) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443028 s, 23.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (12s) == sanity-flr test 44a: lfs mirror split check =========== 08:36:08 (1782218168) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.408349 s, 7.7 MB/s PASS 44a (15s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:36:23 (1782218183) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (13s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:36:38 (1782218198) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.497698 s, 21.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:36:55 (1782218215) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9debc6c52000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.511766 s, 20.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg114-client.virtnet /mnt/lustre (opts:) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9debc6792000.xattr_cache=1 PASS 44e (24s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:37:20 (1782218240) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (65s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:38:25 (1782218305) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (60s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:39:26 (1782218366) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 08:39:32 (1782218372) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.433836 s, 7.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.320486 s, 9.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xdb74f5e4 (nosync,stale) Mirror 2: 0xdb74f5e4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 603fb2e9fdc8281d3053a69a282be175 - mirror 1 checksum: 42ded4a8726608f6feebea6b41087d29 - mirror 2 checksum: 603fb2e9fdc8281d3053a69a282be175 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T08:39:35-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T08:39:35-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 603fb2e9fdc8281d3053a69a282be175 - mirror 1 checksum: 42ded4a8726608f6feebea6b41087d29 - mirror 2 checksum: 42ded4a8726608f6feebea6b41087d29 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T08:39:35-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T08:39:35-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (17s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:39:49 (1782218389) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631501 s, 16.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (14s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:40:03 (1782218403) Failing mds1 on oleg114-server Stopping /mnt/lustre-mds1 (opts:) on oleg114-server 08:40:19 (1782218419) shut down facet: mds1 facet_host: oleg114-server facet_failover_host: oleg114-server Failover mds1 to oleg114-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 08:40:55 (1782218455) targets are mounted 08:40:55 (1782218455) facet_failover done oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207507 s, 50.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (76s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:41:20 (1782218480) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0724444 s, 283 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0329553 s, 621 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (16s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:41:36 (1782218496) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0908712 s, 225 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0142072 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (16s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:41:52 (1782218512) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00878014 s, 1.9 MB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:42:04 (1782218524) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0994932 s, 206 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (13s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:42:17 (1782218537) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0138664 s, 295 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0645204 s, 63.5 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:42:29 (1782218549) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (37s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:43:06 (1782218586) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (37s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:43:43 (1782218623) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (30s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:44:14 (1782218654) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (29s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:44:43 (1782218683) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (13s) == sanity-flr test 70: mirror create and split race ====== 08:44:56 (1782218696) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57447 mirror split pid 57449 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57447 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57449 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (73s) == sanity-flr test 70a: flr mode fsx test ================ 08:46:09 (1782218769) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 8772 000002 1782218773.510967 mapwrite 0x0165d8 thru 0x01c795 (0x61be bytes) truncating to largest ever: 0xee14 000003 1782218773.649499 trunc from 0x01c796 to 0x00ee13 (0xd982 bytes) 000005 1782218773.711715 write 0x02033f thru 0x022180 (0x1e42 bytes) 000006 1782218773.713325 write 0x035fd5 thru 0x03fcaf (0x9cdb bytes) fallocating to largest ever: 0x3caa8 000007 1782218773.720161 fallocate 0x02dfcb thru 0x03caa7 (0xeadd bytes) 000008 1782218773.734778 fallocate 0x024277 thru 0x02851c (0x42a6 bytes) 000009 1782218773.762707 write 0x03f953 thru 0x03ffff (0x6ad bytes) 000010 1782218773.765142 trunc from 0x018700 to 0x0252dd (0xcbde bytes) 000011 1782218773.803920 mapwrite 0x024103 thru 0x0243a8 (0x2a6 bytes) 000012 1782218773.874436 fallocate 0x030291 thru 0x032961 (0x26d1 bytes) 000013 1782218773.893277 mapwrite 0x002b34 thru 0x0059af (0x2e7c bytes) 000014 1782218773.949433 mapread 0x011dda thru 0x015f7e (0x41a5 bytes) 000016 1782218773.964220 trunc from 0x01b1bd to 0x0245a2 (0x93e6 bytes) 000017 1782218773.972763 mapread 0x02694f thru 0x027b3f (0x11f1 bytes) 000018 1782218773.976023 fallocate 0x01aee9 thru 0x01c164 (0x127c bytes) 000020 1782218773.986999 write 0x021648 thru 0x02dc77 (0xc630 bytes) 000021 1782218774.004348 trunc from 0x01cf0a to 0x02583a (0x8931 bytes) 000022 1782218774.026999 read 0x032b64 thru 0x03f5d6 (0xca73 bytes) 000023 1782218774.037888 trunc from 0x008bd0 to 0x00fb2a (0x6f5b bytes) 000024 1782218774.051342 mapwrite 0x0317fc thru 0x036a9e (0x52a3 bytes) 000025 1782218774.119009 trunc from 0x001daa to 0x00b5a8 (0x97ff bytes) 000026 1782218774.133467 trunc from 0x03797c to 0x03ffff (0x8684 bytes) 000027 1782218774.156208 fallocate 0x011532 thru 0x01f3db (0xdeaa bytes) 000028 1782218774.171148 trunc from 0x040000 to 0x0115d2 (0x2ea2d bytes) 000029 1782218774.220817 punch from 0x005f19 to 0x009b7c (0x3c64 bytes) 000030 1782218774.233434 trunc from 0x001ce0 to 0x00d027 (0xb348 bytes) 000031 1782218774.243138 mapread 0x00ff62 thru 0x01111b (0x11ba bytes) 000032 1782218774.267752 trunc from 0x0115d3 to 0x0153e9 (0x3e17 bytes) fallocating to largest ever: 0x3f907 000034 1782218774.295930 fallocate 0x03acf6 thru 0x03f906 (0x4c11 bytes) 000035 1782218774.302449 mapwrite 0x0134b7 thru 0x01e264 (0xadae bytes) 000036 1782218774.392133 fallocate 0x014f65 thru 0x01a4da (0x5576 bytes) 000037 1782218774.400896 mapread 0x0112a0 thru 0x01e264 (0xcfc5 bytes) 000038 1782218774.405481 write 0x02814e thru 0x0376ea (0xf59d bytes) 000040 1782218774.412998 write 0x01158d thru 0x01dcfa (0xc76e bytes) 000041 1782218774.420054 mapwrite 0x024c16 thru 0x02c81e (0x7c09 bytes) 000042 1782218774.506294 trunc from 0x0376eb to 0x00988c (0x2de5e bytes) fallocating to largest ever: 0x40000 000043 1782218774.570704 fallocate 0x03b3ad thru 0x03ffff (0x4c53 bytes) 000044 1782218774.583470 read 0x039748 thru 0x03ffff (0x68b8 bytes) 000045 1782218774.630756 write 0x03e77b thru 0x03ffff (0x1885 bytes) 000046 1782218774.644410 fallocate 0x003ef5 thru 0x0056a3 (0x17af bytes) 000048 1782218774.678080 mapread 0x018872 thru 0x01e112 (0x58a1 bytes) 000049 1782218774.685769 mapread 0x03231a thru 0x0335ba (0x12a1 bytes) 000050 1782218774.701480 mapwrite 0x012bb7 thru 0x017105 (0x454f bytes) 000051 1782218774.773545 trunc from 0x040000 to 0x016a59 (0x295a6 bytes) 000053 1782218774.827252 punch from 0x005f49 to 0x00a82c (0x48e4 bytes) 000055 1782218774.846651 mapwrite 0x00dc1f thru 0x00ea16 (0xdf8 bytes) 000056 1782218774.895871 read 0x014d31 thru 0x016a59 (0x1d29 bytes) 000057 1782218774.902059 trunc from 0x00d632 to 0x00e59b (0xf6a bytes) 000058 1782218774.917139 mapwrite 0x038676 thru 0x03be42 (0x37cd bytes) 000059 1782218775.088594 punch from 0x031d94 to 0x032e4b (0x10b8 bytes) 000060 1782218775.100409 read 0x0238e0 thru 0x025bf2 (0x2313 bytes) 000061 1782218775.103859 trunc from 0x01e23b to 0x020248 (0x200e bytes) 000062 1782218775.110824 punch from 0x03a449 to 0x03b6db (0x1293 bytes) 000063 1782218775.118360 write 0x01a783 thru 0x028a71 (0xe2ef bytes) 000064 1782218775.121817 read 0x03b80e thru 0x03be42 (0x635 bytes) 000066 1782218775.140297 trunc from 0x0360d5 to 0x03b708 (0x5634 bytes) 000068 1782218775.154711 mapread 0x00617a thru 0x007f40 (0x1dc7 bytes) 000069 1782218775.158319 mapwrite 0x0269ee thru 0x028e87 (0x249a bytes) 000070 1782218775.195914 fallocate 0x0009af thru 0x00a55f (0x9bb1 bytes) 000071 1782218775.205549 punch from 0x0342c9 to 0x03600c (0x1d44 bytes) 000072 1782218775.214770 trunc from 0x03be43 to 0x02d0d2 (0xed70 bytes) 000073 1782218775.235808 punch from 0x01509e to 0x0174ce (0x2431 bytes) 000076 1782218775.247291 trunc from 0x02d0d3 to 0x014dbb (0x18317 bytes) 000078 1782218775.294298 mapread 0x00a18b thru 0x011698 (0x750e bytes) 000079 1782218775.296211 write 0x03f501 thru 0x03ffff (0xaff bytes) 000080 1782218775.297595 write 0x01a0fe thru 0x0299a4 (0xf8a7 bytes) 000081 1782218775.359756 fallocate 0x020b62 thru 0x02a9a7 (0x9e46 bytes) 000082 1782218775.381973 write 0x03cb17 thru 0x03dbe3 (0x10cd bytes) 000083 1782218775.392787 trunc from 0x040000 to 0x01eabb (0x21544 bytes) 000084 1782218775.438332 read 0x014c07 thru 0x01eabb (0x9eb5 bytes) 000085 1782218775.469606 read 0x018d35 thru 0x01eabb (0x5d87 bytes) 000086 1782218775.471114 punch from 0x01dba7 to 0x01eabb (0xf15 bytes) 000087 1782218775.493008 mapwrite 0x014756 thru 0x01b4fd (0x6da8 bytes) 000088 1782218775.528002 mapwrite 0x03e7b3 thru 0x03ff20 (0x176e bytes) 000089 1782218775.593215 mapread 0x037b6c thru 0x03fec5 (0x835a bytes) 000090 1782218775.597992 mapwrite 0x02177f thru 0x02ae0b (0x968d bytes) 000091 1782218775.627126 mapread 0x027d3f thru 0x0346f7 (0xc9b9 bytes) 000092 1782218775.629556 trunc from 0x03ff21 to 0x02fe08 (0x10118 bytes) 000093 1782218775.658382 mapread 0x00b612 thru 0x00c057 (0xa46 bytes) 000094 1782218775.660197 trunc from 0x02fe09 to 0x030b3e (0xd36 bytes) 000095 1782218775.693472 fallocate 0x03856a thru 0x03d4e9 (0x4f80 bytes) 000096 1782218775.702405 fallocate 0x02948d thru 0x02aacd (0x1641 bytes) 000097 1782218775.716447 fallocate 0x00e7ec thru 0x015a70 (0x7285 bytes) 000098 1782218775.722533 mapread 0x01e08c thru 0x01fe06 (0x1d7b bytes) 000099 1782218775.729387 trunc from 0x039ae9 to 0x03a170 (0x688 bytes) 000100 1782218775.736244 mapread 0x032520 thru 0x03d4e9 (0xafca bytes) 000104 1782218775.860739 read 0x037800 thru 0x03c08f (0x4890 bytes) 000105 1782218775.866417 punch from 0x03ac0f to 0x03d4e9 (0x28db bytes) 000107 1782218775.876037 mapwrite 0x015fc0 thru 0x0183af (0x23f0 bytes) 000110 1782218775.917271 trunc from 0x020785 to 0x0230a0 (0x291c bytes) 000111 1782218775.951003 fallocate 0x01f0dd thru 0x020629 (0x154d bytes) 000112 1782218775.981893 MIRROR_EXTEND to 2 mirrors 000113 1782218778.159397 mapwrite 0x027ef5 thru 0x02b8b1 (0x39bd bytes) 000114 1782218778.399060 read 0x002403 thru 0x00b172 (0x8d70 bytes) 000115 1782218778.427212 mapwrite 0x039fe0 thru 0x03a7c8 (0x7e9 bytes) 000116 1782218778.494570 mapread 0x01b885 thru 0x01fa0b (0x4187 bytes) 000118 1782218778.546294 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) 000119 1782218778.997159 mapread 0x032b89 thru 0x039553 (0x69cb bytes) 000121 1782218779.097479 trunc from 0x00182a to 0x00f1ee (0xd9c5 bytes) 000123 1782218779.116317 trunc from 0x03d4ea to 0x00b7b6 (0x31d33 bytes) 000125 1782218779.207736 fallocate 0x02163c thru 0x02685c (0x5221 bytes) 000126 1782218779.214386 mapread 0x00a789 thru 0x01610f (0xb987 bytes) 000127 1782218779.330877 mapread 0x017212 thru 0x02333f (0xc12e bytes) 000128 1782218779.364414 mapread 0x015e16 thru 0x020348 (0xa533 bytes) 000129 1782218779.374835 fallocate 0x02fd3a thru 0x03f6bd (0xf984 bytes) 000130 1782218779.408746 read 0x004e6c thru 0x010076 (0xb20b bytes) 000131 1782218779.415551 write 0x01c94f thru 0x01f8c7 (0x2f79 bytes) 000132 1782218779.417387 punch from 0x00c005 to 0x014ee8 (0x8ee4 bytes) 000133 1782218779.434878 mapread 0x029d84 thru 0x02a7c6 (0xa43 bytes) 000134 1782218779.495463 punch from 0x0068cd to 0x00f6b8 (0x8dec bytes) 000136 1782218779.550827 mapread 0x02e056 thru 0x02e3c2 (0x36d bytes) 000137 1782218779.559551 write 0x01fb4d thru 0x02ac07 (0xb0bb bytes) 000138 1782218779.566720 trunc from 0x0315d8 to 0x03bf09 (0xa932 bytes) 000139 1782218779.584384 mapwrite 0x03db73 thru 0x03ffff (0x248d bytes) 000140 1782218779.721602 fallocate 0x013b59 thru 0x01dfbf (0xa467 bytes) 000141 1782218779.729651 trunc from 0x016711 to 0x0180fb (0x19eb bytes) 000142 1782218779.762256 mapread 0x001b6d thru 0x003578 (0x1a0c bytes) 000143 1782218779.807262 mapread 0x0389e8 thru 0x03edde (0x63f7 bytes) 000145 1782218779.815425 mapread 0x0202a0 thru 0x02744c (0x71ad bytes) 000146 1782218779.822065 trunc from 0x007748 to 0x012c13 (0xb4cc bytes) 000147 1782218779.841093 read 0x012a20 thru 0x01560d (0x2bee bytes) 000148 1782218779.872046 mapread 0x021d17 thru 0x02c587 (0xa871 bytes) 000151 1782218779.889064 trunc from 0x040000 to 0x024585 (0x1ba7a bytes) 000152 1782218779.937836 fallocate 0x03d5fb thru 0x03ffff (0x2a05 bytes) 000153 1782218779.978515 punch from 0x00a1f6 to 0x0184e4 (0xe2ef bytes) 000154 1782218780.000473 read 0x00dde4 thru 0x013efd (0x611a bytes) 000155 1782218780.037371 trunc from 0x012eef to 0x018907 (0x5a19 bytes) 000156 1782218780.055038 trunc from 0x020b42 to 0x024585 (0x3a44 bytes) 000157 1782218780.094447 punch from 0x00317d to 0x00328f (0x113 bytes) 000159 1782218780.128251 punch from 0x0006bb to 0x00925f (0x8ba5 bytes) 000160 1782218780.170158 MIRROR_EXTEND to 2 mirrors 000162 1782218781.638002 mapread 0x0043d0 thru 0x00ee87 (0xaab8 bytes) 000164 1782218781.710970 read 0x0239a5 thru 0x024585 (0xbe1 bytes) 000165 1782218781.732437 MIRROR_RESYNC 2 mirrors 000166 1782218782.455000 trunc from 0x024586 to 0x026c14 (0x268f bytes) 000167 1782218782.761863 trunc from 0x026c15 to 0x0376cc (0x10ab8 bytes) 000168 1782218782.890585 fallocate 0x020b90 thru 0x02f3f7 (0xe868 bytes) 000169 1782218782.919595 fallocate 0x00076b thru 0x0037ba (0x3050 bytes) 000170 1782218782.950518 mapread 0x025544 thru 0x029c24 (0x46e1 bytes) 000171 1782218782.990217 mapwrite 0x03a061 thru 0x03f171 (0x5111 bytes) 000173 1782218783.154529 write 0x031aa1 thru 0x0389f3 (0x6f53 bytes) 000174 1782218783.181573 trunc from 0x015e5e to 0x020a87 (0xac2a bytes) 000175 1782218783.206788 trunc from 0x03f172 to 0x0119da (0x2d797 bytes) 000176 1782218783.270280 punch from 0x00b7d9 to 0x0119da (0x6202 bytes) 000177 1782218783.316909 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x119db): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x119db): Mirror 2: 0x9e4120d4 (stale) CRC-32 checksum value for chunk [0, 0x119db): Mirror 1: 0x9e4120d4 Mirror 3: 0x9e4120d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x119db: skipped 000178 1782218785.157865 write 0x017dad thru 0x01ae6b (0x30bf bytes) 000179 1782218785.218503 mapwrite 0x033ca0 thru 0x038895 (0x4bf6 bytes) 000180 1782218785.395167 fallocate 0x0342a3 thru 0x03ffff (0xbd5d bytes) 000181 1782218785.412937 mapread 0x01a06f thru 0x01cf1c (0x2eae bytes) 000182 1782218785.434468 read 0x01e099 thru 0x024225 (0x618d bytes) 000183 1782218785.455365 MIRROR_RESYNC 3 mirrors 000184 1782218786.867794 mapwrite 0x018b57 thru 0x026003 (0xd4ad bytes) 000185 1782218787.151840 trunc from 0x02d13d to 0x02d697 (0x55b bytes) 000186 1782218787.163389 fallocate 0x02c410 thru 0x03520b (0x8dfc bytes) 000187 1782218787.175942 punch from 0x01f7a8 to 0x028aa1 (0x92fa bytes) 000188 1782218787.188320 trunc from 0x038b78 to 0x03f833 (0x6cbc bytes) 000189 1782218787.201259 punch from 0x03edb6 to 0x03ffff (0x124a bytes) 000190 1782218787.223940 mapwrite 0x0363d6 thru 0x03a71b (0x4346 bytes) 000191 1782218787.310295 mapwrite 0x002ab2 thru 0x00f8b2 (0xce01 bytes) 000192 1782218787.357238 write 0x01b414 thru 0x02443b (0x9028 bytes) 000193 1782218787.368128 trunc from 0x02a93f to 0x036d03 (0xc3c5 bytes) 000194 1782218787.376914 fallocate 0x01c9ce thru 0x02814b (0xb77e bytes) 000195 1782218787.385464 mapwrite 0x01fe18 thru 0x02278c (0x2975 bytes) 000196 1782218787.438498 mapwrite 0x00424f thru 0x013839 (0xf5eb bytes) 000197 1782218787.505618 punch from 0x006d54 to 0x00957d (0x282a bytes) 000198 1782218787.514411 trunc from 0x040000 to 0x02defd (0x12102 bytes) 000199 1782218787.566198 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2defe): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2defe): Mirror 2: 0xa09a8350 (stale) Mirror 3: 0xa09a8350 (stale) CRC-32 checksum value for chunk [0, 0x2defe): Mirror 1: 0x4499eb9e Mirror 4: 0x4499eb9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2defe: skipped 000200 1782218789.616653 trunc from 0x02ca47 to 0x02defd (0x14b7 bytes) 000201 1782218789.704193 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2defe): Mirror 3: 0xa09a8350 (stale) Mirror 4: 0x4499eb9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2defe): Mirror 3: 0xa09a8350 (stale) Mirror 4: 0x4499eb9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2defe: skipped 000202 1782218790.314611 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2defe): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2defe): Mirror 3: 0xa09a8350 (stale) Mirror 4: 0x4499eb9e (stale) CRC-32 checksum value for chunk [0, 0x2defe): Mirror 1: 0x4499eb9e Mirror 5: 0x4499eb9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2defe: skipped 000203 1782218791.847154 write 0x024acd thru 0x024bb9 (0xed bytes) 000204 1782218791.969063 punch from 0x009078 to 0x010ba0 (0x7b29 bytes) 000205 1782218791.989652 trunc from 0x02defe to 0x0318b6 (0x39b9 bytes) 000207 1782218792.074216 mapread 0x0254e2 thru 0x0267e3 (0x1302 bytes) 000208 1782218792.102157 mapwrite 0x01c52b thru 0x02a8aa (0xe380 bytes) 000209 1782218792.229224 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x318b7): Mirror 4: 0xc671e923 (stale) Mirror 5: 0xc671e923 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x318b7): Mirror 4: 0xc671e923 (stale) Mirror 5: 0xc671e923 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x318b7: skipped 000210 1782218793.039445 mapread 0x008b35 thru 0x008ddd (0x2a9 bytes) 000211 1782218793.059372 mapread 0x003cdf thru 0x00e99a (0xacbc bytes) 000212 1782218793.131320 trunc from 0x0318b7 to 0x027b57 (0x9d5f bytes) 000213 1782218793.218378 mapread 0x010992 thru 0x01d5a7 (0xcc16 bytes) 000214 1782218793.264592 fallocate 0x03df6a thru 0x03ffff (0x2096 bytes) 000215 1782218793.273123 write 0x00d451 thru 0x016c95 (0x9845 bytes) 000216 1782218793.279183 write 0x016441 thru 0x01bb51 (0x5711 bytes) 000218 1782218793.287606 mapread 0x00a44b thru 0x0141c8 (0x9d7e bytes) 000219 1782218793.290630 write 0x00b30b thru 0x01378c (0x8482 bytes) 000220 1782218793.293369 mapread 0x01bc11 thru 0x024107 (0x84f7 bytes) 000221 1782218793.339177 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27b58): Mirror 4: 0x2f7165 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b58): Mirror 4: 0x2f7165 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b58: skipped 000222 1782218793.864536 write 0x0104c5 thru 0x01e4d3 (0xe00f bytes) 000223 1782218793.895076 fallocate 0x017e19 thru 0x01b7c8 (0x39b0 bytes) 000224 1782218793.908827 read 0x002720 thru 0x00c2ce (0x9baf bytes) 000226 1782218793.995032 read 0x00577d thru 0x00eaf6 (0x937a bytes) 000227 1782218794.026391 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27b58): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b58): Mirror 4: 0x2f7165 (stale) CRC-32 checksum value for chunk [0, 0x27b58): Mirror 1: 0x86565891 Mirror 5: 0x86565891 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b58: skipped 000228 1782218795.982500 trunc from 0x01da7f to 0x025314 (0x7896 bytes) 000229 1782218796.072127 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27b58): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b58): Mirror 4: 0x2f7165 (stale) Mirror 5: 0x86565891 (stale) CRC-32 checksum value for chunk [0, 0x27b58): Mirror 1: 0x9c51e155 Mirror 6: 0x9c51e155 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b58: skipped 000230 1782218798.098167 mapread 0x008a0a thru 0x00b9a8 (0x2f9f bytes) 000231 1782218798.122196 trunc from 0x0208ef to 0x027b57 (0x7269 bytes) 000232 1782218798.190304 mapwrite 0x00ba1c thru 0x00d9c1 (0x1fa6 bytes) 000233 1782218798.238341 trunc from 0x027b58 to 0x02a1fc (0x26a5 bytes) 000234 1782218798.331519 write 0x001028 thru 0x0046f7 (0x36d0 bytes) 000236 1782218798.372196 write 0x02df35 thru 0x03a1b6 (0xc282 bytes) 000238 1782218798.387686 write 0x0097a6 thru 0x00ff6e (0x67c9 bytes) 000239 1782218798.394297 write 0x02e3a2 thru 0x034f16 (0x6b75 bytes) 000240 1782218798.395735 trunc from 0x03a1b7 to 0x02c98b (0xd82b bytes) 000241 1782218798.449621 write 0x02d252 thru 0x035c10 (0x89bf bytes) 000242 1782218798.458397 trunc from 0x019aae to 0x01f281 (0x57d4 bytes) 000243 1782218798.487537 read 0x00f9b1 thru 0x01d368 (0xd9b8 bytes) 000244 1782218798.510114 mapread 0x029512 thru 0x02d251 (0x3d40 bytes) 000246 1782218798.540521 write 0x01ae48 thru 0x01f872 (0x4a2b bytes) 000247 1782218798.553939 trunc from 0x034716 to 0x035c10 (0x14fb bytes) 000248 1782218798.591791 punch from 0x022914 to 0x02a32b (0x7a18 bytes) 000249 1782218798.619953 MIRROR_RESYNC 4 mirrors 000250 1782218800.792697 mapwrite 0x0321ee thru 0x03f92d (0xd740 bytes) 000251 1782218801.017711 mapwrite 0x0249ce thru 0x02ed8d (0xa3c0 bytes) 000252 1782218801.061441 read 0x02bd36 thru 0x02bd3c (0x7 bytes) 000253 1782218801.063163 read 0x02d28c thru 0x0302af (0x3024 bytes) 000254 1782218801.067275 mapwrite 0x02dbb6 thru 0x038256 (0xa6a1 bytes) 000256 1782218801.100388 mapread 0x039f2d thru 0x03f92d (0x5a01 bytes) 000257 1782218801.106610 trunc from 0x025de0 to 0x03412a (0xe34b bytes) 000258 1782218801.126985 trunc from 0x01b0a0 to 0x01ecb2 (0x3c13 bytes) 000259 1782218801.136378 read 0x0100c7 thru 0x016854 (0x678e bytes) 000260 1782218801.161257 mapread 0x039a55 thru 0x03f92d (0x5ed9 bytes) 000261 1782218801.164804 write 0x00a7ce thru 0x016ba2 (0xc3d5 bytes) 000262 1782218801.171049 mapread 0x03610c thru 0x03c3ad (0x62a2 bytes) 000263 1782218801.172816 trunc from 0x03f92e to 0x02d1cf (0x1275e bytes) 000264 1782218801.220276 punch from 0x024d62 to 0x02b4ce (0x676d bytes) 000265 1782218801.228319 fallocate 0x00628c thru 0x014efa (0xec6f bytes) 000266 1782218801.247372 fallocate 0x005985 thru 0x00ada1 (0x541d bytes) 000267 1782218801.265343 mapwrite 0x008c09 thru 0x00dbe1 (0x4fd9 bytes) 000268 1782218801.294850 trunc from 0x02d1d0 to 0x026e0c (0x63c3 bytes) 000269 1782218801.322271 read 0x0124b7 thru 0x01f5ff (0xd149 bytes) 000270 1782218801.323916 mapwrite 0x00ae5d thru 0x00f160 (0x4304 bytes) 000271 1782218801.353996 MIRROR_RESYNC 4 mirrors 000273 1782218803.991379 read 0x00d594 thru 0x011821 (0x428e bytes) 000274 1782218804.017433 fallocate 0x026f91 thru 0x02f6ab (0x871b bytes) 000275 1782218804.132076 read 0x012e61 thru 0x01dbb4 (0xad54 bytes) 000276 1782218804.163705 read 0x025fe2 thru 0x02f6ab (0x96ca bytes) 000277 1782218804.170809 punch from 0x01ffca to 0x02f1b5 (0xf1ec bytes) 000278 1782218804.190183 fallocate 0x037d6d thru 0x03ffff (0x8293 bytes) 000279 1782218804.199575 trunc from 0x001d9f to 0x00bd04 (0x9f66 bytes) 000280 1782218804.221818 MIRROR_RESYNC 4 mirrors 000281 1782218805.987377 mapwrite 0x0168d8 thru 0x0222d4 (0xb9fd bytes) 000282 1782218806.192830 MIRROR_RESYNC 4 mirrors 000283 1782218808.120521 read 0x02a029 thru 0x0339e1 (0x99b9 bytes) 000284 1782218808.165315 mapread 0x0127c5 thru 0x015d5e (0x359a bytes) 000285 1782218808.203110 write 0x01da3e thru 0x0231ce (0x5791 bytes) 000286 1782218808.332783 write 0x033bae thru 0x03a018 (0x646b bytes) 000287 1782218808.353962 read 0x006bb6 thru 0x012172 (0xb5bd bytes) 000288 1782218808.392826 read 0x03fe5a thru 0x03ffff (0x1a6 bytes) 000289 1782218808.393516 fallocate 0x035e8e thru 0x03ffff (0xa172 bytes) 000290 1782218808.408755 mapread 0x023455 thru 0x02ad21 (0x78cd bytes) 000291 1782218808.410179 fallocate 0x016b50 thru 0x01f4db (0x898c bytes) 000292 1782218808.419298 mapread 0x014c18 thru 0x02308d (0xe476 bytes) 000293 1782218808.421060 read 0x037a02 thru 0x03ffff (0x85fe bytes) 000294 1782218808.431369 read 0x030be0 thru 0x03278f (0x1bb0 bytes) 000295 1782218808.441389 read 0x00b06a thru 0x01a71c (0xf6b3 bytes) 000296 1782218808.446895 fallocate 0x03bc46 thru 0x03ffff (0x43ba bytes) 000297 1782218808.468106 mapwrite 0x0148ad thru 0x014c2b (0x37f bytes) 000298 1782218808.509436 read 0x02d5db thru 0x031d3c (0x4762 bytes) 000299 1782218808.534126 MIRROR_RESYNC 4 mirrors 000300 1782218809.909574 mapread 0x0112f7 thru 0x011447 (0x151 bytes) 000301 1782218809.921712 MIRROR_RESYNC 4 mirrors 000303 1782218810.793457 mapwrite 0x02fa95 thru 0x03d054 (0xd5c0 bytes) 000304 1782218811.117170 read 0x0091fb thru 0x00ed2e (0x5b34 bytes) 000305 1782218811.159752 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5094b02a (stale) Mirror 5: 0x5094b02a (stale) Mirror 6: 0x5094b02a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x191683c Mirror 7: 0x191683c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000306 1782218812.662250 write 0x013e99 thru 0x021b7a (0xdce2 bytes) 000307 1782218812.745438 trunc from 0x01f1cf to 0x0218be (0x26f0 bytes) 000308 1782218812.778340 trunc from 0x040000 to 0x0059b9 (0x3a646 bytes) 000309 1782218812.870245 fallocate 0x0184d7 thru 0x02715e (0xec88 bytes) 000310 1782218812.878219 fallocate 0x020f64 thru 0x02eb7b (0xdc18 bytes) 000311 1782218812.888500 mapread 0x002096 thru 0x0059b9 (0x3924 bytes) 000312 1782218812.926054 trunc from 0x00494f to 0x0059b9 (0x106b bytes) 000313 1782218812.968521 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x59ba): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x59ba): Mirror 4: 0xcc4551fd (stale) Mirror 5: 0xcc4551fd (stale) Mirror 6: 0xcc4551fd (stale) Mirror 7: 0xcc4551fd (stale) CRC-32 checksum value for chunk [0, 0x59ba): Mirror 1: 0xcc4551fd Mirror 8: 0xcc4551fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x59ba: skipped 000314 1782218814.787332 mapread 0x001953 thru 0x0059b9 (0x4067 bytes) 000315 1782218814.946076 trunc from 0x003bf6 to 0x0059b9 (0x1dc4 bytes) 000316 1782218815.103586 trunc from 0x0052ed to 0x0059b9 (0x6cd bytes) 000317 1782218815.114438 fallocate 0x03de61 thru 0x03ffff (0x219f bytes) 000318 1782218815.121926 trunc from 0x0059ba to 0x029c65 (0x242ac bytes) 000319 1782218815.201483 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x29c66): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c66): Mirror 4: 0x47d067b3 (stale) Mirror 5: 0x47d067b3 (stale) Mirror 6: 0x47d067b3 (stale) Mirror 7: 0x47d067b3 (stale) Mirror 8: 0x8c276714 (stale) CRC-32 checksum value for chunk [0, 0x29c66): Mirror 1: 0x8c276714 Mirror 9: 0x8c276714 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c66: skipped 000320 1782218817.552590 read 0x020b96 thru 0x028ae9 (0x7f54 bytes) 000321 1782218817.587351 trunc from 0x00e9c0 to 0x01e4ae (0xfaef bytes) 000322 1782218817.658391 punch from 0x01f135 to 0x029c65 (0xab31 bytes) 000323 1782218817.679448 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29c66): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c66): Mirror 4: 0x47d067b3 (stale) Mirror 5: 0x47d067b3 (stale) Mirror 6: 0x47d067b3 (stale) Mirror 7: 0x47d067b3 (stale) Mirror 8: 0x8c276714 (stale) Mirror 9: 0x8c276714 (stale) CRC-32 checksum value for chunk [0, 0x29c66): Mirror 1: 0x8c276714 Mirror 10: 0x8c276714 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c66: skipped 000325 1782218819.531848 punch from 0x00d531 to 0x012717 (0x51e7 bytes) 000326 1782218819.608617 fallocate 0x01aedf thru 0x01bd4d (0xe6f bytes) 000327 1782218819.618364 mapwrite 0x01e1bd thru 0x029bce (0xba12 bytes) 000328 1782218819.704056 read 0x012f25 thru 0x015b65 (0x2c41 bytes) 000329 1782218819.715587 mapread 0x0179a8 thru 0x023fc6 (0xc61f bytes) 000330 1782218819.741017 trunc from 0x00bb86 to 0x017a81 (0xbefc bytes) 000331 1782218819.762991 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29c66): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c66): Mirror 4: 0x47d067b3 (stale) Mirror 5: 0x47d067b3 (stale) Mirror 6: 0x47d067b3 (stale) Mirror 7: 0x47d067b3 (stale) Mirror 8: 0x8c276714 (stale) Mirror 9: 0x8c276714 (stale) Mirror 10: 0x8c276714 (stale) CRC-32 checksum value for chunk [0, 0x29c66): Mirror 1: 0x2f3b9b6e Mirror 11: 0x2f3b9b6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c66: skipped 000333 1782218821.558500 mapread 0x0049ca thru 0x0103b9 (0xb9f0 bytes) 000334 1782218821.644012 trunc from 0x01bcb8 to 0x021c8e (0x5fd7 bytes) 000335 1782218821.778833 fallocate 0x025ce2 thru 0x026a72 (0xd91 bytes) 000336 1782218821.807479 fallocate 0x00f0ca thru 0x017591 (0x84c8 bytes) 000337 1782218821.820347 punch from 0x022f5c to 0x029c65 (0x6d0a bytes) 000338 1782218821.829291 mapread 0x00bb71 thru 0x01a2c9 (0xe759 bytes) 000339 1782218821.902273 mapread 0x010c85 thru 0x019d96 (0x9112 bytes) 000341 1782218821.922096 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29c66): Mirror 5: 0x47d067b3 (stale) Mirror 6: 0x47d067b3 (stale) Mirror 7: 0x47d067b3 (stale) Mirror 8: 0x8c276714 (stale) Mirror 9: 0x8c276714 (stale) Mirror 10: 0x8c276714 (stale) Mirror 11: 0x2f3b9b6e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c66): Mirror 5: 0x47d067b3 (stale) Mirror 6: 0x47d067b3 (stale) Mirror 7: 0x47d067b3 (stale) Mirror 8: 0x8c276714 (stale) Mirror 9: 0x8c276714 (stale) Mirror 10: 0x8c276714 (stale) Mirror 11: 0x2f3b9b6e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c66: skipped 000342 1782218822.981880 trunc from 0x00787f to 0x012a6a (0xb1ec bytes) 000343 1782218823.008608 punch from 0x014175 to 0x019806 (0x5692 bytes) 000344 1782218823.030686 punch from 0x007a2f to 0x00e830 (0x6e02 bytes) 000345 1782218823.058170 trunc from 0x029c66 to 0x02835f (0x1906 bytes) 000346 1782218823.151525 mapread 0x00210a thru 0x00d267 (0xb15e bytes) 000347 1782218823.237634 write 0x0320fc thru 0x03a927 (0x882c bytes) 000348 1782218823.268204 read 0x0160d1 thru 0x01a304 (0x4234 bytes) 000349 1782218823.307916 mapread 0x013999 thru 0x016e8e (0x34f6 bytes) 000350 1782218823.341602 punch from 0x02711d to 0x02eed9 (0x7dbd bytes) 000351 1782218823.358350 mapread 0x00538a thru 0x010cf6 (0xb96d bytes) 000353 1782218823.373359 punch from 0x0271e7 to 0x032b73 (0xb98d bytes) 000354 1782218823.417966 mapread 0x02ed62 thru 0x03a927 (0xbbc6 bytes) 000358 1782218823.471280 mapwrite 0x00375a thru 0x00e0db (0xa982 bytes) 000359 1782218823.579559 mapwrite 0x0110cf thru 0x0148b8 (0x37ea bytes) 000360 1782218823.630936 mapwrite 0x019621 thru 0x0201c0 (0x6ba0 bytes) 000361 1782218823.693209 trunc from 0x03a928 to 0x0355fe (0x5329 bytes) 000362 1782218823.735653 trunc from 0x0355ff to 0x011423 (0x241db bytes) 000363 1782218823.775985 mapread 0x00f4c9 thru 0x011423 (0x1f5b bytes) 000364 1782218823.778734 fallocate 0x031620 thru 0x037b37 (0x6518 bytes) 000365 1782218823.792160 trunc from 0x015726 to 0x0198d7 (0x41b2 bytes) 000366 1782218823.815240 trunc from 0x037b38 to 0x025ed4 (0x11c63 bytes) 000367 1782218823.862210 write 0x0084c7 thru 0x01682a (0xe364 bytes) 000368 1782218823.890309 MIRROR_RESYNC 8 mirrors 000369 1782218826.216185 MIRROR_EXTEND to 9 mirrors 000370 1782218827.838347 trunc from 0x0247ed to 0x025ed4 (0x16e8 bytes) 000371 1782218827.931013 write 0x0052d7 thru 0x00f879 (0xa5a3 bytes) 000372 1782218827.954280 mapread 0x016121 thru 0x019164 (0x3044 bytes) 000373 1782218828.043622 mapwrite 0x009042 thru 0x00e2a3 (0x5262 bytes) 000374 1782218828.109263 trunc from 0x001a5b to 0x010f35 (0xf4db bytes) 000375 1782218828.131065 punch from 0x021a44 to 0x023851 (0x1e0e bytes) 000376 1782218828.139894 punch from 0x002905 to 0x008ee9 (0x65e5 bytes) 000377 1782218828.151960 punch from 0x01b65a to 0x025ed4 (0xa87b bytes) 000378 1782218828.169214 fallocate 0x02d0bb thru 0x034b9e (0x7ae4 bytes) 000380 1782218828.226946 MIRROR_RESYNC 9 mirrors 000381 1782218832.076187 MIRROR_RESYNC 9 mirrors 000382 1782218833.137778 write 0x020665 thru 0x029c6f (0x960b bytes) 000383 1782218833.250928 read 0x02c407 thru 0x034b9e (0x8798 bytes) 000384 1782218833.281705 trunc from 0x01c477 to 0x029ccb (0xd855 bytes) 000386 1782218833.339341 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34b9f): Mirror 5: 0x61ed46c2 (stale) Mirror 6: 0x61ed46c2 (stale) Mirror 7: 0x61ed46c2 (stale) Mirror 9: 0x61ed46c2 (stale) Mirror 10: 0x61ed46c2 (stale) Mirror 11: 0x61ed46c2 (stale) Mirror 12: 0x61ed46c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b9f): Mirror 5: 0x61ed46c2 (stale) Mirror 6: 0x61ed46c2 (stale) Mirror 7: 0x61ed46c2 (stale) Mirror 9: 0x61ed46c2 (stale) Mirror 10: 0x61ed46c2 (stale) Mirror 11: 0x61ed46c2 (stale) Mirror 12: 0x61ed46c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b9f: skipped 000387 1782218834.069450 punch from 0x021d6c to 0x029cb9 (0x7f4e bytes) 000388 1782218834.082787 write 0x03f6c3 thru 0x03ffff (0x93d bytes) 000389 1782218834.103742 MIRROR_RESYNC 8 mirrors 000390 1782218837.353331 trunc from 0x040000 to 0x010fb2 (0x2f04d bytes) 000391 1782218837.423837 mapread 0x00d634 thru 0x010fb2 (0x397f bytes) 000392 1782218837.461329 write 0x035039 thru 0x037c95 (0x2c5d bytes) 000394 1782218837.472095 write 0x020dc7 thru 0x024f42 (0x417c bytes) 000396 1782218837.492645 mapwrite 0x01c41e thru 0x01e2e2 (0x1ec5 bytes) 000397 1782218837.571448 fallocate 0x00500e thru 0x013ad9 (0xeacc bytes) 000398 1782218837.592441 mapread 0x033c92 thru 0x037c95 (0x4004 bytes) 000399 1782218837.641963 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c96): Mirror 6: 0x72e407cf (stale) Mirror 7: 0x72e407cf (stale) Mirror 9: 0x72e407cf (stale) Mirror 10: 0x72e407cf (stale) Mirror 11: 0x72e407cf (stale) Mirror 12: 0x72e407cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c96): Mirror 6: 0x72e407cf (stale) Mirror 7: 0x72e407cf (stale) Mirror 9: 0x72e407cf (stale) Mirror 10: 0x72e407cf (stale) Mirror 11: 0x72e407cf (stale) Mirror 12: 0x72e407cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c96: skipped 000400 1782218838.454329 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37c96): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c96): Mirror 6: 0x72e407cf (stale) Mirror 7: 0x72e407cf (stale) Mirror 9: 0x72e407cf (stale) Mirror 10: 0x72e407cf (stale) Mirror 11: 0x72e407cf (stale) Mirror 12: 0x72e407cf (stale) CRC-32 checksum value for chunk [0, 0x37c96): Mirror 1: 0xc5bba791 Mirror 13: 0xc5bba791 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c96: skipped 000401 1782218840.610060 mapread 0x0259dd thru 0x032cbd (0xd2e1 bytes) 000402 1782218840.746723 mapread 0x0066ba thru 0x01490d (0xe254 bytes) 000403 1782218840.845918 mapread 0x00d2a1 thru 0x01b951 (0xe6b1 bytes) 000404 1782218840.939204 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37c96): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c96): Mirror 6: 0x72e407cf (stale) Mirror 7: 0x72e407cf (stale) Mirror 9: 0x72e407cf (stale) Mirror 10: 0x72e407cf (stale) Mirror 11: 0x72e407cf (stale) Mirror 12: 0x72e407cf (stale) CRC-32 checksum value for chunk [0, 0x37c96): Mirror 1: 0xc5bba791 Mirror 13: 0xc5bba791 Mirror 14: 0xc5bba791 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c96: skipped 000405 1782218843.432831 fallocate 0x02a995 thru 0x03a1e1 (0xf84d bytes) 000407 1782218843.501998 fallocate 0x026fc3 thru 0x030539 (0x9577 bytes) 000408 1782218843.520386 punch from 0x005b2c to 0x009d03 (0x41d8 bytes) 000409 1782218843.527818 mapread 0x0394cf thru 0x03a1e1 (0xd13 bytes) 000410 1782218843.546370 mapread 0x026a14 thru 0x02b3da (0x49c7 bytes) 000411 1782218843.571429 write 0x026e28 thru 0x02b1c4 (0x439d bytes) 000412 1782218843.592942 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1e2): Mirror 6: 0xf32e5667 (stale) Mirror 7: 0xf32e5667 (stale) Mirror 9: 0xf32e5667 (stale) Mirror 10: 0xf32e5667 (stale) Mirror 11: 0xf32e5667 (stale) Mirror 12: 0xf32e5667 (stale) Mirror 13: 0x9915aff0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1e2): Mirror 6: 0xf32e5667 (stale) Mirror 7: 0xf32e5667 (stale) Mirror 9: 0xf32e5667 (stale) Mirror 10: 0xf32e5667 (stale) Mirror 11: 0xf32e5667 (stale) Mirror 12: 0xf32e5667 (stale) Mirror 13: 0x9915aff0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1e2: skipped 000413 1782218844.553096 mapwrite 0x02d6c6 thru 0x02eb6d (0x14a8 bytes) 000414 1782218844.586114 trunc from 0x03a1e2 to 0x016ad6 (0x2370b bytes) 000415 1782218844.624917 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16ad7): Mirror 6: 0xb40164c2 (stale) Mirror 7: 0xb40164c2 (stale) Mirror 9: 0xb40164c2 (stale) Mirror 10: 0xb40164c2 (stale) Mirror 11: 0xb40164c2 (stale) Mirror 12: 0xb40164c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16ad7): Mirror 6: 0xb40164c2 (stale) Mirror 7: 0xb40164c2 (stale) Mirror 9: 0xb40164c2 (stale) Mirror 10: 0xb40164c2 (stale) Mirror 11: 0xb40164c2 (stale) Mirror 12: 0xb40164c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16ad7: skipped 000416 1782218845.393207 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16ad7): Mirror 7: 0xb40164c2 (stale) Mirror 9: 0xb40164c2 (stale) Mirror 10: 0xb40164c2 (stale) Mirror 11: 0xb40164c2 (stale) Mirror 12: 0xb40164c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16ad7): Mirror 7: 0xb40164c2 (stale) Mirror 9: 0xb40164c2 (stale) Mirror 10: 0xb40164c2 (stale) Mirror 11: 0xb40164c2 (stale) Mirror 12: 0xb40164c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16ad7: skipped 000417 1782218846.335507 read 0x011566 thru 0x016ad6 (0x5571 bytes) 000419 1782218846.367471 trunc from 0x008e6d to 0x015e10 (0xcfa4 bytes) 000420 1782218846.377970 mapwrite 0x00a22f thru 0x010bdd (0x69af bytes) 000421 1782218846.468331 mapwrite 0x0110d8 thru 0x01e3de (0xd307 bytes) 000422 1782218846.622964 fallocate 0x01e488 thru 0x023ea4 (0x5a1d bytes) 000423 1782218846.659755 read 0x01916e thru 0x01cd98 (0x3c2b bytes) 000425 1782218846.667917 write 0x002342 thru 0x010354 (0xe013 bytes) 000426 1782218846.741639 MIRROR_RESYNC 6 mirrors 000427 1782218848.077982 trunc from 0x003b1c to 0x00d4a7 (0x998c bytes) 000428 1782218848.146270 read 0x00c64e thru 0x00ea29 (0x23dc bytes) 000429 1782218848.166521 write 0x030c82 thru 0x034306 (0x3685 bytes) 000430 1782218848.168169 read 0x02a4aa thru 0x032450 (0x7fa7 bytes) 000431 1782218848.245235 mapread 0x00d535 thru 0x01412d (0x6bf9 bytes) 000432 1782218848.260175 read 0x034280 thru 0x034306 (0x87 bytes) 000434 1782218848.268087 read 0x002590 thru 0x00909e (0x6b0f bytes) 000435 1782218848.279758 mapread 0x02af34 thru 0x02e37f (0x344c bytes) 000436 1782218848.284122 write 0x02b052 thru 0x02f285 (0x4234 bytes) 000437 1782218848.290235 write 0x00dbf0 thru 0x015681 (0x7a92 bytes) 000438 1782218848.293049 trunc from 0x034307 to 0x01442a (0x1fedc bytes) 000439 1782218848.347295 trunc from 0x01074b to 0x01442a (0x3ce0 bytes) 000440 1782218848.379940 write 0x016982 thru 0x01749e (0xb1d bytes) 000442 1782218848.395116 trunc from 0x01749f to 0x009b73 (0xd92b bytes) 000443 1782218848.444258 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9b74): Mirror 7: 0xc0fb43f4 (stale) Mirror 9: 0xc0fb43f4 (stale) Mirror 10: 0xc0fb43f4 (stale) Mirror 12: 0xc0fb43f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9b74): Mirror 7: 0xc0fb43f4 (stale) Mirror 9: 0xc0fb43f4 (stale) Mirror 10: 0xc0fb43f4 (stale) Mirror 12: 0xc0fb43f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9b74: skipped 000444 1782218849.107518 mapwrite 0x03ccf8 thru 0x03ffff (0x3308 bytes) 000445 1782218849.208193 trunc from 0x0017f1 to 0x00fb57 (0xe367 bytes) 000446 1782218849.225834 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x86897e1e (stale) Mirror 9: 0x86897e1e (stale) Mirror 10: 0x86897e1e (stale) Mirror 12: 0x86897e1e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6aaa4ae3 Mirror 13: 0x6aaa4ae3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000447 1782218850.413013 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x86897e1e (stale) Mirror 9: 0x86897e1e (stale) Mirror 10: 0x86897e1e (stale) Mirror 12: 0x86897e1e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6aaa4ae3 Mirror 13: 0x6aaa4ae3 Mirror 14: 0x6aaa4ae3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000448 1782218852.377129 trunc from 0x040000 to 0x00e83e (0x317c1 bytes) 000449 1782218852.486712 punch from 0x00058b to 0x00af8b (0xaa01 bytes) 000450 1782218852.496858 fallocate 0x033d81 thru 0x034c9a (0xf1a bytes) 000451 1782218852.507348 fallocate 0x00871c thru 0x013636 (0xaf1b bytes) 000452 1782218852.514154 trunc from 0x0228eb to 0x027e72 (0x5588 bytes) 000453 1782218852.520422 punch from 0x031558 to 0x034c9a (0x3743 bytes) 000454 1782218852.528251 write 0x036ba7 thru 0x03ffff (0x9459 bytes) 000455 1782218852.530389 trunc from 0x040000 to 0x001a52 (0x3e5ad bytes) 000456 1782218852.567034 read 0x000175 thru 0x001a52 (0x18de bytes) 000457 1782218852.577470 write 0x02264f thru 0x0236dc (0x108e bytes) 000458 1782218852.579086 trunc from 0x0236dd to 0x033abe (0x103e2 bytes) 000459 1782218852.606021 trunc from 0x033abf to 0x00e15d (0x25961 bytes) 000460 1782218852.628766 mapwrite 0x0197b4 thru 0x020867 (0x70b4 bytes) 000461 1782218852.720640 write 0x01c5d1 thru 0x0267a2 (0xa1d2 bytes) 000462 1782218852.724297 write 0x026305 thru 0x035476 (0xf172 bytes) 000464 1782218852.734309 trunc from 0x02b098 to 0x02ff1d (0x4e86 bytes) 000465 1782218852.786198 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35477): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35477): Mirror 7: 0xb63f8e04 (stale) Mirror 9: 0xb63f8e04 (stale) Mirror 10: 0xb63f8e04 (stale) Mirror 12: 0xb63f8e04 (stale) Mirror 13: 0x7a892891 (stale) Mirror 14: 0x7a892891 (stale) CRC-32 checksum value for chunk [0, 0x35477): Mirror 1: 0xa88438f8 Mirror 15: 0xa88438f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35477: skipped 000466 1782218854.390042 trunc from 0x028db7 to 0x02c6ae (0x38f8 bytes) 000467 1782218854.481491 write 0x02fe38 thru 0x035539 (0x5702 bytes) 000468 1782218854.502181 mapread 0x0111f0 thru 0x015766 (0x4577 bytes) 000471 1782218854.531145 trunc from 0x03553a to 0x010118 (0x25421 bytes) 000473 1782218854.583837 trunc from 0x010119 to 0x027003 (0x16eeb bytes) 000474 1782218854.634485 read 0x0046fc thru 0x00ebb3 (0xa4b8 bytes) 000475 1782218854.649403 write 0x0062a4 thru 0x0117a2 (0xb4ff bytes) 000476 1782218854.657564 read 0x01ccf6 thru 0x0205f9 (0x3904 bytes) 000477 1782218854.685176 fallocate 0x038f24 thru 0x03ffff (0x70dc bytes) 000478 1782218854.705976 fallocate 0x012a03 thru 0x01e2e6 (0xb8e4 bytes) 000479 1782218854.734873 read 0x025b9f thru 0x033b44 (0xdfa6 bytes) 000480 1782218854.777845 punch from 0x03d056 to 0x03f582 (0x252d bytes) 000481 1782218854.787145 mapread 0x03d61d thru 0x03e488 (0xe6c bytes) 000482 1782218854.814425 punch from 0x01b0f6 to 0x02a2b4 (0xf1bf bytes) 000484 1782218854.835400 mapread 0x0230a6 thru 0x031486 (0xe3e1 bytes) 000486 1782218854.930113 mapread 0x02d9fc thru 0x034932 (0x6f37 bytes) 000487 1782218854.936417 mapread 0x038a0d thru 0x03b83d (0x2e31 bytes) 000488 1782218854.957545 read 0x001144 thru 0x003975 (0x2832 bytes) 000489 1782218854.959515 fallocate 0x010dad thru 0x0113e9 (0x63d bytes) 000490 1782218854.983156 trunc from 0x040000 to 0x029e3c (0x161c3 bytes) 000491 1782218855.039772 mapread 0x00ad54 thru 0x00ca2d (0x1cda bytes) 000492 1782218855.046354 mapread 0x027e0a thru 0x029e3c (0x2033 bytes) 000494 1782218855.050489 fallocate 0x00fb99 thru 0x01a96d (0xadd5 bytes) 000495 1782218855.069253 trunc from 0x029e3d to 0x002cf6 (0x27146 bytes) 000496 1782218855.118438 write 0x03d821 thru 0x03ffff (0x27df bytes) 000497 1782218855.130762 MIRROR_RESYNC 8 mirrors 000498 1782218857.019206 mapwrite 0x039123 thru 0x03ffff (0x6edd bytes) 000499 1782218857.145495 fallocate 0x0267f9 thru 0x028535 (0x1d3d bytes) 000500 1782218857.153489 mapwrite 0x010bb4 thru 0x01ebeb (0xe038 bytes) 000504 1782218857.228268 fallocate 0x02e244 thru 0x032dc4 (0x4b81 bytes) 000505 1782218857.244884 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x61e1a96c (stale) Mirror 9: 0x61e1a96c (stale) Mirror 10: 0x61e1a96c (stale) Mirror 12: 0x61e1a96c (stale) Mirror 13: 0x61e1a96c (stale) Mirror 15: 0x61e1a96c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x61e1a96c (stale) Mirror 9: 0x61e1a96c (stale) Mirror 10: 0x61e1a96c (stale) Mirror 12: 0x61e1a96c (stale) Mirror 13: 0x61e1a96c (stale) Mirror 15: 0x61e1a96c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000506 1782218858.065361 punch from 0x022756 to 0x02ceab (0xa756 bytes) 000507 1782218858.075879 trunc from 0x026eaf to 0x02c4dc (0x562e bytes) 000508 1782218858.083409 trunc from 0x017082 to 0x021b21 (0xaaa0 bytes) 000509 1782218858.091077 mapwrite 0x001b84 thru 0x004e15 (0x3292 bytes) 000510 1782218858.144085 mapwrite 0x031446 thru 0x035242 (0x3dfd bytes) 000511 1782218858.185462 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x61e1a96c (stale) Mirror 9: 0x61e1a96c (stale) Mirror 10: 0x61e1a96c (stale) Mirror 12: 0x61e1a96c (stale) Mirror 13: 0x61e1a96c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x61e1a96c (stale) Mirror 9: 0x61e1a96c (stale) Mirror 10: 0x61e1a96c (stale) Mirror 12: 0x61e1a96c (stale) Mirror 13: 0x61e1a96c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000512 1782218859.164556 trunc from 0x040000 to 0x03a963 (0x569c bytes) 000513 1782218859.191138 fallocate 0x017755 thru 0x01ca31 (0x52dd bytes) 000514 1782218859.202936 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a964): Mirror 7: 0x53e0f5a4 (stale) Mirror 9: 0x53e0f5a4 (stale) Mirror 10: 0x53e0f5a4 (stale) Mirror 13: 0x53e0f5a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a964): Mirror 7: 0x53e0f5a4 (stale) Mirror 9: 0x53e0f5a4 (stale) Mirror 10: 0x53e0f5a4 (stale) Mirror 13: 0x53e0f5a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a964: skipped 000515 1782218859.823431 trunc from 0x03147c to 0x03a963 (0x94e8 bytes) 000516 1782218859.833063 mapwrite 0x00fe58 thru 0x013ed5 (0x407e bytes) 000517 1782218860.013807 mapwrite 0x01c515 thru 0x0248f8 (0x83e4 bytes) 000518 1782218860.112210 write 0x011b99 thru 0x014cd0 (0x3138 bytes) 000519 1782218860.127537 mapwrite 0x00e766 thru 0x0101a8 (0x1a43 bytes) 000520 1782218860.170339 trunc from 0x00a06d to 0x014c8b (0xac1f bytes) 000521 1782218860.185019 mapwrite 0x029ad4 thru 0x031939 (0x7e66 bytes) 000522 1782218860.259876 read 0x006ac8 thru 0x012efe (0xc437 bytes) 000523 1782218860.316344 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a964): Mirror 9: 0x53e0f5a4 (stale) Mirror 10: 0x53e0f5a4 (stale) Mirror 13: 0x53e0f5a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a964): Mirror 9: 0x53e0f5a4 (stale) Mirror 10: 0x53e0f5a4 (stale) Mirror 13: 0x53e0f5a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a964: skipped 000524 1782218861.269048 mapread 0x035515 thru 0x03a963 (0x544f bytes) 000525 1782218861.315414 punch from 0x0379db to 0x03a963 (0x2f89 bytes) 000526 1782218861.325936 trunc from 0x00b86e to 0x00f2b3 (0x3a46 bytes) 000527 1782218861.335523 trunc from 0x03a964 to 0x03c3b7 (0x1a54 bytes) 000528 1782218861.415498 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c3b8): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3b8): Mirror 9: 0x116dd0ff (stale) Mirror 10: 0x116dd0ff (stale) Mirror 13: 0x116dd0ff (stale) CRC-32 checksum value for chunk [0, 0x3c3b8): Mirror 1: 0x6ebe158c Mirror 14: 0x6ebe158c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3b8: skipped 000530 1782218863.672586 mapread 0x02d459 thru 0x03bcd3 (0xe87b bytes) 000531 1782218863.796563 mapwrite 0x039945 thru 0x03ffff (0x66bb bytes) 000532 1782218864.078387 punch from 0x017e04 to 0x018dd7 (0xfd4 bytes) 000533 1782218864.089228 write 0x03a198 thru 0x03abd5 (0xa3e bytes) 000535 1782218864.094530 punch from 0x039d32 to 0x03a3c6 (0x695 bytes) 000536 1782218864.121796 mapwrite 0x019f72 thru 0x021f75 (0x8004 bytes) 000538 1782218864.202321 mapwrite 0x01a0db thru 0x028743 (0xe669 bytes) 000540 1782218864.349201 trunc from 0x011c27 to 0x0159e6 (0x3dc0 bytes) 000541 1782218864.364191 punch from 0x01305a to 0x01e7e1 (0xb788 bytes) 000542 1782218864.376203 mapread 0x01be2f thru 0x01bfc2 (0x194 bytes) 000543 1782218864.432845 mapwrite 0x02ed14 thru 0x03c4f9 (0xd7e6 bytes) 000544 1782218864.618439 trunc from 0x040000 to 0x02b169 (0x14e96 bytes) 000545 1782218864.680117 trunc from 0x02b16a to 0x03c299 (0x11130 bytes) 000546 1782218864.708408 read 0x02f0d0 thru 0x03a5d5 (0xb506 bytes) 000547 1782218864.739358 fallocate 0x01fed0 thru 0x02ca64 (0xcb95 bytes) 000548 1782218864.746572 mapread 0x0171ab thru 0x02566e (0xe4c4 bytes) 000549 1782218864.752076 mapwrite 0x0185d8 thru 0x020afb (0x8524 bytes) 000550 1782218864.779614 write 0x006c94 thru 0x0136b5 (0xca22 bytes) 000551 1782218864.806244 trunc from 0x01d120 to 0x02cee0 (0xfdc1 bytes) 000553 1782218864.835186 MIRROR_RESYNC 5 mirrors 000554 1782218867.700952 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) 000555 1782218868.483170 mapwrite 0x015ebd thru 0x01b2bc (0x5400 bytes) 000556 1782218868.764584 fallocate 0x024cbe thru 0x0310cf (0xc412 bytes) 000557 1782218868.793552 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c29a): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c29a): Mirror 10: 0x7e150c93 (stale) Mirror 13: 0x7e150c93 (stale) Mirror 14: 0x7e150c93 (stale) CRC-32 checksum value for chunk [0, 0x3c29a): Mirror 1: 0xfc969388 Mirror 15: 0xfc969388 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c29a: skipped 000558 1782218870.576892 fallocate 0x017d36 thru 0x024f0b (0xd1d6 bytes) 000559 1782218870.654741 fallocate 0x010636 thru 0x01bf47 (0xb912 bytes) 000560 1782218870.662311 fallocate 0x019f72 thru 0x021c28 (0x7cb7 bytes) 000561 1782218870.672454 mapread 0x008501 thru 0x00d4ab (0x4fab bytes) 000562 1782218870.712754 mapwrite 0x0000fb thru 0x009355 (0x925b bytes) 000563 1782218870.780886 mapread 0x0339ce thru 0x0376d7 (0x3d0a bytes) 000564 1782218870.804416 trunc from 0x012677 to 0x01cba7 (0xa531 bytes) 000565 1782218870.833011 trunc from 0x03c29a to 0x01dbb9 (0x1e6e0 bytes) 000566 1782218870.912382 MIRROR_RESYNC 5 mirrors 000567 1782218872.546235 mapread 0x010400 thru 0x01b49b (0xb09c bytes) 000568 1782218872.647586 punch from 0x0130fa to 0x01dbb9 (0xaac0 bytes) 000569 1782218872.799786 MIRROR_RESYNC 5 mirrors 000570 1782218874.954885 MIRROR_EXTEND to 6 mirrors 000571 1782218875.847625 trunc from 0x01dbba to 0x00ef44 (0xec75 bytes) 000573 1782218875.931773 read 0x005b34 thru 0x00ef44 (0x9411 bytes) 000574 1782218875.941054 read 0x00835e thru 0x00abd2 (0x2875 bytes) 000575 1782218875.941836 trunc from 0x00ef45 to 0x03b0fc (0x2c1b8 bytes) 000576 1782218875.970385 mapread 0x00a9eb thru 0x0109e8 (0x5ffe bytes) 000577 1782218875.988975 trunc from 0x03b0fd to 0x01cca9 (0x1e453 bytes) 000578 1782218876.022719 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ccaa): Mirror 10: 0xe51c7caf (stale) Mirror 13: 0xe51c7caf (stale) Mirror 15: 0xe51c7caf (stale) Mirror 16: 0xe51c7caf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ccaa): Mirror 10: 0xe51c7caf (stale) Mirror 13: 0xe51c7caf (stale) Mirror 15: 0xe51c7caf (stale) Mirror 16: 0xe51c7caf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ccaa: skipped 000580 1782218876.681529 trunc from 0x005293 to 0x00a217 (0x4f85 bytes) 000581 1782218876.692819 trunc from 0x001b69 to 0x004588 (0x2a20 bytes) 000583 1782218876.717774 write 0x036401 thru 0x037d99 (0x1999 bytes) 000584 1782218876.731352 punch from 0x02ff9a to 0x037d99 (0x7e00 bytes) 000585 1782218876.763417 mapread 0x02ff96 thru 0x037d99 (0x7e04 bytes) 000586 1782218876.853389 read 0x011d67 thru 0x01f7e1 (0xda7b bytes) 000587 1782218876.888333 mapread 0x019a6e thru 0x0247fe (0xad91 bytes) 000588 1782218876.936143 mapwrite 0x002e72 thru 0x012baa (0xfd39 bytes) 000589 1782218877.017984 write 0x00e426 thru 0x014dea (0x69c5 bytes) 000590 1782218877.023309 punch from 0x00b1ba to 0x0192dc (0xe123 bytes) 000591 1782218877.065972 write 0x026543 thru 0x028251 (0x1d0f bytes) 000592 1782218877.067578 read 0x00b1ae thru 0x00cb67 (0x19ba bytes) 000593 1782218877.102989 read 0x006190 thru 0x01309c (0xcf0d bytes) 000594 1782218877.104159 mapread 0x00442b thru 0x011895 (0xd46b bytes) 000595 1782218877.116072 MIRROR_RESYNC 5 mirrors 000597 1782218879.825276 punch from 0x01a035 to 0x025232 (0xb1fe bytes) 000598 1782218879.904972 fallocate 0x02d5b8 thru 0x03138b (0x3dd4 bytes) 000599 1782218879.930076 mapread 0x01cbfa thru 0x024150 (0x7557 bytes) 000600 1782218880.018885 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37d9a): Mirror 13: 0x42e0acfc (stale) Mirror 15: 0x42e0acfc (stale) Mirror 16: 0x42e0acfc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d9a): Mirror 13: 0x42e0acfc (stale) Mirror 15: 0x42e0acfc (stale) Mirror 16: 0x42e0acfc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d9a: skipped 000602 1782218880.920156 mapwrite 0x01e893 thru 0x021595 (0x2d03 bytes) 000603 1782218880.989200 mapwrite 0x023047 thru 0x02de0c (0xadc6 bytes) 000604 1782218881.042517 write 0x0161a0 thru 0x01e7b0 (0x8611 bytes) 000605 1782218881.059425 write 0x0318fd thru 0x033dc9 (0x24cd bytes) 000607 1782218881.068376 read 0x0048ea thru 0x00af4d (0x6664 bytes) 000608 1782218881.084538 mapwrite 0x021763 thru 0x02bf5d (0xa7fb bytes) 000609 1782218881.143364 fallocate 0x013a59 thru 0x015ed6 (0x247e bytes) 000611 1782218881.161075 punch from 0x01d609 to 0x02d012 (0xfa0a bytes) 000612 1782218881.183392 punch from 0x02ad10 to 0x02f255 (0x4546 bytes) 000614 1782218881.220849 fallocate 0x022800 thru 0x02d509 (0xad0a bytes) 000615 1782218881.244005 trunc from 0x01b4cf to 0x0291e5 (0xdd17 bytes) 000616 1782218881.259642 write 0x03e279 thru 0x03fe6f (0x1bf7 bytes) 000617 1782218881.264950 trunc from 0x037c25 to 0x03fe6f (0x824b bytes) 000618 1782218881.300443 trunc from 0x03fe70 to 0x0262d8 (0x19b97 bytes) 000619 1782218881.342582 mapread 0x00b792 thru 0x011537 (0x5da6 bytes) 000620 1782218881.366318 write 0x03c72d thru 0x03fc6c (0x3540 bytes) 000621 1782218881.380275 read 0x01c2be thru 0x023a26 (0x7769 bytes) 000622 1782218881.408080 MIRROR_RESYNC 4 mirrors 000623 1782218883.512254 mapwrite 0x02db4f thru 0x030134 (0x25e6 bytes) 000624 1782218883.710987 MIRROR_RESYNC 4 mirrors 000625 1782218885.786453 read 0x017fe3 thru 0x0194f3 (0x1511 bytes) 000626 1782218885.794048 fallocate 0x02bf2a thru 0x03ace3 (0xedba bytes) 000627 1782218885.853013 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc6d): Mirror 13: 0x70ce381b (stale) Mirror 15: 0x70ce381b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc6d): Mirror 13: 0x70ce381b (stale) Mirror 15: 0x70ce381b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc6d: skipped 000628 1782218886.313210 fallocate 0x0268a1 thru 0x02d65c (0x6dbc bytes) 000629 1782218886.326510 MIRROR_RESYNC 3 mirrors 000630 1782218887.730044 mapwrite 0x01e60b thru 0x02da55 (0xf44b bytes) 000631 1782218887.897031 mapwrite 0x013ff8 thru 0x023cdd (0xfce6 bytes) 000632 1782218887.998363 write 0x035819 thru 0x03ffff (0xa7e7 bytes) 000633 1782218888.009566 mapread 0x00ffa0 thru 0x01b27c (0xb2dd bytes) 000634 1782218888.030212 write 0x02ec60 thru 0x034fcb (0x636c bytes) 000635 1782218888.042197 mapwrite 0x009ed9 thru 0x00c29b (0x23c3 bytes) 000636 1782218888.105108 fallocate 0x036f52 thru 0x03e46d (0x751c bytes) 000637 1782218888.112490 trunc from 0x008d4d to 0x017206 (0xe4ba bytes) 000638 1782218888.124443 write 0x031643 thru 0x032756 (0x1114 bytes) 000639 1782218888.127882 punch from 0x023d5a to 0x02eada (0xad81 bytes) 000640 1782218888.148552 mapread 0x03e14f thru 0x03ffff (0x1eb1 bytes) 000641 1782218888.153576 read 0x01b53f thru 0x023cd1 (0x8793 bytes) 000642 1782218888.175584 fallocate 0x030e52 thru 0x030e55 (0x4 bytes) 000643 1782218888.197942 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x591af97d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x591af97d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000644 1782218888.628559 trunc from 0x040000 to 0x01c574 (0x23a8b bytes) 000645 1782218888.660499 mapwrite 0x011a83 thru 0x018edd (0x745b bytes) 000646 1782218888.717938 trunc from 0x00cdcf to 0x010852 (0x3a84 bytes) 000647 1782218888.731636 trunc from 0x000e61 to 0x0016de (0x87e bytes) 000648 1782218888.744414 write 0x02755b thru 0x0374cd (0xff73 bytes) 000649 1782218888.755772 punch from 0x0088c1 to 0x011c79 (0x93b9 bytes) 000650 1782218888.767349 mapwrite 0x0384f6 thru 0x03ffff (0x7b0a bytes) 000651 1782218888.864271 punch from 0x02b368 to 0x031a84 (0x671d bytes) 000652 1782218888.874187 mapread 0x01d917 thru 0x029801 (0xbeeb bytes) 000653 1782218888.916507 write 0x0173d0 thru 0x017a32 (0x663 bytes) 000654 1782218888.918871 fallocate 0x000e60 thru 0x0080be (0x725f bytes) 000655 1782218888.942097 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x591af97d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x82552bb9 Mirror 14: 0x82552bb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000656 1782218890.174830 trunc from 0x0264bd to 0x02a3a7 (0x3eeb bytes) 000657 1782218890.253305 trunc from 0x03aa9f to 0x03ffff (0x5561 bytes) 000658 1782218890.264782 punch from 0x0260a6 to 0x02bd01 (0x5c5c bytes) 000659 1782218890.274031 mapwrite 0x00e204 thru 0x015b27 (0x7924 bytes) 000660 1782218890.386452 mapwrite 0x0125f7 thru 0x01c3f3 (0x9dfd bytes) 000661 1782218890.482261 read 0x03a73c thru 0x03ffff (0x58c4 bytes) 000662 1782218890.508441 fallocate 0x0170bd thru 0x022cb1 (0xbbf5 bytes) 000663 1782218890.522493 trunc from 0x0152f0 to 0x023052 (0xdd63 bytes) 000664 1782218890.532078 read 0x03db55 thru 0x03ffff (0x24ab bytes) 000665 1782218890.540246 mapwrite 0x024f28 thru 0x029090 (0x4169 bytes) 000666 1782218890.604809 fallocate 0x03155b thru 0x03ffff (0xeaa5 bytes) 000667 1782218890.618116 read 0x006005 thru 0x014f32 (0xef2e bytes) 000668 1782218890.620593 punch from 0x00275f to 0x0043e6 (0x1c88 bytes) 000669 1782218890.643550 trunc from 0x040000 to 0x012008 (0x2dff7 bytes) 000670 1782218890.722806 punch from 0x00e0aa to 0x010229 (0x2180 bytes) 000671 1782218890.745326 read 0x005c66 thru 0x012008 (0xc3a3 bytes) 000672 1782218890.785366 trunc from 0x00c992 to 0x012008 (0x5677 bytes) 000674 1782218890.838054 MIRROR_RESYNC 3 mirrors 000675 1782218892.765156 read 0x01139e thru 0x012008 (0xc6b bytes) 000676 1782218892.786284 mapwrite 0x0171c5 thru 0x01f4db (0x8317 bytes) 000677 1782218892.989150 mapwrite 0x02badf thru 0x02e28a (0x27ac bytes) 000678 1782218893.149300 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e28b): Mirror 13: 0xd8cd0860 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e28b): Mirror 13: 0xd8cd0860 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e28b: skipped 000679 1782218893.968870 MIRROR_RESYNC 2 mirrors 000680 1782218895.656723 trunc from 0x02e28b to 0x008d4b (0x2553f bytes) 000681 1782218895.793242 punch from 0x007b64 to 0x008d4b (0x11e8 bytes) 000682 1782218895.799485 fallocate 0x02d806 thru 0x031e6e (0x4669 bytes) 000683 1782218895.814150 mapwrite 0x010151 thru 0x01fe47 (0xfcf7 bytes) 000685 1782218896.176132 fallocate 0x0238ac thru 0x027556 (0x3cab bytes) 000686 1782218896.202145 mapread 0x003fdc thru 0x008c58 (0x4c7d bytes) 000687 1782218896.238382 trunc from 0x00db06 to 0x01059e (0x2a99 bytes) 000688 1782218896.253473 trunc from 0x0174b4 to 0x01a3db (0x2f28 bytes) 000689 1782218896.271309 trunc from 0x025ee3 to 0x027556 (0x1674 bytes) 000690 1782218896.288110 read 0x004a9c thru 0x009ae4 (0x5049 bytes) 000691 1782218896.294117 trunc from 0x00dc0c to 0x017e62 (0xa257 bytes) 000692 1782218896.316494 write 0x0322a8 thru 0x03ae87 (0x8be0 bytes) 000693 1782218896.319811 fallocate 0x000c71 thru 0x008a4a (0x7dda bytes) 000694 1782218896.330467 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ae88): Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae88): Mirror 13: 0x3fb23cd (stale) CRC-32 checksum value for chunk [0, 0x3ae88): Mirror 1: 0x42f72ea4 Mirror 14: 0x42f72ea4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae88: skipped 000695 1782218898.326148 punch from 0x037aac to 0x03ae87 (0x33dc bytes) 000696 1782218898.430943 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae88): Mirror 13: 0x3fb23cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae88): Mirror 13: 0x3fb23cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae88: skipped 000697 1782218898.800507 trunc from 0x0246f7 to 0x03069d (0xbfa7 bytes) 000698 1782218898.805839 trunc from 0x03ae88 to 0x02219a (0x18ced bytes) 000700 1782218898.840018 mapwrite 0x030c12 thru 0x03d7aa (0xcb99 bytes) 000701 1782218898.960414 fallocate 0x01a987 thru 0x027071 (0xc6eb bytes) 000702 1782218898.976489 fallocate 0x01f3d7 thru 0x026f46 (0x7b70 bytes) 000703 1782218898.986250 read 0x02289a thru 0x02f05c (0xc7c3 bytes) 000704 1782218899.000758 write 0x012c53 thru 0x019919 (0x6cc7 bytes) 000705 1782218899.016934 trunc from 0x03ca40 to 0x03d7aa (0xd6b bytes) 000706 1782218899.024069 mapwrite 0x00483e thru 0x00a303 (0x5ac6 bytes) 000707 1782218899.072553 write 0x03c2bd thru 0x03ffff (0x3d43 bytes) 000708 1782218899.083934 write 0x006263 thru 0x010192 (0x9f30 bytes) 000710 1782218899.089885 fallocate 0x0286af thru 0x02b2cb (0x2c1d bytes) 000711 1782218899.098033 write 0x015aed thru 0x01aeae (0x53c2 bytes) 000712 1782218899.106380 trunc from 0x040000 to 0x00fd85 (0x3027a bytes) 000713 1782218899.135120 punch from 0x00744c to 0x00ef4c (0x7b01 bytes) 000714 1782218899.157986 mapwrite 0x017cae thru 0x01bb19 (0x3e6c bytes) 000715 1782218899.236476 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1bb1a): Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bb1a): Mirror 13: 0xc62cded9 (stale) CRC-32 checksum value for chunk [0, 0x1bb1a): Mirror 1: 0x8c8a4347 Mirror 14: 0x8c8a4347 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bb1a: skipped 000716 1782218901.566506 read 0x019e61 thru 0x01b4da (0x167a bytes) 000717 1782218901.620891 mapwrite 0x0338a2 thru 0x03ffff (0xc75e bytes) 000718 1782218901.952102 mapwrite 0x021cf0 thru 0x027f33 (0x6244 bytes) 000719 1782218902.018257 trunc from 0x0003a4 to 0x00eb6c (0xe7c9 bytes) 000720 1782218902.029845 read 0x0022fa thru 0x0096b3 (0x73ba bytes) 000721 1782218902.080450 read 0x035919 thru 0x03ffff (0xa6e7 bytes) 000722 1782218902.081855 fallocate 0x005985 thru 0x00f2db (0x9957 bytes) 000723 1782218902.095815 trunc from 0x035aa3 to 0x03d601 (0x7b5f bytes) 000724 1782218902.109651 punch from 0x020bb2 to 0x027db4 (0x7203 bytes) 000725 1782218902.120933 mapwrite 0x01e508 thru 0x0224cb (0x3fc4 bytes) 000726 1782218902.176811 fallocate 0x02d994 thru 0x033469 (0x5ad6 bytes) 000727 1782218902.188890 write 0x01a398 thru 0x022b54 (0x87bd bytes) 000729 1782218902.197749 fallocate 0x03cf2d thru 0x03ffff (0x30d3 bytes) 000730 1782218902.207833 trunc from 0x01ae70 to 0x0232d3 (0x8464 bytes) 000731 1782218902.245843 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xdccb9986 (stale) Mirror 14: 0x784d1fbe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbd3b6ba5 Mirror 15: 0xbd3b6ba5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000732 1782218904.536530 trunc from 0x040000 to 0x0354dd (0xab22 bytes) 000733 1782218904.650821 trunc from 0x00b2fe to 0x016641 (0xb344 bytes) 000734 1782218904.663949 trunc from 0x0354de to 0x01b30c (0x1a1d1 bytes) 000735 1782218904.719436 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1b30d): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b30d): Mirror 13: 0x7ecd16e6 (stale) Mirror 14: 0xb241350c (stale) Mirror 15: 0xa7578ec5 (stale) CRC-32 checksum value for chunk [0, 0x1b30d): Mirror 1: 0xc60f5cd9 Mirror 16: 0xc60f5cd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b30d: skipped 000736 1782218906.504562 mapwrite 0x0047a9 thru 0x013156 (0xe9ae bytes) 000737 1782218906.752998 write 0x01f374 thru 0x02e9bd (0xf64a bytes) 000738 1782218906.773463 punch from 0x01f3a8 to 0x020a59 (0x16b2 bytes) 000739 1782218906.832255 read 0x01e563 thru 0x022a59 (0x44f7 bytes) 000740 1782218906.868452 trunc from 0x02e9be to 0x030602 (0x1c45 bytes) 000742 1782218906.923454 punch from 0x0238aa to 0x028b1a (0x5271 bytes) 000743 1782218906.934168 mapread 0x009e98 thru 0x00f822 (0x598b bytes) 000745 1782218906.937262 fallocate 0x03b9ff thru 0x03ffff (0x4601 bytes) 000746 1782218906.954743 read 0x032c27 thru 0x03a2af (0x7689 bytes) 000747 1782218906.968467 read 0x00fd0c thru 0x01d07e (0xd373 bytes) 000748 1782218906.969763 mapwrite 0x03adf0 thru 0x03f0bf (0x42d0 bytes) 000749 1782218907.024390 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xdccb9986 (stale) Mirror 14: 0x784d1fbe (stale) Mirror 15: 0xbd3b6ba5 (stale) Mirror 16: 0x5358701d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8c3fbe8e Mirror 17: 0x8c3fbe8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000750 1782218909.362467 fallocate 0x03077e thru 0x032a6b (0x22ee bytes) 000751 1782218909.426475 trunc from 0x040000 to 0x021471 (0x1eb8e bytes) 000752 1782218909.475817 trunc from 0x021472 to 0x015ea2 (0xb5cf bytes) 000753 1782218909.505413 mapread 0x0010ad thru 0x010d30 (0xfc84 bytes) 000754 1782218909.556649 trunc from 0x015ea3 to 0x01e3de (0x853c bytes) 000757 1782218909.586338 trunc from 0x017410 to 0x01da9c (0x668d bytes) 000758 1782218909.594101 mapread 0x002f42 thru 0x00ac60 (0x7d1f bytes) 000759 1782218909.598533 trunc from 0x005a20 to 0x01170d (0xbcee bytes) 000761 1782218909.610141 mapwrite 0x00dabf thru 0x018ba8 (0xb0ea bytes) 000762 1782218909.687847 read 0x01bd29 thru 0x01e3de (0x26b6 bytes) 000763 1782218909.689311 write 0x02db25 thru 0x035eca (0x83a6 bytes) 000764 1782218909.692551 trunc from 0x00036c to 0x007f7e (0x7c13 bytes) 000765 1782218909.704150 mapread 0x019680 thru 0x026ab6 (0xd437 bytes) 000766 1782218909.719533 mapwrite 0x030e3a thru 0x0373a2 (0x6569 bytes) 000767 1782218909.847422 read 0x0073a6 thru 0x00f906 (0x8561 bytes) 000768 1782218909.893847 trunc from 0x0373a3 to 0x02240e (0x14f94 bytes) 000769 1782218909.959512 mapwrite 0x015aa8 thru 0x01c30e (0x6867 bytes) 000770 1782218910.008502 mapwrite 0x004f6a thru 0x009493 (0x452a bytes) 000771 1782218910.055968 punch from 0x00ddd3 to 0x01d04e (0xf27c bytes) 000772 1782218910.074458 mapwrite 0x02e841 thru 0x036b71 (0x8331 bytes) 000773 1782218910.229312 trunc from 0x02ed48 to 0x032110 (0x33c9 bytes) 000774 1782218910.237874 mapread 0x012294 thru 0x015ede (0x3c4b bytes) 000775 1782218910.267039 write 0x03e967 thru 0x03ffff (0x1699 bytes) 000776 1782218910.269440 write 0x0388e6 thru 0x03ffff (0x771a bytes) 000777 1782218910.284824 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xdccb9986 (stale) Mirror 14: 0x784d1fbe (stale) Mirror 15: 0xbd3b6ba5 (stale) Mirror 16: 0x5358701d (stale) Mirror 17: 0x8c3fbe8e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9fbabc60 Mirror 18: 0x9fbabc60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000778 1782218912.250402 mapread 0x022417 thru 0x02ce95 (0xaa7f bytes) 000779 1782218912.347633 write 0x01451f thru 0x017d99 (0x387b bytes) 000780 1782218912.456128 punch from 0x000d9d to 0x0067d3 (0x5a37 bytes) 000781 1782218912.467205 fallocate 0x0082e2 thru 0x012f97 (0xacb6 bytes) 000782 1782218912.478174 mapwrite 0x033f33 thru 0x03ffff (0xc0cd bytes) 000783 1782218912.582459 trunc from 0x040000 to 0x019cb8 (0x26347 bytes) 000785 1782218912.637905 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x19cb9): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19cb9): Mirror 13: 0x597cd5c0 (stale) Mirror 14: 0xded55f35 (stale) Mirror 15: 0x5cfa0ccf (stale) Mirror 16: 0x85e7880 (stale) Mirror 17: 0x4b263545 (stale) Mirror 18: 0x65725ca2 (stale) CRC-32 checksum value for chunk [0, 0x19cb9): Mirror 1: 0x5bd4a63e Mirror 19: 0x5bd4a63e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19cb9: skipped 000786 1782218914.853005 punch from 0x00580a to 0x009c41 (0x4438 bytes) 000787 1782218914.941370 trunc from 0x008da5 to 0x00db19 (0x4d75 bytes) 000788 1782218914.955002 read 0x0023c3 thru 0x00462b (0x2269 bytes) 000789 1782218914.974346 mapread 0x014712 thru 0x019cb8 (0x55a7 bytes) 000790 1782218915.061841 MIRROR_RESYNC 8 mirrors 000791 1782218919.153945 read 0x010e3c thru 0x019cb8 (0x8e7d bytes) 000792 1782218919.192221 MIRROR_EXTEND to 9 mirrors 000793 1782218921.312629 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) 000794 1782218922.494186 fallocate 0x00f547 thru 0x0111a3 (0x1c5d bytes) 000795 1782218922.634185 trunc from 0x005cb3 to 0x011bed (0xbf3b bytes) 000796 1782218922.649625 trunc from 0x00460f to 0x0088ae (0x42a0 bytes) 000797 1782218922.665606 mapread 0x00916a thru 0x014484 (0xb31b bytes) 000798 1782218922.733230 mapread 0x0060d6 thru 0x0081bf (0x20ea bytes) 000799 1782218922.735971 trunc from 0x017d7d to 0x019cb8 (0x1f3c bytes) 000800 1782218922.752958 MIRROR_RESYNC 8 mirrors 000801 1782218925.640967 fallocate 0x017765 thru 0x0227b4 (0xb050 bytes) 000802 1782218925.698156 mapread 0x00a86a thru 0x0138fa (0x9091 bytes) 000804 1782218925.738859 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x19cb9): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19cb9): Mirror 13: 0xd9dd5a15 (stale) Mirror 14: 0xd9dd5a15 (stale) Mirror 15: 0xd9dd5a15 (stale) Mirror 16: 0xd9dd5a15 (stale) Mirror 18: 0xd9dd5a15 (stale) Mirror 19: 0xd9dd5a15 (stale) Mirror 20: 0xd9dd5a15 (stale) CRC-32 checksum value for chunk [0, 0x19cb9): Mirror 1: 0xd9dd5a15 Mirror 21: 0xd9dd5a15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19cb9: skipped 000805 1782218928.368466 punch from 0x013d8e to 0x019cb8 (0x5f2b bytes) 000806 1782218928.442215 fallocate 0x00a827 thru 0x00b355 (0xb2f bytes) 000807 1782218928.451454 trunc from 0x019cb9 to 0x03b264 (0x215ac bytes) 000808 1782218928.502368 trunc from 0x03b265 to 0x00681e (0x34a46 bytes) 000809 1782218928.543832 trunc from 0x00681f to 0x0193e4 (0x12bc6 bytes) 000811 1782218928.584152 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x193e5): Mirror 14: 0x49daba8e (stale) Mirror 15: 0x49daba8e (stale) Mirror 16: 0x49daba8e (stale) Mirror 18: 0x49daba8e (stale) Mirror 19: 0x49daba8e (stale) Mirror 20: 0x49daba8e (stale) Mirror 21: 0x49daba8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x193e5): Mirror 14: 0x49daba8e (stale) Mirror 15: 0x49daba8e (stale) Mirror 16: 0x49daba8e (stale) Mirror 18: 0x49daba8e (stale) Mirror 19: 0x49daba8e (stale) Mirror 20: 0x49daba8e (stale) Mirror 21: 0x49daba8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x193e5: skipped 000812 1782218929.679851 read 0x0094ee thru 0x0101a5 (0x6cb8 bytes) 000813 1782218929.714545 mapread 0x010dd7 thru 0x011126 (0x350 bytes) 000814 1782218929.760848 trunc from 0x0193e5 to 0x021a43 (0x865f bytes) 000815 1782218929.810230 read 0x020576 thru 0x021a43 (0x14ce bytes) 000816 1782218929.827251 punch from 0x003d67 to 0x007774 (0x3a0e bytes) 000817 1782218929.834789 fallocate 0x031864 thru 0x034f20 (0x36bd bytes) 000818 1782218929.852368 punch from 0x010d3c to 0x016691 (0x5956 bytes) 000819 1782218929.882256 punch from 0x01f82b to 0x021a43 (0x2219 bytes) 000820 1782218929.911342 write 0x00465c thru 0x011df4 (0xd799 bytes) 000821 1782218929.955186 read 0x008be8 thru 0x016882 (0xdc9b bytes) 000822 1782218929.992478 write 0x02f907 thru 0x034bca (0x52c4 bytes) 000823 1782218929.997847 mapread 0x00d915 thru 0x01498b (0x7077 bytes) 000824 1782218930.006278 punch from 0x026b72 to 0x02b6ad (0x4b3c bytes) 000825 1782218930.026963 mapwrite 0x00e018 thru 0x00e2f9 (0x2e2 bytes) 000826 1782218930.090596 write 0x01a0c0 thru 0x0277d8 (0xd719 bytes) 000827 1782218930.108374 mapwrite 0x01f149 thru 0x0275cc (0x8484 bytes) 000828 1782218930.185704 fallocate 0x025f43 thru 0x03088f (0xa94d bytes) 000829 1782218930.211026 trunc from 0x034bcb to 0x01c279 (0x18951 bytes) 000830 1782218930.255746 mapwrite 0x02c655 thru 0x037354 (0xad00 bytes) 000831 1782218930.385222 trunc from 0x037355 to 0x00744a (0x2ff0a bytes) 000832 1782218930.440290 mapwrite 0x0261e9 thru 0x03444f (0xe267 bytes) 000833 1782218930.762275 write 0x00952d thru 0x00a18e (0xc62 bytes) 000834 1782218930.793847 fallocate 0x004b3d thru 0x009d41 (0x5205 bytes) 000835 1782218930.808287 read 0x0242c2 thru 0x02dee1 (0x9c20 bytes) 000836 1782218930.841932 read 0x00df97 thru 0x00eef1 (0xf5b bytes) 000837 1782218930.849527 MIRROR_RESYNC 8 mirrors 000838 1782218935.411882 mapwrite 0x02112e thru 0x02d94e (0xc821 bytes) 000839 1782218935.659639 read 0x021e2a thru 0x028fc0 (0x7197 bytes) 000840 1782218935.660651 trunc from 0x0037fb to 0x00a718 (0x6f1e bytes) 000841 1782218935.685032 trunc from 0x006ef4 to 0x00cdeb (0x5ef8 bytes) 000842 1782218935.697130 mapwrite 0x02b74e thru 0x02bb06 (0x3b9 bytes) 000843 1782218935.728389 mapwrite 0x03d0d9 thru 0x03ffff (0x2f27 bytes) 000844 1782218935.845456 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x33e37257 (stale) Mirror 15: 0x33e37257 (stale) Mirror 16: 0x33e37257 (stale) Mirror 18: 0x33e37257 (stale) Mirror 19: 0x33e37257 (stale) Mirror 20: 0x33e37257 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x33e37257 (stale) Mirror 15: 0x33e37257 (stale) Mirror 16: 0x33e37257 (stale) Mirror 18: 0x33e37257 (stale) Mirror 19: 0x33e37257 (stale) Mirror 20: 0x33e37257 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000846 1782218937.198687 read 0x026211 thru 0x033f65 (0xdd55 bytes) 000847 1782218937.226116 read 0x00cd80 thru 0x012b2b (0x5dac bytes) 000848 1782218937.272223 punch from 0x03a769 to 0x03ffff (0x5897 bytes) 000849 1782218937.286295 punch from 0x00efb0 to 0x01ce49 (0xde9a bytes) 000850 1782218937.306623 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x33e37257 (stale) Mirror 16: 0x33e37257 (stale) Mirror 18: 0x33e37257 (stale) Mirror 19: 0x33e37257 (stale) Mirror 20: 0x33e37257 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x33e37257 (stale) Mirror 16: 0x33e37257 (stale) Mirror 18: 0x33e37257 (stale) Mirror 19: 0x33e37257 (stale) Mirror 20: 0x33e37257 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000851 1782218938.570268 mapread 0x00c1c1 thru 0x00fc46 (0x3a86 bytes) 000852 1782218938.660332 trunc from 0x01c1ec to 0x0278e3 (0xb6f8 bytes) 000853 1782218938.676685 mapwrite 0x014675 thru 0x020067 (0xb9f3 bytes) 000854 1782218938.862009 trunc from 0x040000 to 0x002754 (0x3d8ab bytes) 000856 1782218938.949248 trunc from 0x00182f to 0x002754 (0xf26 bytes) 000858 1782218938.978276 trunc from 0x002755 to 0x011ee6 (0xf792 bytes) 000859 1782218939.017221 read 0x0000d1 thru 0x009a7b (0x99ab bytes) 000860 1782218939.050463 trunc from 0x011ee7 to 0x01af83 (0x909d bytes) 000861 1782218939.108244 punch from 0x00d6dd to 0x0108f7 (0x321b bytes) 000862 1782218939.122160 trunc from 0x000db7 to 0x0011db (0x425 bytes) 000863 1782218939.133217 fallocate 0x0157df thru 0x01ff89 (0xa7ab bytes) 000864 1782218939.149256 trunc from 0x01ff8a to 0x003891 (0x1c6f8 bytes) 000865 1782218939.195423 punch from 0x0037ff to 0x003891 (0x93 bytes) 000866 1782218939.215110 trunc from 0x0003d9 to 0x003891 (0x34b9 bytes) 000867 1782218939.231919 read 0x0001c6 thru 0x003891 (0x36cc bytes) 000868 1782218939.251735 write 0x012559 thru 0x01ab16 (0x85be bytes) 000869 1782218939.261168 mapwrite 0x03c440 thru 0x03ffff (0x3bc0 bytes) 000870 1782218939.438288 read 0x026e0e thru 0x036b89 (0xfd7c bytes) 000871 1782218939.449983 write 0x009fba thru 0x016b8b (0xcbd2 bytes) 000872 1782218939.457360 mapread 0x01d8af thru 0x0260c3 (0x8815 bytes) 000873 1782218939.475328 fallocate 0x00d9c0 thru 0x00fdfb (0x243c bytes) 000874 1782218939.489628 write 0x01358c thru 0x01e916 (0xb38b bytes) 000875 1782218939.513162 write 0x010267 thru 0x0112d2 (0x106c bytes) 000876 1782218939.514497 read 0x0182f3 thru 0x01c900 (0x460e bytes) 000877 1782218939.516444 punch from 0x02b180 to 0x0364bf (0xb340 bytes) 000878 1782218939.526646 read 0x03cfb5 thru 0x03ffff (0x304b bytes) 000879 1782218939.530045 punch from 0x02cded to 0x03aa99 (0xdcad bytes) 000880 1782218939.539190 mapread 0x01d35e thru 0x02225c (0x4eff bytes) 000881 1782218939.553717 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x33e37257 (stale) Mirror 16: 0x33e37257 (stale) Mirror 18: 0x33e37257 (stale) Mirror 19: 0x33e37257 (stale) Mirror 20: 0x33e37257 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x99609573 Mirror 21: 0x99609573 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000882 1782218941.789234 punch from 0x01a88d to 0x026f3a (0xc6ae bytes) 000883 1782218941.878132 trunc from 0x02f87f to 0x0386de (0x8e60 bytes) 000884 1782218941.887985 mapread 0x02bed4 thru 0x032f5b (0x7088 bytes) 000885 1782218941.942280 read 0x032574 thru 0x036ca3 (0x4730 bytes) 000886 1782218941.966736 trunc from 0x006b20 to 0x0120ab (0xb58c bytes) 000887 1782218941.980318 trunc from 0x02b122 to 0x03a274 (0xf153 bytes) 000889 1782218941.989204 trunc from 0x011129 to 0x0131e9 (0x20c1 bytes) 000890 1782218941.997521 write 0x03b8f7 thru 0x03ffff (0x4709 bytes) 000891 1782218942.017057 fallocate 0x02765c thru 0x0361af (0xeb54 bytes) 000892 1782218942.026030 write 0x0128ad thru 0x017311 (0x4a65 bytes) 000895 1782218942.041829 punch from 0x02c029 to 0x02ce0c (0xde4 bytes) 000896 1782218942.054506 MIRROR_RESYNC 7 mirrors 000897 1782218945.832586 MIRROR_RESYNC 7 mirrors 000898 1782218946.656843 trunc from 0x040000 to 0x0088d8 (0x37727 bytes) 000899 1782218946.696529 mapwrite 0x00fbeb thru 0x010b1c (0xf32 bytes) 000900 1782218946.773222 write 0x02b2fb thru 0x030d42 (0x5a48 bytes) 000902 1782218946.790313 fallocate 0x03533e thru 0x03b828 (0x64eb bytes) 000905 1782218946.802206 mapread 0x01fcac thru 0x02f6c9 (0xfa1e bytes) 000907 1782218946.853223 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30d43): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d43): Mirror 15: 0x52e7bcda (stale) Mirror 16: 0x52e7bcda (stale) Mirror 18: 0x52e7bcda (stale) Mirror 19: 0x52e7bcda (stale) Mirror 20: 0x52e7bcda (stale) Mirror 21: 0x52e7bcda (stale) CRC-32 checksum value for chunk [0, 0x30d43): Mirror 1: 0xc277196a Mirror 22: 0xc277196a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d43: skipped 000908 1782218948.210343 trunc from 0x030d43 to 0x01733d (0x19a05 bytes) 000909 1782218948.340765 punch from 0x002f3d to 0x005f4b (0x300f bytes) 000910 1782218948.358744 trunc from 0x01021b to 0x01733d (0x7123 bytes) 000911 1782218948.367869 punch from 0x008dfa to 0x00d0ee (0x42f5 bytes) 000912 1782218948.388614 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1733e): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1733e): Mirror 15: 0xaf2073ae (stale) Mirror 16: 0xaf2073ae (stale) Mirror 18: 0xaf2073ae (stale) Mirror 19: 0xaf2073ae (stale) Mirror 20: 0xaf2073ae (stale) Mirror 21: 0xaf2073ae (stale) Mirror 22: 0x33c84ea0 (stale) CRC-32 checksum value for chunk [0, 0x1733e): Mirror 1: 0xe9074d42 Mirror 23: 0xe9074d42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1733e: skipped 000913 1782218950.196408 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1733e): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1733e): Mirror 15: 0xaf2073ae (stale) Mirror 16: 0xaf2073ae (stale) Mirror 18: 0xaf2073ae (stale) Mirror 19: 0xaf2073ae (stale) Mirror 20: 0xaf2073ae (stale) Mirror 21: 0xaf2073ae (stale) Mirror 22: 0x33c84ea0 (stale) CRC-32 checksum value for chunk [0, 0x1733e): Mirror 1: 0xe9074d42 Mirror 23: 0xe9074d42 Mirror 24: 0xe9074d42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1733e: skipped 000914 1782218951.313187 trunc from 0x00cf28 to 0x01733d (0xa416 bytes) 000916 1782218951.344442 trunc from 0x00f2b9 to 0x01733d (0x8085 bytes) 000917 1782218951.349268 mapread 0x0000f7 thru 0x00a53a (0xa444 bytes) 000918 1782218951.392587 mapread 0x012bc9 thru 0x01733d (0x4775 bytes) 000919 1782218951.403813 punch from 0x0031f8 to 0x003ac0 (0x8c9 bytes) 000920 1782218951.410604 mapread 0x010ad7 thru 0x012914 (0x1e3e bytes) 000921 1782218951.412024 write 0x03cc00 thru 0x03ffff (0x3400 bytes) 000922 1782218951.424535 trunc from 0x010e96 to 0x012f73 (0x20de bytes) 000924 1782218951.432353 trunc from 0x040000 to 0x00ab67 (0x35498 bytes) 000925 1782218951.465745 fallocate 0x016675 thru 0x01f8d0 (0x925c bytes) 000926 1782218951.476553 fallocate 0x01dfbb thru 0x02ad93 (0xcdd9 bytes) 000927 1782218951.484136 mapwrite 0x01a48e thru 0x026349 (0xbebc bytes) 000928 1782218951.654463 mapwrite 0x03fcf4 thru 0x03ffff (0x30c bytes) 000929 1782218951.736888 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xe7c1f711 (stale) Mirror 16: 0xe7c1f711 (stale) Mirror 18: 0xe7c1f711 (stale) Mirror 19: 0xe7c1f711 (stale) Mirror 20: 0xe7c1f711 (stale) Mirror 21: 0xe7c1f711 (stale) Mirror 22: 0xef1a9be0 (stale) Mirror 23: 0x6301a475 (stale) Mirror 24: 0x6301a475 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3d10b6a Mirror 25: 0xc3d10b6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000930 1782218953.132841 MIRROR_RESYNC 11 mirrors 000931 1782218955.849961 punch from 0x030d0c to 0x03ab63 (0x9e58 bytes) 000932 1782218955.935328 fallocate 0x02610e thru 0x035443 (0xf336 bytes) 000935 1782218955.958966 trunc from 0x01c770 to 0x02b319 (0xebaa bytes) 000936 1782218955.980910 mapwrite 0x03741d thru 0x03d0ba (0x5c9e bytes) 000937 1782218956.082122 punch from 0x014e72 to 0x01fb54 (0xace3 bytes) 000938 1782218956.096420 write 0x021c53 thru 0x02d6e3 (0xba91 bytes) 000939 1782218956.112321 fallocate 0x00b117 thru 0x00c431 (0x131b bytes) 000940 1782218956.119363 write 0x00be19 thru 0x00e24e (0x2436 bytes) 000941 1782218956.138695 write 0x01e133 thru 0x02b756 (0xd624 bytes) 000942 1782218956.165808 read 0x03c274 thru 0x03ffff (0x3d8c bytes) 000945 1782218956.173311 write 0x018cd4 thru 0x024b2a (0xbe57 bytes) 000946 1782218956.209203 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc3d10b6a (stale) Mirror 16: 0xc3d10b6a (stale) Mirror 18: 0xc3d10b6a (stale) Mirror 19: 0xc3d10b6a (stale) Mirror 20: 0xc3d10b6a (stale) Mirror 21: 0xc3d10b6a (stale) Mirror 23: 0xc3d10b6a (stale) Mirror 24: 0xc3d10b6a (stale) Mirror 25: 0xc3d10b6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc3d10b6a (stale) Mirror 16: 0xc3d10b6a (stale) Mirror 18: 0xc3d10b6a (stale) Mirror 19: 0xc3d10b6a (stale) Mirror 20: 0xc3d10b6a (stale) Mirror 21: 0xc3d10b6a (stale) Mirror 23: 0xc3d10b6a (stale) Mirror 24: 0xc3d10b6a (stale) Mirror 25: 0xc3d10b6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000947 1782218958.174980 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc3d10b6a (stale) Mirror 16: 0xc3d10b6a (stale) Mirror 18: 0xc3d10b6a (stale) Mirror 19: 0xc3d10b6a (stale) Mirror 20: 0xc3d10b6a (stale) Mirror 21: 0xc3d10b6a (stale) Mirror 23: 0xc3d10b6a (stale) Mirror 24: 0xc3d10b6a (stale) Mirror 25: 0xc3d10b6a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ddb4cb0 Mirror 26: 0x5ddb4cb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000948 1782218961.061195 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x124:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc3d10b6a (stale) Mirror 16: 0xc3d10b6a (stale) Mirror 18: 0xc3d10b6a (stale) Mirror 19: 0xc3d10b6a (stale) Mirror 20: 0xc3d10b6a (stale) Mirror 23: 0xc3d10b6a (stale) Mirror 24: 0xc3d10b6a (stale) Mirror 25: 0xc3d10b6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xc3d10b6a (stale) Mirror 16: 0xc3d10b6a (stale) Mirror 18: 0xc3d10b6a (stale) Mirror 19: 0xc3d10b6a (stale) Mirror 20: 0xc3d10b6a (stale) Mirror 23: 0xc3d10b6a (stale) Mirror 24: 0xc3d10b6a (stale) Mirror 25: 0xc3d10b6a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ddb4cb0 Mirror 26: 0x5ddb4cb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000949 1782218963.724811 read 0x03e2a6 thru 0x03ffff (0x1d5a bytes) 000950 1782218963.773614 mapread 0x036c3d thru 0x03eceb (0x80af bytes) 000951 1782218963.868199 mapread 0x01ac5b thru 0x026971 (0xbd17 bytes) 000953 1782218964.086757 trunc from 0x040000 to 0x0354c7 (0xab38 bytes) 000954 1782218964.324135 mapwrite 0x0062b0 thru 0x00eac8 (0x8819 bytes) 000955 1782218964.508110 mapwrite 0x0347d3 thru 0x03fe89 (0xb6b7 bytes) 000956 1782218964.744561 punch from 0x027ee1 to 0x034b89 (0xcca9 bytes) 000957 1782218964.772886 mapread 0x02cfe1 thru 0x033aac (0x6acc bytes) 000958 1782218964.846731 mapwrite 0x03d7f5 thru 0x03ffff (0x280b bytes) 000960 1782218964.943301 punch from 0x003879 to 0x00859c (0x4d24 bytes) 000961 1782218964.962499 mapwrite 0x020694 thru 0x026666 (0x5fd3 bytes) 000962 1782218965.062213 mapread 0x024bbe thru 0x028c10 (0x4053 bytes) 000964 1782218965.075250 mapread 0x03524c thru 0x036707 (0x14bc bytes) 000965 1782218965.077392 punch from 0x02d925 to 0x0340ad (0x6789 bytes) 000966 1782218965.096199 punch from 0x03b535 to 0x03ffff (0x4acb bytes) 000967 1782218965.106600 trunc from 0x03ef08 to 0x03ffff (0x10f8 bytes) 000968 1782218965.129262 trunc from 0x040000 to 0x02f142 (0x10ebd bytes) 000969 1782218965.157004 punch from 0x029be3 to 0x02e107 (0x4525 bytes) 000970 1782218965.168886 mapwrite 0x0284e4 thru 0x02d43b (0x4f58 bytes) 000971 1782218965.229022 write 0x02e4c8 thru 0x037758 (0x9291 bytes) 000973 1782218965.231861 read 0x02cdfd thru 0x034460 (0x7664 bytes) 000974 1782218965.232722 write 0x02b739 thru 0x02d062 (0x192a bytes) 000975 1782218965.234353 trunc from 0x037759 to 0x0003f6 (0x37362 bytes) 000976 1782218965.255901 trunc from 0x0001e8 to 0x0003f6 (0x20f bytes) 000977 1782218965.263375 mapwrite 0x03d7b2 thru 0x03f375 (0x1bc4 bytes) 000978 1782218965.330103 trunc from 0x03f376 to 0x00a27d (0x350f8 bytes) 000979 1782218965.383298 fallocate 0x03275b thru 0x0335ff (0xea5 bytes) 000980 1782218965.450399 MIRROR_RESYNC 10 mirrors 000981 1782218967.500724 fallocate 0x02227d thru 0x0251c0 (0x2f44 bytes) 000982 1782218967.556791 mapwrite 0x024a13 thru 0x028729 (0x3d17 bytes) 000983 1782218967.595156 trunc from 0x0321a5 to 0x0335ff (0x145b bytes) 000984 1782218967.600377 read 0x01afc1 thru 0x0292c6 (0xe306 bytes) 000985 1782218967.612992 MIRROR_RESYNC 10 mirrors 000986 1782218971.062886 mapwrite 0x024084 thru 0x0307a7 (0xc724 bytes) 000987 1782218971.365445 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33600): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33600): Mirror 15: 0x294b1c47 (stale) Mirror 16: 0x294b1c47 (stale) Mirror 18: 0x294b1c47 (stale) Mirror 19: 0x294b1c47 (stale) Mirror 20: 0x294b1c47 (stale) Mirror 23: 0x294b1c47 (stale) Mirror 24: 0x294b1c47 (stale) Mirror 25: 0x294b1c47 (stale) Mirror 26: 0x294b1c47 (stale) CRC-32 checksum value for chunk [0, 0x33600): Mirror 1: 0x196a0810 Mirror 27: 0x196a0810 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33600: skipped 000988 1782218973.668370 punch from 0x00c6d8 to 0x013d19 (0x7642 bytes) 000989 1782218973.746636 mapread 0x02d03e thru 0x02f1c4 (0x2187 bytes) 000990 1782218973.788328 punch from 0x0032ee to 0x0040f2 (0xe05 bytes) 000991 1782218973.813954 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x33600): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33600): Mirror 15: 0x294b1c47 (stale) Mirror 16: 0x294b1c47 (stale) Mirror 18: 0x294b1c47 (stale) Mirror 19: 0x294b1c47 (stale) Mirror 20: 0x294b1c47 (stale) Mirror 23: 0x294b1c47 (stale) Mirror 24: 0x294b1c47 (stale) Mirror 25: 0x294b1c47 (stale) Mirror 26: 0x294b1c47 (stale) Mirror 27: 0x196a0810 (stale) CRC-32 checksum value for chunk [0, 0x33600): Mirror 1: 0x196a0810 Mirror 28: 0x196a0810 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33600: skipped 000992 1782218975.533842 mapread 0x006a21 thru 0x009bc7 (0x31a7 bytes) 000993 1782218975.584551 trunc from 0x01f3d4 to 0x027558 (0x8185 bytes) 000994 1782218975.654373 write 0x03267f thru 0x03d250 (0xabd2 bytes) 000995 1782218975.674982 mapwrite 0x002dad thru 0x00347a (0x6ce bytes) 000996 1782218975.725955 write 0x01f38f thru 0x0293b9 (0xa02b bytes) 000997 1782218975.747715 fallocate 0x031cbc thru 0x034364 (0x26a9 bytes) 000998 1782218975.760054 write 0x01963f thru 0x0245d0 (0xaf92 bytes) 000999 1782218975.770970 write 0x00e61e thru 0x013784 (0x5167 bytes) 001000 1782218975.787299 write 0x000a19 thru 0x003faf (0x3597 bytes) All operations completed A-OK! PASS 70a (214s) == sanity-flr test 71: check mirror extend parent fid ==== 08:49:44 (1782218984) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x52:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (15s) == sanity-flr test 72: check append on FLR file ========== 08:49:59 (1782218999) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.492752 s, 4.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264492 s, 4.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (14s) == sanity-flr test 200a: stress test ===================== 08:50:13 (1782219013) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7505344 .. Extending file size to 1050848 .. Extending file size to 5629184 .. Extending file size to 8894752 .. Extending file size to 7909024 .. Extending file size to 7948672 .. Extending file size to 8135456 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8526080 .. Waiting 61056 61057 61058 61059 61062 done 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (94s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:51:47 (1782219107) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4284192 .. Extending file size to 1114784 .. Extending file size to 6951840 .. Extending file size to 7890560 .. Extending file size to 8946400 .. Extending file size to 3322560 .. Extending file size to 6817184 .. Extending file size to 7893024 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8994592 .. Extending file size to 9125824 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62420 62421 62424 62426 62429 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) 192.168.201.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:53:13 (1782219193) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2212 1269092 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24444 3582576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18768 3588252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43212 7170828 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2212 1269092 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2208 1269096 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24444 3582576 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18768 3588252 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43212 7170828 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19242 s, 10.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.327084 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x79bd6f42 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 200c (27s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:53:42 (1782219222) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20613 s, 10.2 MB/s PASS 202 (12s) == sanity-flr test 203: mirror file preserve mirror ID === 08:53:54 (1782219234) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:54:06 (1782219246) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186386 s, 11.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:54:19 (1782219259) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131483 s, 15.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 204b (15s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:54:34 (1782219274) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.317265 s, 6.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204c (41s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:55:15 (1782219315) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193234 s, 5.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ad:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:56:17 (1782219377) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183781 s, 11.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:57:17 (1782219437) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg114-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg114-server: Pool lustre.test_204f created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17185 s, 12.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg114-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:58:44 (1782219524) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:58:54 (1782219534) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:59:04 (1782219544) oleg114-server: Pool lustre.test_206 created Adding targets to pool oleg114-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg114-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg114-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg114-server: Pool lustre.test_206 destroyed PASS 206 (36s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:59:41 (1782219581) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051997 s, 20.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107373 s, 9.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbcdc7194 Mirror 3: 0xbcdc7194 PASS 207 (20s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:00:01 (1782219601) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (9s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:00:09 (1782219609) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (7s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:00:16 (1782219616) Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0826433 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x20a0000 Read this (4 bytes): BBBB 192.168.201.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg114-client.virtnet /mnt/lustre2 (opts:) PASS 209a (51s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:01:07 (1782219667) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.02648 s, 309 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:01:17 (1782219677) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625327 s, 16.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-23 09:01:19.000000000 -0400 Modify: 2026-06-23 09:01:19.000000000 -0400 Change: 2026-06-23 09:01:21.000000000 -0400 Birth: 2026-06-23 09:01:18.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:01:28 (1782219688) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691417 s, 15.2 MB/s oleg114-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg114-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg114-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (29s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:01:57 (1782219717) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.293752 s, 139 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.038895 s, 105 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 23 09:02 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 23 09:02 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:02:08 (1782219728) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:02:20 (1782219740) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 2576 sec =========== 09:02:33 (1782219753) === sanity-flr: start cleanup 09:02:37 (1782219757) === === sanity-flr: finish cleanup 09:02:50 (1782219770) ===