-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 02:36:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg604-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 02:37:02 (1786689422) === oleg604-client.virtnet: executing check_config_client /mnt/lustre oleg604-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg604-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff922483390800.idle_timeout=debug osc.lustre-OST0001-osc-ffff922483390800.idle_timeout=debug disable quota as required oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 02:37:42 (1786689462) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8219] == sanity-flr test 0a: lfs mirror create with -N option == 02:37:44 (1786689464) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 02:38:01 (1786689481) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 02:38:04 (1786689484) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 02:38:07 (1786689487) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 02:38:29 (1786689509) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 02:38:32 (1786689512) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 02:38:36 (1786689516) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff922483390800.stats=clear osc.lustre-OST0001-osc-ffff922483390800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 02:38:43 (1786689523) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 02:38:55 (1786689535) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 02:39:02 (1786689542) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 02:39:05 (1786689545) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 02:39:12 (1786689552) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 02:39:21 (1786689561) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 02:39:28 (1786689568) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 02:39:35 (1786689575) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 02:39:41 (1786689581) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:39:50 (1786689590) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:39:57 (1786689597) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.405034 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855084 s, 12.3 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:40:03 (1786689603) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.195375 s, 59.0 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:40:10 (1786689610) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:40:19 (1786689619) Stopping /mnt/lustre-ost1 (opts:) on oleg604-server oleg604-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg604-server: oleg604-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg604-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg604-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg604-server oleg604-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg604-server: oleg604-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg604-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (123s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:42:23 (1786689743) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.244569 s, 17.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg604-server oleg604-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg604-server: oleg604-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg604-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:43:10 (1786689790) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg604-server oleg604-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg604-server: oleg604-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg604-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg604-server oleg604-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg604-server: oleg604-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg604-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:44:29 (1786689869) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.64798 s, 19.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 02:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.894s user 0m0.001s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg604-server oleg604-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff922483390800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.994s user 0m0.000s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg604-server: oleg604-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg604-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg604-server oleg604-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.923s user 0m0.001s sys 0m0.177s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg604-server: oleg604-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg604-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg604-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff922483390800.ost_server_uuid in FULL state after 0 sec PASS 33b (83s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:45:52 (1786689952) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:45:56 (1786689956) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:45:59 (1786689959) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 02:46:03 (1786689963) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 02:46:09 (1786689969) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.44187 s, 28.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:46:20 (1786689980) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286927 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495127 s, 21.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:46:27 (1786689987) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:46:35 (1786689995) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00512387 s, 200 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00150303 s, 681 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00202986 s, 504 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0064836 s, 158 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 02:46:45 (1786690005) create test files with size 15739851 .. aa18e0194aced9467b1673fa9c3d470e - 3115dae811f380664530b5dc99680d0c - dc92162ad1c0607b9dbba921a5e38097 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg604-client.virtnet /mnt/lustre (opts:) Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 02:47:07 (1786690027) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:47:20 (1786690040) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:47:27 (1786690047) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112542 s, 9.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xa6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429967 s, 24.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xa7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xa8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 02:47:35 (1786690055) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0980628 s, 21.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.388741 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203815 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 02:47:50 (1786690070) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:47:53 (1786690073) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:47:56 (1786690076) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325332 s, 32.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 02:48:05 (1786690085) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.253644 s, 12.4 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:48:14 (1786690094) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:48:24 (1786690104) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.362782 s, 28.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg604-client.virtnet /mnt/lustre (opts:) Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:48:35 (1786690115) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff922498fcd000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.374774 s, 28.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg604-client.virtnet /mnt/lustre (opts:) Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff922498bc4000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:48:55 (1786690135) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:49:30 (1786690170) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:50:03 (1786690203) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 02:50:06 (1786690206) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.248158 s, 12.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14082 s, 22.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8f391801 (nosync,stale) Mirror 2: 0x8f391801 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5ebb2b8af3b04246ab28455c38d6edee - mirror 1 checksum: b714fd163e9d1df852d491209eecb284 - mirror 2 checksum: 5ebb2b8af3b04246ab28455c38d6edee - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T02:50:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T02:50:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5ebb2b8af3b04246ab28455c38d6edee - mirror 1 checksum: b714fd163e9d1df852d491209eecb284 - mirror 2 checksum: b714fd163e9d1df852d491209eecb284 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T02:50:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T02:50:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdf:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 02:50:14 (1786690214) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402963 s, 26.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:50:21 (1786690221) Failing mds1 on oleg604-server Stopping /mnt/lustre-mds1 (opts:) on oleg604-server 02:50:32 (1786690232) shut down facet: mds1 facet_host: oleg604-server facet_failover_host: oleg604-server Failover mds1 to oleg604-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg604-server: oleg604-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-MDT0000 02:50:54 (1786690254) targets are mounted 02:50:54 (1786690254) facet_failover done oleg604-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394318 s, 26.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe1:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:51:09 (1786690269) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0606835 s, 337 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0295827 s, 692 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:51:17 (1786690277) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0594708 s, 344 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0266273 s, 769 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:51:27 (1786690287) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0102747 s, 1.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:51:35 (1786690295) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0346675 s, 591 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:51:43 (1786690303) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00919512 s, 445 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.02821 s, 145 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:51:52 (1786690312) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:52:16 (1786690336) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:52:44 (1786690364) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:53:07 (1786690387) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:53:30 (1786690410) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 02:53:37 (1786690417) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57540 mirror split pid 57541 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57540 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57541 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 02:54:44 (1786690484) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 487 000001 1786690487.300515 write 0x0157d5 thru 0x022474 (0xcca0 bytes) truncating to largest ever: 0xd4c1 000002 1786690487.310675 trunc from 0x022475 to 0x00d4c0 (0x14fb4 bytes) 000003 1786690487.328897 trunc from 0x003444 to 0x008326 (0x4ee3 bytes) 000004 1786690487.333173 punch from 0x00b9a0 to 0x00d4c0 (0x1b21 bytes) 000005 1786690487.337970 read 0x00659e thru 0x00c7d9 (0x623c bytes) truncating to largest ever: 0x34708 000006 1786690487.347952 trunc from 0x00d4c1 to 0x034707 (0x27247 bytes) 000007 1786690487.367650 read 0x02335b thru 0x025889 (0x252f bytes) 000008 1786690487.392004 mapread 0x03101a thru 0x034707 (0x36ee bytes) 000009 1786690487.393133 mapwrite 0x023b22 thru 0x030e32 (0xd311 bytes) 000010 1786690487.432373 write 0x00930e thru 0x010ed1 (0x7bc4 bytes) 000013 1786690487.437670 punch from 0x0269ad to 0x029140 (0x2794 bytes) 000014 1786690487.443163 mapread 0x01d7b8 thru 0x02c1c0 (0xea09 bytes) 000015 1786690487.466804 punch from 0x0169e2 to 0x020efe (0xa51d bytes) 000016 1786690487.472768 read 0x033126 thru 0x034707 (0x15e2 bytes) truncating to largest ever: 0x39d7b 000019 1786690487.476195 trunc from 0x034708 to 0x039d7a (0x5673 bytes) 000020 1786690487.506932 fallocate 0x02a3a1 thru 0x032b53 (0x87b3 bytes) 000021 1786690487.520418 MIRROR_EXTEND to 2 mirrors 000022 1786690488.796065 mapwrite 0x00c3e2 thru 0x01a243 (0xde62 bytes) 000023 1786690488.949204 punch from 0x0388c2 to 0x039d7a (0x14b9 bytes) 000024 1786690488.953578 mapread 0x031b2d thru 0x034a1f (0x2ef3 bytes) 000026 1786690488.963672 write 0x02f001 thru 0x038e89 (0x9e89 bytes) 000027 1786690488.969688 trunc from 0x039d7b to 0x0258e3 (0x14497 bytes) 000028 1786690488.997683 fallocate 0x001bf2 thru 0x005d4f (0x415e bytes) 000030 1786690489.005092 read 0x00088e thru 0x0050f2 (0x4865 bytes) 000031 1786690489.015188 trunc from 0x00f6ec to 0x01a045 (0xa95a bytes) 000033 1786690489.021859 fallocate 0x0062f2 thru 0x009573 (0x3282 bytes) 000034 1786690489.026426 trunc from 0x0258e4 to 0x031c4f (0xc36c bytes) 000035 1786690489.047310 punch from 0x02efe6 to 0x031c4f (0x2c6a bytes) 000036 1786690489.052671 write 0x014322 thru 0x01b424 (0x7103 bytes) 000037 1786690489.059431 trunc from 0x031c50 to 0x0143da (0x1d875 bytes) 000038 1786690489.081328 punch from 0x007a1a to 0x0143da (0xc9c1 bytes) 000039 1786690489.104385 MIRROR_RESYNC 2 mirrors 000040 1786690489.939551 trunc from 0x0042a5 to 0x00701a (0x2d76 bytes) 000041 1786690489.996570 fallocate 0x00188e thru 0x00e5e3 (0xcd56 bytes) 000042 1786690490.005195 write 0x01efa8 thru 0x021390 (0x23e9 bytes) 000043 1786690490.010299 mapwrite 0x00abbc thru 0x015e57 (0xb29c bytes) fallocating to largest ever: 0x40000 000044 1786690490.148423 fallocate 0x03893b thru 0x03ffff (0x76c5 bytes) 000045 1786690490.160786 write 0x0085b7 thru 0x00e8bb (0x6305 bytes) 000046 1786690490.168808 mapread 0x014844 thru 0x01681d (0x1fda bytes) 000047 1786690490.173977 trunc from 0x0063c3 to 0x00b1dc (0x4e1a bytes) 000048 1786690490.196966 fallocate 0x031677 thru 0x03582f (0x41b9 bytes) 000049 1786690490.215798 write 0x012d8d thru 0x0195aa (0x681e bytes) 000050 1786690490.219175 punch from 0x020ea4 to 0x02b553 (0xa6b0 bytes) 000051 1786690490.229626 trunc from 0x000b2b to 0x005cd8 (0x51ae bytes) 000052 1786690490.246761 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35830): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35830): Mirror 2: 0x1c9f0d63 (stale) CRC-32 checksum value for chunk [0, 0x35830): Mirror 1: 0x6b9e2ac6 Mirror 3: 0x6b9e2ac6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35830: skipped 000053 1786690491.253867 fallocate 0x03ca58 thru 0x03e197 (0x1740 bytes) 000054 1786690491.311331 fallocate 0x00f0a3 thru 0x0183df (0x933d bytes) 000056 1786690491.321331 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35830): Mirror 2: 0x1c9f0d63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35830): Mirror 2: 0x1c9f0d63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35830: skipped 000057 1786690491.642464 write 0x011e7c thru 0x01b385 (0x950a bytes) 000058 1786690491.660069 write 0x005562 thru 0x00f028 (0x9ac7 bytes) 000061 1786690491.680589 mapwrite 0x019737 thru 0x021acf (0x8399 bytes) 000062 1786690491.732095 fallocate 0x02a9ec thru 0x0309d1 (0x5fe6 bytes) 000063 1786690491.743929 read 0x01925d thru 0x028371 (0xf115 bytes) 000064 1786690491.768444 read 0x00788a thru 0x00ccb9 (0x5430 bytes) 000065 1786690491.769173 read 0x030464 thru 0x03582f (0x53cc bytes) 000066 1786690491.791383 mapwrite 0x035e29 thru 0x0380fe (0x22d6 bytes) 000067 1786690491.877067 read 0x02d81f thru 0x0380fe (0xa8e0 bytes) 000068 1786690491.878539 read 0x037ea3 thru 0x0380fe (0x25c bytes) 000069 1786690491.880022 trunc from 0x0380ff to 0x031e7c (0x6282 bytes) 000071 1786690491.897156 fallocate 0x01f2f6 thru 0x02edfc (0xfb07 bytes) 000072 1786690491.905179 write 0x000ce0 thru 0x0025fa (0x191b bytes) 000073 1786690491.907506 trunc from 0x031e7d to 0x03fa0a (0xdb8e bytes) 000074 1786690491.938178 trunc from 0x03aa65 to 0x03b951 (0xeed bytes) 000075 1786690491.950508 punch from 0x00b5fb to 0x017dd5 (0xc7db bytes) 000076 1786690491.956114 trunc from 0x030a0e to 0x03ea2e (0xe021 bytes) 000078 1786690491.967144 fallocate 0x02ae2d thru 0x03179d (0x6971 bytes) 000079 1786690491.976818 trunc from 0x03fa0b to 0x01d621 (0x223e9 bytes) 000080 1786690492.012126 mapwrite 0x037e4a thru 0x03ffff (0x81b6 bytes) 000081 1786690492.058847 trunc from 0x004538 to 0x00f59e (0xb067 bytes) 000082 1786690492.072074 MIRROR_RESYNC 2 mirrors 000083 1786690492.808529 trunc from 0x040000 to 0x002925 (0x3d6da bytes) 000084 1786690492.862957 mapwrite 0x01fbf0 thru 0x02ba69 (0xbe7a bytes) 000085 1786690492.986801 mapread 0x0057e6 thru 0x011a2f (0xc24a bytes) 000087 1786690493.021150 trunc from 0x02ba6a to 0x02b87b (0x1ee bytes) 000088 1786690493.036450 read 0x017c10 thru 0x019712 (0x1b03 bytes) 000089 1786690493.037207 read 0x0275c8 thru 0x02b87b (0x42b4 bytes) 000092 1786690493.041143 trunc from 0x02194b to 0x02b87b (0x9f31 bytes) 000093 1786690493.055873 MIRROR_RESYNC 2 mirrors 000094 1786690493.640606 write 0x031ebe thru 0x03ec0e (0xcd51 bytes) 000095 1786690493.678630 write 0x01d3c8 thru 0x028505 (0xb13e bytes) 000096 1786690493.689747 punch from 0x02db79 to 0x0321d2 (0x465a bytes) 000097 1786690493.717267 fallocate 0x00f093 thru 0x010636 (0x15a4 bytes) 000098 1786690493.724810 fallocate 0x01b253 thru 0x024a92 (0x9840 bytes) 000099 1786690493.736680 punch from 0x02fa54 to 0x03bdd4 (0xc381 bytes) 000100 1786690493.752735 read 0x013ca2 thru 0x019e8a (0x61e9 bytes) 000101 1786690493.765694 write 0x011c0b thru 0x014b01 (0x2ef7 bytes) 000102 1786690493.779783 trunc from 0x03ec0f to 0x017a05 (0x27209 bytes) 000104 1786690493.832597 trunc from 0x017a06 to 0x025434 (0xda2f bytes) 000106 1786690493.867125 fallocate 0x01ae4b thru 0x0207b1 (0x5967 bytes) 000107 1786690493.880018 mapread 0x01dda5 thru 0x025434 (0x7690 bytes) 000108 1786690493.926176 trunc from 0x025435 to 0x01b8b5 (0x9b7f bytes) 000109 1786690493.974799 trunc from 0x01b8b6 to 0x02e212 (0x1295d bytes) 000110 1786690493.992913 punch from 0x00604e to 0x00fc35 (0x9be8 bytes) 000111 1786690494.005220 punch from 0x01ee6a to 0x01f64b (0x7e2 bytes) 000112 1786690494.021782 mapwrite 0x0141ed thru 0x01ab7e (0x6992 bytes) 000113 1786690494.067828 mapwrite 0x03bca5 thru 0x03ffff (0x435b bytes) 000114 1786690494.138909 read 0x018d5f thru 0x022266 (0x9508 bytes) 000115 1786690494.162716 mapread 0x02665b thru 0x030c92 (0xa638 bytes) 000116 1786690494.166753 trunc from 0x000115 to 0x0035c1 (0x34ad bytes) 000117 1786690494.211772 MIRROR_RESYNC 2 mirrors 000118 1786690495.068003 fallocate 0x023366 thru 0x02d42d (0xa0c8 bytes) 000119 1786690495.114117 punch from 0x029bd5 to 0x02d93a (0x3d66 bytes) 000120 1786690495.119684 fallocate 0x02b791 thru 0x03224c (0x6abc bytes) 000121 1786690495.124585 mapwrite 0x029ea7 thru 0x02da66 (0x3bc0 bytes) 000122 1786690495.175820 fallocate 0x034c57 thru 0x03ef87 (0xa331 bytes) 000124 1786690495.182750 trunc from 0x01e742 to 0x024323 (0x5be2 bytes) 000125 1786690495.188020 trunc from 0x040000 to 0x027f80 (0x1807f bytes) 000126 1786690495.269606 MIRROR_RESYNC 2 mirrors 000127 1786690495.911419 punch from 0x00525a to 0x01008e (0xae35 bytes) 000128 1786690495.958480 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27f81): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f81): Mirror 2: 0x523d22d6 (stale) CRC-32 checksum value for chunk [0, 0x27f81): Mirror 1: 0x523d22d6 Mirror 3: 0x523d22d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f81: skipped 000129 1786690497.162462 read 0x0115d5 thru 0x0147a8 (0x31d4 bytes) 000130 1786690497.175763 trunc from 0x007d1a to 0x00945f (0x1746 bytes) 000131 1786690497.242948 write 0x024ced thru 0x02a63d (0x5951 bytes) 000132 1786690497.287390 trunc from 0x024727 to 0x02a63d (0x5f17 bytes) 000133 1786690497.329047 trunc from 0x02a63e to 0x013aad (0x16b90 bytes) 000134 1786690497.383475 read 0x00c9fd thru 0x013aad (0x70b1 bytes) 000135 1786690497.409235 mapread 0x0029ba thru 0x00a369 (0x79b0 bytes) 000136 1786690497.414432 trunc from 0x001eb4 to 0x00392e (0x1a7b bytes) 000138 1786690497.431286 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x13aae): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13aae): Mirror 2: 0x1bbf06e1 (stale) Mirror 3: 0x1bbf06e1 (stale) CRC-32 checksum value for chunk [0, 0x13aae): Mirror 1: 0x1bbf06e1 Mirror 4: 0x1bbf06e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13aae: skipped 000140 1786690498.410169 read 0x00befc thru 0x00f9d9 (0x3ade bytes) 000141 1786690498.469786 MIRROR_RESYNC 4 mirrors 000142 1786690499.642270 fallocate 0x03fd10 thru 0x03ffff (0x2f0 bytes) 000143 1786690499.709417 punch from 0x018809 to 0x021a77 (0x926f bytes) 000144 1786690499.714891 write 0x01fe66 thru 0x022881 (0x2a1c bytes) 000145 1786690499.726022 fallocate 0x012526 thru 0x022257 (0xfd32 bytes) 000147 1786690499.732463 mapread 0x0156db thru 0x02417f (0xeaa5 bytes) 000148 1786690499.823393 fallocate 0x02da2f thru 0x02ea26 (0xff8 bytes) 000149 1786690499.829610 read 0x02a153 thru 0x034153 (0xa001 bytes) 000150 1786690499.855851 write 0x01d15c thru 0x023628 (0x64cd bytes) 000151 1786690499.875158 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x779e1dd3 (stale) Mirror 3: 0x779e1dd3 (stale) Mirror 4: 0x779e1dd3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe06ff550 Mirror 5: 0xe06ff550 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000152 1786690500.853995 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x779e1dd3 (stale) Mirror 3: 0x779e1dd3 (stale) Mirror 4: 0x779e1dd3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe06ff550 Mirror 5: 0xe06ff550 Mirror 6: 0xe06ff550 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000153 1786690502.097923 write 0x00961d thru 0x0187dd (0xf1c1 bytes) 000154 1786690502.159479 punch from 0x007b57 to 0x0154ad (0xd957 bytes) 000155 1786690502.178876 trunc from 0x040000 to 0x015ce3 (0x2a31c bytes) 000156 1786690502.209949 trunc from 0x015ce4 to 0x00233f (0x139a4 bytes) 000157 1786690502.246018 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2340): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2340): Mirror 2: 0x8f986756 (stale) Mirror 3: 0x8f986756 (stale) Mirror 4: 0x8f986756 (stale) Mirror 5: 0x8f986756 (stale) Mirror 6: 0x8f986756 (stale) CRC-32 checksum value for chunk [0, 0x2340): Mirror 1: 0x8f986756 Mirror 7: 0x8f986756 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2340: skipped 000158 1786690503.191055 mapread 0x000d20 thru 0x00233f (0x1620 bytes) 000159 1786690503.241544 read 0x00190c thru 0x00233f (0xa34 bytes) 000161 1786690503.277130 fallocate 0x0193ef thru 0x01b75e (0x2370 bytes) 000163 1786690503.344567 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1b75f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b75f): Mirror 2: 0xce7ddde5 (stale) Mirror 3: 0xce7ddde5 (stale) Mirror 4: 0xce7ddde5 (stale) Mirror 5: 0xce7ddde5 (stale) Mirror 6: 0xce7ddde5 (stale) Mirror 7: 0xe92709da (stale) CRC-32 checksum value for chunk [0, 0x1b75f): Mirror 1: 0xe92709da Mirror 8: 0xe92709da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b75f: skipped 000164 1786690504.971398 trunc from 0x0167ce to 0x01b75e (0x4f91 bytes) 000165 1786690505.017250 fallocate 0x03fca6 thru 0x03ffff (0x35a bytes) 000166 1786690505.024624 trunc from 0x01b75f to 0x01b5e9 (0x175 bytes) 000167 1786690505.063914 read 0x00998b thru 0x00fc49 (0x62bf bytes) 000168 1786690505.077068 punch from 0x0108a6 to 0x01b5e9 (0xad44 bytes) 000170 1786690505.081522 trunc from 0x01b5ea to 0x012967 (0x8c82 bytes) 000171 1786690505.108681 trunc from 0x010fb8 to 0x012967 (0x19b0 bytes) 000172 1786690505.113741 read 0x00bf0c thru 0x012967 (0x6a5c bytes) 000173 1786690505.133203 trunc from 0x012968 to 0x00a474 (0x84f3 bytes) 000174 1786690505.153930 mapread 0x003cb2 thru 0x00a474 (0x67c3 bytes) 000175 1786690505.155177 mapwrite 0x00ee99 thru 0x0134d1 (0x4639 bytes) 000176 1786690505.292517 MIRROR_RESYNC 8 mirrors 000177 1786690506.946132 trunc from 0x0134d2 to 0x030c09 (0x1d738 bytes) 000178 1786690507.033730 fallocate 0x022a4d thru 0x02a099 (0x764d bytes) 000179 1786690507.051255 mapread 0x017f21 thru 0x01b56a (0x364a bytes) 000180 1786690507.079804 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x30c0a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30c0a): Mirror 2: 0x8cc0f4cb (stale) Mirror 3: 0x8cc0f4cb (stale) Mirror 4: 0x8cc0f4cb (stale) Mirror 5: 0x8cc0f4cb (stale) Mirror 6: 0x8cc0f4cb (stale) Mirror 7: 0x8cc0f4cb (stale) Mirror 8: 0x8cc0f4cb (stale) CRC-32 checksum value for chunk [0, 0x30c0a): Mirror 1: 0x8cc0f4cb Mirror 9: 0x8cc0f4cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30c0a: skipped 000181 1786690508.314688 MIRROR_RESYNC 9 mirrors 000182 1786690510.646140 mapread 0x016cab thru 0x01cad7 (0x5e2d bytes) 000183 1786690510.677682 mapwrite 0x00f49e thru 0x01092d (0x1490 bytes) 000184 1786690510.800203 write 0x03ff71 thru 0x03ffff (0x8f bytes) 000185 1786690510.805953 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xea5c741f (stale) Mirror 3: 0xea5c741f (stale) Mirror 4: 0xea5c741f (stale) Mirror 5: 0xea5c741f (stale) Mirror 6: 0xea5c741f (stale) Mirror 7: 0xea5c741f (stale) Mirror 8: 0xea5c741f (stale) Mirror 9: 0xea5c741f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x936ba3d4 Mirror 10: 0x936ba3d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000186 1786690512.053482 write 0x033365 thru 0x03fc26 (0xc8c2 bytes) 000187 1786690512.121059 punch from 0x02ef73 to 0x030b29 (0x1bb7 bytes) 000188 1786690512.135129 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xea5c741f (stale) Mirror 4: 0xea5c741f (stale) Mirror 5: 0xea5c741f (stale) Mirror 6: 0xea5c741f (stale) Mirror 7: 0xea5c741f (stale) Mirror 8: 0xea5c741f (stale) Mirror 9: 0xea5c741f (stale) Mirror 10: 0x936ba3d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xea5c741f (stale) Mirror 4: 0xea5c741f (stale) Mirror 5: 0xea5c741f (stale) Mirror 6: 0xea5c741f (stale) Mirror 7: 0xea5c741f (stale) Mirror 8: 0xea5c741f (stale) Mirror 9: 0xea5c741f (stale) Mirror 10: 0x936ba3d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1786690512.795641 mapwrite 0x038f8e thru 0x03bfdb (0x304e bytes) 000190 1786690512.844229 MIRROR_RESYNC 9 mirrors 000191 1786690515.163856 read 0x0220eb thru 0x02a044 (0x7f5a bytes) 000193 1786690515.174022 punch from 0x02f133 to 0x03337f (0x424d bytes) 000194 1786690515.235579 MIRROR_RESYNC 9 mirrors 000196 1786690517.169056 read 0x00beeb thru 0x014874 (0x898a bytes) 000198 1786690517.179507 trunc from 0x020caa to 0x026c09 (0x5f60 bytes) 000199 1786690517.248929 write 0x03af28 thru 0x03ffff (0x50d8 bytes) 000200 1786690517.263824 write 0x0022ff thru 0x00732e (0x5030 bytes) 000201 1786690517.301280 write 0x03b15a thru 0x03ffff (0x4ea6 bytes) 000202 1786690517.302339 fallocate 0x0035a8 thru 0x00b027 (0x7a80 bytes) 000203 1786690517.324615 mapread 0x0299e3 thru 0x02a950 (0xf6e bytes) 000204 1786690517.346796 trunc from 0x040000 to 0x00f423 (0x30bdc bytes) 000205 1786690517.392782 trunc from 0x0052ad to 0x00614c (0xea0 bytes) 000207 1786690517.426928 trunc from 0x00f424 to 0x00fdaf (0x98c bytes) 000208 1786690517.454047 read 0x003329 thru 0x008c4b (0x5923 bytes) 000209 1786690517.465143 mapwrite 0x01446a thru 0x018ae6 (0x467d bytes) 000210 1786690517.570660 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x18ae7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ae7): Mirror 3: 0xb66edd7e (stale) Mirror 4: 0xb66edd7e (stale) Mirror 5: 0xb66edd7e (stale) Mirror 6: 0xb66edd7e (stale) Mirror 7: 0xb66edd7e (stale) Mirror 8: 0xb66edd7e (stale) Mirror 9: 0xb66edd7e (stale) Mirror 10: 0xb66edd7e (stale) CRC-32 checksum value for chunk [0, 0x18ae7): Mirror 1: 0x99adc588 Mirror 11: 0x99adc588 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ae7: skipped 000211 1786690518.640588 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x18ae7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18ae7): Mirror 3: 0xb66edd7e (stale) Mirror 4: 0xb66edd7e (stale) Mirror 5: 0xb66edd7e (stale) Mirror 6: 0xb66edd7e (stale) Mirror 7: 0xb66edd7e (stale) Mirror 8: 0xb66edd7e (stale) Mirror 9: 0xb66edd7e (stale) Mirror 10: 0xb66edd7e (stale) CRC-32 checksum value for chunk [0, 0x18ae7): Mirror 1: 0x99adc588 Mirror 11: 0x99adc588 Mirror 12: 0x99adc588 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18ae7: skipped 000212 1786690519.911161 MIRROR_RESYNC 11 mirrors 000213 1786690522.723754 read 0x013c9a thru 0x018ae6 (0x4e4d bytes) 000214 1786690522.738181 fallocate 0x000145 thru 0x005228 (0x50e4 bytes) 000215 1786690522.777136 write 0x0223dd thru 0x029375 (0x6f99 bytes) 000216 1786690522.779079 punch from 0x01de1a to 0x0204e7 (0x26ce bytes) 000217 1786690522.785297 punch from 0x01ff86 to 0x02026a (0x2e5 bytes) 000218 1786690522.793723 write 0x03e058 thru 0x03ffff (0x1fa8 bytes) 000219 1786690522.795256 fallocate 0x03f996 thru 0x03ffff (0x66a bytes) 000220 1786690522.800200 trunc from 0x040000 to 0x032345 (0xdcba bytes) 000222 1786690522.847307 trunc from 0x032346 to 0x0115a8 (0x20d9d bytes) 000223 1786690522.868690 punch from 0x00d220 to 0x0115a8 (0x4389 bytes) 000224 1786690522.874570 trunc from 0x00d973 to 0x010aea (0x3178 bytes) 000226 1786690522.882753 read 0x0026e7 thru 0x0115a8 (0xeec2 bytes) 000227 1786690522.902139 MIRROR_RESYNC 11 mirrors 000228 1786690525.023210 mapread 0x002f03 thru 0x00ce90 (0x9f8e bytes) 000229 1786690525.051726 write 0x011e4a thru 0x01e031 (0xc1e8 bytes) 000230 1786690525.088572 write 0x020110 thru 0x027c2a (0x7b1b bytes) 000231 1786690525.091230 mapread 0x02663c thru 0x027c2a (0x15ef bytes) 000232 1786690525.093577 fallocate 0x00a13e thru 0x00eaab (0x496e bytes) 000233 1786690525.101951 fallocate 0x000a70 thru 0x004c1d (0x41ae bytes) 000234 1786690525.108324 write 0x030446 thru 0x037b03 (0x76be bytes) 000235 1786690525.111716 fallocate 0x031579 thru 0x037014 (0x5a9c bytes) 000236 1786690525.118487 punch from 0x008077 to 0x011eab (0x9e35 bytes) 000238 1786690525.152717 trunc from 0x00a159 to 0x011b7b (0x7a23 bytes) 000239 1786690525.160834 fallocate 0x00bc29 thru 0x0148a8 (0x8c80 bytes) 000240 1786690525.175168 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x37b04): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b04): Mirror 3: 0x55404799 (stale) Mirror 4: 0x55404799 (stale) Mirror 5: 0x55404799 (stale) Mirror 6: 0x55404799 (stale) Mirror 7: 0x55404799 (stale) Mirror 8: 0x55404799 (stale) Mirror 9: 0x55404799 (stale) Mirror 10: 0x55404799 (stale) Mirror 11: 0x55404799 (stale) Mirror 12: 0x55404799 (stale) CRC-32 checksum value for chunk [0, 0x37b04): Mirror 1: 0xc1eba287 Mirror 13: 0xc1eba287 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b04: skipped 000241 1786690526.763916 write 0x00d4d9 thru 0x0119ec (0x4514 bytes) 000242 1786690526.844650 trunc from 0x037b04 to 0x029da3 (0xdd60 bytes) 000243 1786690526.908865 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x29da4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29da4): Mirror 3: 0x940f4527 (stale) Mirror 4: 0x940f4527 (stale) Mirror 5: 0x940f4527 (stale) Mirror 6: 0x940f4527 (stale) Mirror 7: 0x940f4527 (stale) Mirror 8: 0x940f4527 (stale) Mirror 9: 0x940f4527 (stale) Mirror 10: 0x940f4527 (stale) Mirror 11: 0x940f4527 (stale) Mirror 12: 0x940f4527 (stale) Mirror 13: 0xe2408ace (stale) CRC-32 checksum value for chunk [0, 0x29da4): Mirror 1: 0x3e896c95 Mirror 14: 0x3e896c95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29da4: skipped 000244 1786690528.495067 mapread 0x02228b thru 0x024eba (0x2c30 bytes) 000245 1786690528.537878 fallocate 0x007e2a thru 0x00b26b (0x3442 bytes) 000247 1786690528.608234 read 0x007ab4 thru 0x015487 (0xd9d4 bytes) 000248 1786690528.625610 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x29da4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29da4): Mirror 3: 0x940f4527 (stale) Mirror 4: 0x940f4527 (stale) Mirror 5: 0x940f4527 (stale) Mirror 6: 0x940f4527 (stale) Mirror 7: 0x940f4527 (stale) Mirror 8: 0x940f4527 (stale) Mirror 9: 0x940f4527 (stale) Mirror 10: 0x940f4527 (stale) Mirror 11: 0x940f4527 (stale) Mirror 12: 0x940f4527 (stale) Mirror 13: 0xe2408ace (stale) Mirror 14: 0x3e896c95 (stale) CRC-32 checksum value for chunk [0, 0x29da4): Mirror 1: 0x3e896c95 Mirror 15: 0x3e896c95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29da4: skipped 000249 1786690530.154593 mapwrite 0x039660 thru 0x03ded2 (0x4873 bytes) 000250 1786690530.262540 mapwrite 0x00ad63 thru 0x017d38 (0xcfd6 bytes) 000251 1786690530.310729 mapwrite 0x034b0d thru 0x03da2c (0x8f20 bytes) 000252 1786690530.329643 punch from 0x00796c to 0x0121a2 (0xa837 bytes) 000253 1786690530.335180 fallocate 0x01590a thru 0x019b24 (0x421b bytes) 000254 1786690530.347183 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ded3): Mirror 3: 0xc8938be9 (stale) Mirror 4: 0xc8938be9 (stale) Mirror 5: 0xc8938be9 (stale) Mirror 6: 0xc8938be9 (stale) Mirror 7: 0xc8938be9 (stale) Mirror 8: 0xc8938be9 (stale) Mirror 10: 0xc8938be9 (stale) Mirror 11: 0xc8938be9 (stale) Mirror 12: 0xc8938be9 (stale) Mirror 13: 0x1a536510 (stale) Mirror 14: 0xc72e540b (stale) Mirror 15: 0xc72e540b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ded3): Mirror 3: 0xc8938be9 (stale) Mirror 4: 0xc8938be9 (stale) Mirror 5: 0xc8938be9 (stale) Mirror 6: 0xc8938be9 (stale) Mirror 7: 0xc8938be9 (stale) Mirror 8: 0xc8938be9 (stale) Mirror 10: 0xc8938be9 (stale) Mirror 11: 0xc8938be9 (stale) Mirror 12: 0xc8938be9 (stale) Mirror 13: 0x1a536510 (stale) Mirror 14: 0xc72e540b (stale) Mirror 15: 0xc72e540b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ded3: skipped 000255 1786690531.135715 trunc from 0x03ded3 to 0x031828 (0xc6aa bytes) 000256 1786690531.166762 fallocate 0x02ee2c thru 0x03246a (0x363f bytes) 000257 1786690531.174758 mapwrite 0x030884 thru 0x03ac61 (0xa3de bytes) 000258 1786690531.244838 fallocate 0x03ada5 thru 0x03ffff (0x525b bytes) 000259 1786690531.257118 write 0x0006cc thru 0x00f78f (0xf0c4 bytes) 000260 1786690531.278064 mapread 0x013d41 thru 0x01b1b2 (0x7472 bytes) 000261 1786690531.304971 read 0x029a1a thru 0x035731 (0xbd18 bytes) 000262 1786690531.316419 mapread 0x014046 thru 0x016d47 (0x2d02 bytes) 000263 1786690531.317462 punch from 0x022e49 to 0x0291dd (0x6395 bytes) 000264 1786690531.347843 trunc from 0x040000 to 0x028df4 (0x1720b bytes) 000265 1786690531.380484 trunc from 0x00795f to 0x0139a8 (0xc04a bytes) 000266 1786690531.413982 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x28df5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28df5): Mirror 3: 0xdc22d388 (stale) Mirror 4: 0xdc22d388 (stale) Mirror 5: 0xdc22d388 (stale) Mirror 6: 0xdc22d388 (stale) Mirror 7: 0xdc22d388 (stale) Mirror 8: 0xdc22d388 (stale) Mirror 10: 0xdc22d388 (stale) Mirror 11: 0xdc22d388 (stale) Mirror 12: 0xdc22d388 (stale) Mirror 13: 0x7c84e16a (stale) Mirror 14: 0xcd011764 (stale) Mirror 15: 0xcd011764 (stale) CRC-32 checksum value for chunk [0, 0x28df5): Mirror 1: 0xa9d8fbb2 Mirror 16: 0xa9d8fbb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28df5: skipped 000267 1786690532.680088 trunc from 0x028df5 to 0x020a68 (0x838c bytes) 000268 1786690532.820211 mapread 0x003c49 thru 0x00b6ae (0x7a66 bytes) 000269 1786690532.856961 read 0x018292 thru 0x018496 (0x205 bytes) 000271 1786690532.862029 read 0x014c93 thru 0x020a68 (0xbdd6 bytes) 000273 1786690532.881667 trunc from 0x020a69 to 0x03a7fb (0x19d93 bytes) 000274 1786690532.940638 punch from 0x031a15 to 0x03a7fb (0x8de7 bytes) 000275 1786690532.949713 mapwrite 0x02cc8c thru 0x039589 (0xc8fe bytes) 000276 1786690533.023964 read 0x038f21 thru 0x03a7fb (0x18db bytes) 000277 1786690533.024711 trunc from 0x03a7fc to 0x02c1ec (0xe60f bytes) 000279 1786690533.053252 mapwrite 0x03f99c thru 0x03ffff (0x664 bytes) 000280 1786690533.120670 read 0x00fc98 thru 0x0116f5 (0x1a5e bytes) 000281 1786690533.122453 mapread 0x026782 thru 0x031c03 (0xb482 bytes) 000282 1786690533.124817 trunc from 0x040000 to 0x037ad4 (0x852b bytes) 000283 1786690533.146162 fallocate 0x00a844 thru 0x00c239 (0x19f6 bytes) 000285 1786690533.160333 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x37ad5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ad5): Mirror 3: 0x972ed768 (stale) Mirror 4: 0x972ed768 (stale) Mirror 5: 0x972ed768 (stale) Mirror 6: 0x972ed768 (stale) Mirror 7: 0x972ed768 (stale) Mirror 8: 0x972ed768 (stale) Mirror 10: 0x972ed768 (stale) Mirror 11: 0x972ed768 (stale) Mirror 12: 0x972ed768 (stale) Mirror 13: 0x98936556 (stale) Mirror 14: 0x1f3c1b15 (stale) Mirror 15: 0x1f3c1b15 (stale) Mirror 16: 0xf06a7012 (stale) CRC-32 checksum value for chunk [0, 0x37ad5): Mirror 1: 0x77585dbd Mirror 17: 0x77585dbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ad5: skipped 000286 1786690534.694321 trunc from 0x024ecc to 0x034793 (0xf8c8 bytes) 000287 1786690534.744690 trunc from 0x037ad5 to 0x03b0ba (0x35e6 bytes) 000288 1786690534.769913 trunc from 0x012027 to 0x0171b5 (0x518f bytes) 000289 1786690534.786158 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b0bb): Mirror 3: 0x282c0afc (stale) Mirror 4: 0x282c0afc (stale) Mirror 6: 0x282c0afc (stale) Mirror 7: 0x282c0afc (stale) Mirror 8: 0x282c0afc (stale) Mirror 10: 0x282c0afc (stale) Mirror 11: 0x282c0afc (stale) Mirror 12: 0x282c0afc (stale) Mirror 13: 0x3e62d9cf (stale) Mirror 14: 0x4309694c (stale) Mirror 15: 0x4309694c (stale) Mirror 16: 0x1b5fb25d (stale) Mirror 17: 0x1f37bdb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b0bb): Mirror 3: 0x282c0afc (stale) Mirror 4: 0x282c0afc (stale) Mirror 6: 0x282c0afc (stale) Mirror 7: 0x282c0afc (stale) Mirror 8: 0x282c0afc (stale) Mirror 10: 0x282c0afc (stale) Mirror 11: 0x282c0afc (stale) Mirror 12: 0x282c0afc (stale) Mirror 13: 0x3e62d9cf (stale) Mirror 14: 0x4309694c (stale) Mirror 15: 0x4309694c (stale) Mirror 16: 0x1b5fb25d (stale) Mirror 17: 0x1f37bdb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b0bb: skipped 000290 1786690535.542700 punch from 0x02bada to 0x039f4f (0xe476 bytes) 000291 1786690535.548781 read 0x02eebe thru 0x0391b0 (0xa2f3 bytes) 000292 1786690535.565514 MIRROR_RESYNC 14 mirrors 000293 1786690539.627058 read 0x01c0c2 thru 0x0208d0 (0x480f bytes) 000294 1786690539.643081 fallocate 0x0063ec thru 0x00db6f (0x7784 bytes) 000295 1786690539.698820 punch from 0x02489b to 0x026906 (0x206c bytes) 000296 1786690539.704536 trunc from 0x03b0bb to 0x0214e9 (0x19bd1 bytes) 000297 1786690539.738911 MIRROR_RESYNC 14 mirrors 000298 1786690543.276082 write 0x02f7c0 thru 0x034aae (0x52ef bytes) 000299 1786690543.343084 fallocate 0x0291cb thru 0x0313b6 (0x81ec bytes) 000300 1786690543.348740 fallocate 0x010c11 thru 0x01142f (0x81f bytes) 000301 1786690543.356021 write 0x0069d5 thru 0x00cee2 (0x650e bytes) 000302 1786690543.374524 trunc from 0x034aaf to 0x0301bb (0x48f3 bytes) 000303 1786690543.404226 mapwrite 0x037e57 thru 0x03cbbd (0x4d67 bytes) 000304 1786690543.474863 mapwrite 0x019237 thru 0x023a4c (0xa816 bytes) 000306 1786690543.534976 trunc from 0x027b17 to 0x035d53 (0xe23d bytes) 000307 1786690543.540783 trunc from 0x03cbbe to 0x014c73 (0x27f4a bytes) 000308 1786690543.556215 punch from 0x0088b7 to 0x00a68d (0x1dd7 bytes) 000309 1786690543.566051 mapread 0x002513 thru 0x011be2 (0xf6d0 bytes) 000310 1786690543.588919 trunc from 0x00aff8 to 0x00d44a (0x2453 bytes) 000311 1786690543.603900 trunc from 0x004a25 to 0x00a765 (0x5d41 bytes) 000312 1786690543.614411 read 0x007563 thru 0x010218 (0x8cb6 bytes) 000314 1786690543.640397 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14c74): Mirror 3: 0x1ba53718 (stale) Mirror 4: 0x1ba53718 (stale) Mirror 6: 0x1ba53718 (stale) Mirror 7: 0x1ba53718 (stale) Mirror 8: 0x1ba53718 (stale) Mirror 10: 0x1ba53718 (stale) Mirror 11: 0x1ba53718 (stale) Mirror 12: 0x1ba53718 (stale) Mirror 13: 0x1ba53718 (stale) Mirror 14: 0x1ba53718 (stale) Mirror 15: 0x1ba53718 (stale) Mirror 17: 0x1ba53718 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14c74): Mirror 3: 0x1ba53718 (stale) Mirror 4: 0x1ba53718 (stale) Mirror 6: 0x1ba53718 (stale) Mirror 7: 0x1ba53718 (stale) Mirror 8: 0x1ba53718 (stale) Mirror 10: 0x1ba53718 (stale) Mirror 11: 0x1ba53718 (stale) Mirror 12: 0x1ba53718 (stale) Mirror 13: 0x1ba53718 (stale) Mirror 14: 0x1ba53718 (stale) Mirror 15: 0x1ba53718 (stale) Mirror 17: 0x1ba53718 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14c74: skipped 000315 1786690544.358039 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14c74): Mirror 3: 0x1ba53718 (stale) Mirror 4: 0x1ba53718 (stale) Mirror 6: 0x1ba53718 (stale) Mirror 7: 0x1ba53718 (stale) Mirror 8: 0x1ba53718 (stale) Mirror 10: 0x1ba53718 (stale) Mirror 11: 0x1ba53718 (stale) Mirror 13: 0x1ba53718 (stale) Mirror 14: 0x1ba53718 (stale) Mirror 15: 0x1ba53718 (stale) Mirror 17: 0x1ba53718 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14c74): Mirror 3: 0x1ba53718 (stale) Mirror 4: 0x1ba53718 (stale) Mirror 6: 0x1ba53718 (stale) Mirror 7: 0x1ba53718 (stale) Mirror 8: 0x1ba53718 (stale) Mirror 10: 0x1ba53718 (stale) Mirror 11: 0x1ba53718 (stale) Mirror 13: 0x1ba53718 (stale) Mirror 14: 0x1ba53718 (stale) Mirror 15: 0x1ba53718 (stale) Mirror 17: 0x1ba53718 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14c74: skipped 000316 1786690544.937526 punch from 0x013e47 to 0x014c73 (0xe2d bytes) 000317 1786690544.942223 mapwrite 0x00700d thru 0x012e8a (0xbe7e bytes) 000318 1786690544.983515 trunc from 0x014c74 to 0x0239b3 (0xed40 bytes) 000319 1786690545.001936 fallocate 0x024d93 thru 0x02a10f (0x537d bytes) 000321 1786690545.012327 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x239b4): Mirror 3: 0x258616ee (stale) Mirror 4: 0x258616ee (stale) Mirror 6: 0x258616ee (stale) Mirror 7: 0x258616ee (stale) Mirror 8: 0x258616ee (stale) Mirror 10: 0x258616ee (stale) Mirror 11: 0x258616ee (stale) Mirror 13: 0x258616ee (stale) Mirror 15: 0x258616ee (stale) Mirror 17: 0x258616ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x239b4): Mirror 3: 0x258616ee (stale) Mirror 4: 0x258616ee (stale) Mirror 6: 0x258616ee (stale) Mirror 7: 0x258616ee (stale) Mirror 8: 0x258616ee (stale) Mirror 10: 0x258616ee (stale) Mirror 11: 0x258616ee (stale) Mirror 13: 0x258616ee (stale) Mirror 15: 0x258616ee (stale) Mirror 17: 0x258616ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x239b4: skipped 000322 1786690545.594019 fallocate 0x00c8c0 thru 0x01548d (0x8bce bytes) 000323 1786690545.605546 mapwrite 0x012a4e thru 0x013ae2 (0x1095 bytes) 000325 1786690545.639534 punch from 0x012b46 to 0x01953a (0x69f5 bytes) 000326 1786690545.645267 trunc from 0x0239b4 to 0x00d4dc (0x164d7 bytes) 000327 1786690545.667369 mapread 0x003bf6 thru 0x0074e9 (0x38f4 bytes) 000328 1786690545.678027 trunc from 0x00d4dd to 0x033700 (0x26224 bytes) 000329 1786690545.693483 trunc from 0x0233f9 to 0x02c44a (0x9052 bytes) 000331 1786690545.699680 punch from 0x01a517 to 0x0216ab (0x7195 bytes) 000332 1786690545.705142 trunc from 0x033701 to 0x03b2ba (0x7bba bytes) 000333 1786690545.749018 write 0x016029 thru 0x01a8ae (0x4886 bytes) 000334 1786690545.765706 punch from 0x038ca2 to 0x03b2ba (0x2619 bytes) 000335 1786690545.775956 mapwrite 0x004970 thru 0x011c5d (0xd2ee bytes) 000336 1786690545.834027 trunc from 0x03b2bb to 0x00634b (0x34f6f bytes) 000337 1786690545.864234 trunc from 0x00240c to 0x00634b (0x3f40 bytes) 000338 1786690545.870133 write 0x0092b5 thru 0x015234 (0xbf80 bytes) 000339 1786690545.872190 trunc from 0x009b63 to 0x015234 (0xb6d2 bytes) 000340 1786690545.897853 fallocate 0x003add thru 0x00e785 (0xaca9 bytes) 000341 1786690545.918823 trunc from 0x015235 to 0x008ffa (0xc23a bytes) 000342 1786690545.938053 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8ffb): Mirror 3: 0xf8985f50 (stale) Mirror 4: 0xf8985f50 (stale) Mirror 7: 0xf8985f50 (stale) Mirror 8: 0xf8985f50 (stale) Mirror 10: 0xf8985f50 (stale) Mirror 11: 0xf8985f50 (stale) Mirror 13: 0xf8985f50 (stale) Mirror 15: 0xf8985f50 (stale) Mirror 17: 0xf8985f50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8ffb): Mirror 3: 0xf8985f50 (stale) Mirror 4: 0xf8985f50 (stale) Mirror 7: 0xf8985f50 (stale) Mirror 8: 0xf8985f50 (stale) Mirror 10: 0xf8985f50 (stale) Mirror 11: 0xf8985f50 (stale) Mirror 13: 0xf8985f50 (stale) Mirror 15: 0xf8985f50 (stale) Mirror 17: 0xf8985f50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8ffb: skipped 000344 1786690546.556051 read 0x00517f thru 0x005976 (0x7f8 bytes) 000345 1786690546.561440 trunc from 0x008ffb to 0x03f173 (0x36179 bytes) 000346 1786690546.585164 mapwrite 0x028e87 thru 0x03630f (0xd489 bytes) 000347 1786690546.657063 mapwrite 0x033cb5 thru 0x03ae15 (0x7161 bytes) 000348 1786690546.679387 write 0x02a9d6 thru 0x032c62 (0x828d bytes) 000350 1786690546.682138 read 0x020bc6 thru 0x02b36b (0xa7a6 bytes) 000351 1786690546.695397 mapwrite 0x011655 thru 0x017b14 (0x64c0 bytes) 000352 1786690546.716803 trunc from 0x03f174 to 0x0309cc (0xe7a7 bytes) 000353 1786690546.741179 trunc from 0x0309cd to 0x034720 (0x3d54 bytes) 000354 1786690546.766638 trunc from 0x0154eb to 0x0224cf (0xcfe5 bytes) 000355 1786690546.775862 mapwrite 0x015d27 thru 0x01ab4c (0x4e26 bytes) 000356 1786690546.829482 fallocate 0x022af2 thru 0x024864 (0x1d73 bytes) 000357 1786690546.835790 trunc from 0x034721 to 0x0333ee (0x1332 bytes) 000359 1786690546.858740 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x333ef): Mirror 3: 0xfdf2e74f (stale) Mirror 4: 0xfdf2e74f (stale) Mirror 7: 0xfdf2e74f (stale) Mirror 8: 0xfdf2e74f (stale) Mirror 10: 0xfdf2e74f (stale) Mirror 11: 0xfdf2e74f (stale) Mirror 13: 0xfdf2e74f (stale) Mirror 17: 0xfdf2e74f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x333ef): Mirror 3: 0xfdf2e74f (stale) Mirror 4: 0xfdf2e74f (stale) Mirror 7: 0xfdf2e74f (stale) Mirror 8: 0xfdf2e74f (stale) Mirror 10: 0xfdf2e74f (stale) Mirror 11: 0xfdf2e74f (stale) Mirror 13: 0xfdf2e74f (stale) Mirror 17: 0xfdf2e74f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x333ef: skipped 000360 1786690547.356046 trunc from 0x0333ef to 0x013a5a (0x1f994 bytes) 000361 1786690547.404338 trunc from 0x00cb01 to 0x0119b8 (0x4eb8 bytes) 000362 1786690547.417875 punch from 0x011667 to 0x013a5a (0x23f4 bytes) 000363 1786690547.428804 mapwrite 0x03de6b thru 0x03ffff (0x2195 bytes) 000364 1786690547.547238 write 0x01b5ec thru 0x028292 (0xcca7 bytes) 000365 1786690547.576605 trunc from 0x01ab5e to 0x0281c7 (0xd66a bytes) 000366 1786690547.618829 read 0x0176c1 thru 0x01d8ab (0x61eb bytes) 000367 1786690547.628638 mapwrite 0x02f6c6 thru 0x039d61 (0xa69c bytes) 000368 1786690547.665854 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4d46c656 (stale) Mirror 4: 0x4d46c656 (stale) Mirror 7: 0x4d46c656 (stale) Mirror 8: 0x4d46c656 (stale) Mirror 10: 0x4d46c656 (stale) Mirror 11: 0x4d46c656 (stale) Mirror 13: 0x4d46c656 (stale) Mirror 17: 0x4d46c656 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85f5b1b9 Mirror 18: 0x85f5b1b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000369 1786690549.094903 mapread 0x02e256 thru 0x03c5f6 (0xe3a1 bytes) 000370 1786690549.177927 read 0x00899d thru 0x009e23 (0x1487 bytes) 000371 1786690549.193726 mapread 0x02e359 thru 0x03d692 (0xf33a bytes) 000373 1786690549.239709 mapread 0x015d9d thru 0x01c492 (0x66f6 bytes) 000374 1786690549.306191 read 0x02a739 thru 0x036987 (0xc24f bytes) 000376 1786690549.333623 trunc from 0x016ecf to 0x01eb9c (0x7cce bytes) 000377 1786690549.417001 mapwrite 0x02ee29 thru 0x033e0d (0x4fe5 bytes) 000378 1786690549.463408 write 0x01002a thru 0x012f45 (0x2f1c bytes) 000380 1786690549.475203 punch from 0x03b88b to 0x03ffff (0x4775 bytes) 000381 1786690549.498073 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4d46c656 (stale) Mirror 7: 0x4d46c656 (stale) Mirror 8: 0x4d46c656 (stale) Mirror 10: 0x4d46c656 (stale) Mirror 11: 0x4d46c656 (stale) Mirror 13: 0x4d46c656 (stale) Mirror 17: 0x4d46c656 (stale) Mirror 18: 0x85f5b1b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4d46c656 (stale) Mirror 7: 0x4d46c656 (stale) Mirror 8: 0x4d46c656 (stale) Mirror 10: 0x4d46c656 (stale) Mirror 11: 0x4d46c656 (stale) Mirror 13: 0x4d46c656 (stale) Mirror 17: 0x4d46c656 (stale) Mirror 18: 0x85f5b1b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1786690550.207465 trunc from 0x040000 to 0x011712 (0x2e8ed bytes) 000383 1786690550.242116 trunc from 0x011713 to 0x024330 (0x12c1e bytes) 000384 1786690550.278371 trunc from 0x00456c to 0x010e54 (0xc8e9 bytes) 000385 1786690550.293403 write 0x02baed thru 0x03666a (0xab7e bytes) 000386 1786690550.295071 read 0x010a46 thru 0x0112a5 (0x860 bytes) 000387 1786690550.331187 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3666b): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3666b): Mirror 3: 0xc7c9b4bf (stale) Mirror 7: 0xc7c9b4bf (stale) Mirror 8: 0xc7c9b4bf (stale) Mirror 10: 0xc7c9b4bf (stale) Mirror 11: 0xc7c9b4bf (stale) Mirror 13: 0xc7c9b4bf (stale) Mirror 17: 0xc7c9b4bf (stale) Mirror 18: 0xb23ad3a9 (stale) CRC-32 checksum value for chunk [0, 0x3666b): Mirror 1: 0x1518be8d Mirror 19: 0x1518be8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3666b: skipped 000388 1786690551.938459 mapwrite 0x029aa4 thru 0x02a64f (0xbac bytes) 000389 1786690552.056276 MIRROR_RESYNC 10 mirrors 000390 1786690554.514120 trunc from 0x03666b to 0x037f00 (0x1896 bytes) 000391 1786690554.565730 read 0x0092dd thru 0x00ce03 (0x3b27 bytes) 000392 1786690554.573743 punch from 0x011d43 to 0x01342d (0x16eb bytes) 000393 1786690554.579171 trunc from 0x01effd to 0x024503 (0x5507 bytes) 000394 1786690554.584991 punch from 0x00fad7 to 0x01c4ae (0xc9d8 bytes) 000395 1786690554.591115 trunc from 0x01f67c to 0x029a5d (0xa3e2 bytes) 000396 1786690554.597683 read 0x00ef57 thru 0x012230 (0x32da bytes) 000397 1786690554.611559 trunc from 0x037f01 to 0x027767 (0x10799 bytes) 000398 1786690554.639021 write 0x00ab3c thru 0x01a447 (0xf90c bytes) 000399 1786690554.642715 write 0x0234d1 thru 0x0260f3 (0x2c23 bytes) 000400 1786690554.644312 punch from 0x022f36 to 0x027767 (0x4832 bytes) 000401 1786690554.661125 trunc from 0x001fcd to 0x009eaa (0x7ede bytes) 000403 1786690554.667325 fallocate 0x025d79 thru 0x028936 (0x2bbe bytes) 000404 1786690554.673147 write 0x01bf98 thru 0x01d3e6 (0x144f bytes) 000405 1786690554.674733 write 0x036f3e thru 0x03dab7 (0x6b7a bytes) 000406 1786690554.676929 mapwrite 0x01e184 thru 0x0220e7 (0x3f64 bytes) 000407 1786690554.713992 read 0x03c3ee thru 0x03dab7 (0x16ca bytes) 000408 1786690554.714672 fallocate 0x00768d thru 0x016b2f (0xf4a3 bytes) 000409 1786690554.728211 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dab8): Mirror 3: 0x3f1da7a5 (stale) Mirror 7: 0x3f1da7a5 (stale) Mirror 8: 0x3f1da7a5 (stale) Mirror 11: 0x3f1da7a5 (stale) Mirror 13: 0x3f1da7a5 (stale) Mirror 17: 0x3f1da7a5 (stale) Mirror 18: 0x3f1da7a5 (stale) Mirror 19: 0x3f1da7a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dab8): Mirror 3: 0x3f1da7a5 (stale) Mirror 7: 0x3f1da7a5 (stale) Mirror 8: 0x3f1da7a5 (stale) Mirror 11: 0x3f1da7a5 (stale) Mirror 13: 0x3f1da7a5 (stale) Mirror 17: 0x3f1da7a5 (stale) Mirror 18: 0x3f1da7a5 (stale) Mirror 19: 0x3f1da7a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dab8: skipped 000410 1786690555.517520 read 0x03c120 thru 0x03c7e8 (0x6c9 bytes) 000411 1786690555.523448 trunc from 0x03dab8 to 0x00ccad (0x30e0a bytes) 000412 1786690555.549334 write 0x0389bf thru 0x03ffff (0x7641 bytes) 000413 1786690555.552748 write 0x025ad1 thru 0x0278a7 (0x1dd7 bytes) 000415 1786690555.568315 fallocate 0x0139e5 thru 0x01515e (0x177a bytes) 000416 1786690555.573969 mapread 0x000db6 thru 0x007fa5 (0x71f0 bytes) 000417 1786690555.597975 trunc from 0x040000 to 0x02cfc5 (0x1303a bytes) 000418 1786690555.617996 read 0x01c024 thru 0x0243a5 (0x8382 bytes) 000419 1786690555.627656 write 0x022b58 thru 0x023218 (0x6c1 bytes) 000420 1786690555.628990 punch from 0x01afc7 to 0x028f7e (0xdfb8 bytes) 000421 1786690555.640887 mapread 0x028cb5 thru 0x02cfc5 (0x4311 bytes) 000422 1786690555.651010 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cfc6): Mirror 3: 0x9b73c140 (stale) Mirror 7: 0x9b73c140 (stale) Mirror 8: 0x9b73c140 (stale) Mirror 11: 0x9b73c140 (stale) Mirror 13: 0x9b73c140 (stale) Mirror 17: 0x9b73c140 (stale) Mirror 19: 0x9b73c140 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cfc6): Mirror 3: 0x9b73c140 (stale) Mirror 7: 0x9b73c140 (stale) Mirror 8: 0x9b73c140 (stale) Mirror 11: 0x9b73c140 (stale) Mirror 13: 0x9b73c140 (stale) Mirror 17: 0x9b73c140 (stale) Mirror 19: 0x9b73c140 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cfc6: skipped 000423 1786690556.130047 mapwrite 0x00312e thru 0x0068f3 (0x37c6 bytes) 000424 1786690556.161736 punch from 0x006347 to 0x00adc1 (0x4a7b bytes) 000426 1786690556.167931 mapwrite 0x035cfe thru 0x03d869 (0x7b6c bytes) 000427 1786690556.229139 mapread 0x031f41 thru 0x034c03 (0x2cc3 bytes) 000428 1786690556.237146 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 3: 0xc3184c0a (stale) Mirror 7: 0xc3184c0a (stale) Mirror 8: 0xc3184c0a (stale) Mirror 11: 0xc3184c0a (stale) Mirror 13: 0xc3184c0a (stale) Mirror 17: 0xc3184c0a (stale) Mirror 19: 0xc3184c0a (stale) CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 1: 0x274852d5 Mirror 20: 0x274852d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d86a: skipped 000429 1786690557.909697 mapread 0x0265e9 thru 0x030216 (0x9c2e bytes) 000430 1786690557.978618 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 3: 0xc3184c0a (stale) Mirror 7: 0xc3184c0a (stale) Mirror 8: 0xc3184c0a (stale) Mirror 11: 0xc3184c0a (stale) Mirror 13: 0xc3184c0a (stale) Mirror 17: 0xc3184c0a (stale) Mirror 19: 0xc3184c0a (stale) CRC-32 checksum value for chunk [0, 0x3d86a): Mirror 1: 0x274852d5 Mirror 20: 0x274852d5 Mirror 21: 0x274852d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d86a: skipped 000431 1786690559.488164 trunc from 0x03d86a to 0x01d7af (0x200ba bytes) 000432 1786690559.605648 fallocate 0x02c9a7 thru 0x03b0ff (0xe759 bytes) 000434 1786690559.612884 fallocate 0x02e9bc thru 0x035f79 (0x75be bytes) 000435 1786690559.619647 trunc from 0x03b100 to 0x029146 (0x11fb9 bytes) 000436 1786690559.643163 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x29147): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29147): Mirror 3: 0x9e7298e3 (stale) Mirror 7: 0x9e7298e3 (stale) Mirror 8: 0x9e7298e3 (stale) Mirror 11: 0x9e7298e3 (stale) Mirror 13: 0x9e7298e3 (stale) Mirror 17: 0x9e7298e3 (stale) Mirror 19: 0x9e7298e3 (stale) Mirror 20: 0x94903ce0 (stale) Mirror 21: 0x94903ce0 (stale) CRC-32 checksum value for chunk [0, 0x29147): Mirror 1: 0x94903ce0 Mirror 22: 0x94903ce0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29147: skipped 000437 1786690561.225662 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x29147): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29147): Mirror 3: 0x9e7298e3 (stale) Mirror 7: 0x9e7298e3 (stale) Mirror 8: 0x9e7298e3 (stale) Mirror 11: 0x9e7298e3 (stale) Mirror 13: 0x9e7298e3 (stale) Mirror 17: 0x9e7298e3 (stale) Mirror 19: 0x9e7298e3 (stale) Mirror 20: 0x94903ce0 (stale) Mirror 21: 0x94903ce0 (stale) CRC-32 checksum value for chunk [0, 0x29147): Mirror 1: 0x94903ce0 Mirror 22: 0x94903ce0 Mirror 23: 0x94903ce0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29147: skipped 000438 1786690562.557883 fallocate 0x009bca thru 0x00f1f5 (0x562c bytes) 000439 1786690562.613566 write 0x0080d8 thru 0x01590d (0xd836 bytes) 000440 1786690562.652512 fallocate 0x03a96d thru 0x03ffff (0x5693 bytes) 000442 1786690562.663918 read 0x0123f8 thru 0x01e226 (0xbe2f bytes) 000443 1786690562.685701 mapread 0x00af1a thru 0x0156de (0xa7c5 bytes) 000444 1786690562.688114 mapread 0x00f2a2 thru 0x01cd4f (0xdaae bytes) 000445 1786690562.699276 MIRROR_RESYNC 12 mirrors 000446 1786690565.672407 trunc from 0x0065b2 to 0x00994d (0x339c bytes) 000447 1786690565.729582 MIRROR_RESYNC 12 mirrors 000448 1786690568.538429 punch from 0x0228d8 to 0x024bee (0x2317 bytes) 000449 1786690568.586190 trunc from 0x029147 to 0x02335a (0x5dec bytes) 000450 1786690568.611157 fallocate 0x01cb79 thru 0x01e0e2 (0x156a bytes) 000451 1786690568.615809 trunc from 0x02335b to 0x02b3e8 (0x808e bytes) 000452 1786690568.632830 trunc from 0x028a2c to 0x02b3e8 (0x29bd bytes) 000454 1786690568.644623 read 0x02699e thru 0x028e06 (0x2469 bytes) 000455 1786690568.669810 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b3e9): Mirror 3: 0xdc25854c (stale) Mirror 7: 0xdc25854c (stale) Mirror 8: 0xdc25854c (stale) Mirror 11: 0xdc25854c (stale) Mirror 13: 0xdc25854c (stale) Mirror 19: 0xdc25854c (stale) Mirror 20: 0xdc25854c (stale) Mirror 21: 0xdc25854c (stale) Mirror 22: 0xdc25854c (stale) Mirror 23: 0xdc25854c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3e9): Mirror 3: 0xdc25854c (stale) Mirror 7: 0xdc25854c (stale) Mirror 8: 0xdc25854c (stale) Mirror 11: 0xdc25854c (stale) Mirror 13: 0xdc25854c (stale) Mirror 19: 0xdc25854c (stale) Mirror 20: 0xdc25854c (stale) Mirror 21: 0xdc25854c (stale) Mirror 22: 0xdc25854c (stale) Mirror 23: 0xdc25854c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3e9: skipped 000456 1786690569.448168 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b3e9): Mirror 3: 0xdc25854c (stale) Mirror 7: 0xdc25854c (stale) Mirror 8: 0xdc25854c (stale) Mirror 13: 0xdc25854c (stale) Mirror 19: 0xdc25854c (stale) Mirror 20: 0xdc25854c (stale) Mirror 21: 0xdc25854c (stale) Mirror 22: 0xdc25854c (stale) Mirror 23: 0xdc25854c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3e9): Mirror 3: 0xdc25854c (stale) Mirror 7: 0xdc25854c (stale) Mirror 8: 0xdc25854c (stale) Mirror 13: 0xdc25854c (stale) Mirror 19: 0xdc25854c (stale) Mirror 20: 0xdc25854c (stale) Mirror 21: 0xdc25854c (stale) Mirror 22: 0xdc25854c (stale) Mirror 23: 0xdc25854c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3e9: skipped 000457 1786690569.931950 MIRROR_RESYNC 10 mirrors 000459 1786690572.914765 fallocate 0x00c95b thru 0x00fc00 (0x32a6 bytes) 000460 1786690572.961155 fallocate 0x01b713 thru 0x01c7b6 (0x10a4 bytes) 000461 1786690572.969311 fallocate 0x01f20e thru 0x02db3a (0xe92d bytes) 000462 1786690572.978708 mapread 0x00a1bd thru 0x014b90 (0xa9d4 bytes) 000463 1786690573.007293 read 0x003c18 thru 0x0047b5 (0xb9e bytes) 000464 1786690573.014009 mapwrite 0x035b0e thru 0x03dbd9 (0x80cc bytes) 000465 1786690573.081507 read 0x010f40 thru 0x016d1c (0x5ddd bytes) 000467 1786690573.092890 fallocate 0x00173e thru 0x00f833 (0xe0f6 bytes) 000468 1786690573.097571 trunc from 0x03dbda to 0x01cf91 (0x20c48 bytes) 000469 1786690573.122638 fallocate 0x03c524 thru 0x03ffff (0x3adc bytes) 000470 1786690573.128374 mapwrite 0x029d0b thru 0x02c188 (0x247e bytes) 000471 1786690573.177473 mapwrite 0x02414f thru 0x030f0a (0xcdbc bytes) 000472 1786690573.270264 trunc from 0x030f0b to 0x01ee39 (0x120d1 bytes) 000473 1786690573.290583 read 0x00c6bc thru 0x0178d1 (0xb216 bytes) 000474 1786690573.292457 mapread 0x00c57a thru 0x0141ec (0x7c73 bytes) 000475 1786690573.300958 MIRROR_RESYNC 10 mirrors 000476 1786690576.248312 trunc from 0x008071 to 0x015234 (0xd1c4 bytes) 000477 1786690576.302303 mapread 0x010a34 thru 0x01998a (0x8f57 bytes) 000478 1786690576.339222 mapread 0x0158bd thru 0x01c63c (0x6d80 bytes) 000479 1786690576.347949 mapread 0x009c51 thru 0x00eba8 (0x4f58 bytes) 000480 1786690576.364873 fallocate 0x03892b thru 0x03ffff (0x76d5 bytes) 000481 1786690576.372301 mapread 0x01d1ff thru 0x01f47f (0x2281 bytes) 000482 1786690576.412303 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x354807a5 (stale) Mirror 7: 0x354807a5 (stale) Mirror 8: 0x354807a5 (stale) Mirror 13: 0x354807a5 (stale) Mirror 19: 0x354807a5 (stale) Mirror 20: 0x354807a5 (stale) Mirror 21: 0x354807a5 (stale) Mirror 22: 0x354807a5 (stale) Mirror 23: 0x354807a5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa6e095e6 Mirror 24: 0xa6e095e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000483 1786690577.774515 write 0x00e829 thru 0x01a237 (0xba0f bytes) 000484 1786690577.859336 punch from 0x035210 to 0x03b25e (0x604f bytes) 000485 1786690577.870546 MIRROR_RESYNC 11 mirrors 000486 1786690580.894596 trunc from 0x040000 to 0x021a28 (0x1e5d7 bytes) 000487 1786690580.961256 write 0x006d36 thru 0x0126dd (0xb9a8 bytes) 000488 1786690580.973915 write 0x011c8d thru 0x019b10 (0x7e84 bytes) 000489 1786690580.981073 mapwrite 0x0254bc thru 0x03071a (0xb25f bytes) 000490 1786690581.069266 trunc from 0x00b085 to 0x00e8a6 (0x3822 bytes) 000491 1786690581.080398 read 0x00071f thru 0x004085 (0x3967 bytes) 000492 1786690581.092232 trunc from 0x019bf3 to 0x01fe12 (0x6220 bytes) 000493 1786690581.099386 write 0x033184 thru 0x03ffff (0xce7c bytes) 000494 1786690581.103893 read 0x02e334 thru 0x0369c5 (0x8692 bytes) 000495 1786690581.125238 fallocate 0x00d99a thru 0x018ede (0xb545 bytes) 000496 1786690581.132542 read 0x02cf3c thru 0x03ab22 (0xdbe7 bytes) 000497 1786690581.134408 mapread 0x03da9c thru 0x03ffff (0x2564 bytes) 000499 1786690581.137270 read 0x010e74 thru 0x015e46 (0x4fd3 bytes) 000500 1786690581.138180 mapwrite 0x006b8e thru 0x00be31 (0x52a4 bytes) 000501 1786690581.160491 mapwrite 0x0077e0 thru 0x016742 (0xef63 bytes) 000502 1786690581.185990 punch from 0x025d6d to 0x027b63 (0x1df7 bytes) 000503 1786690581.191667 fallocate 0x018bd5 thru 0x023676 (0xaaa2 bytes) 000504 1786690581.197259 punch from 0x020a8a to 0x02803e (0x75b5 bytes) 000505 1786690581.203299 punch from 0x00a690 to 0x011b80 (0x74f1 bytes) 000506 1786690581.207960 fallocate 0x036c17 thru 0x03ffff (0x93e9 bytes) 000507 1786690581.212800 read 0x0247a3 thru 0x034010 (0xf86e bytes) 000508 1786690581.226306 mapread 0x031068 thru 0x03ff9f (0xef38 bytes) 000509 1786690581.227903 read 0x038174 thru 0x03e14a (0x5fd7 bytes) 000510 1786690581.233551 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ea99edd (stale) Mirror 7: 0x4ea99edd (stale) Mirror 8: 0x4ea99edd (stale) Mirror 13: 0x4ea99edd (stale) Mirror 19: 0x4ea99edd (stale) Mirror 20: 0x4ea99edd (stale) Mirror 21: 0x4ea99edd (stale) Mirror 22: 0x4ea99edd (stale) Mirror 23: 0x4ea99edd (stale) Mirror 24: 0x4ea99edd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7616ecfa Mirror 25: 0x7616ecfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1786690582.660686 write 0x005e0a thru 0x012896 (0xca8d bytes) 000512 1786690582.771641 write 0x0378b0 thru 0x038b4f (0x12a0 bytes) 000513 1786690582.785003 fallocate 0x00c275 thru 0x00d5c8 (0x1354 bytes) 000514 1786690582.796148 mapwrite 0x008a0a thru 0x00c500 (0x3af7 bytes) 000515 1786690582.824561 read 0x018bcc thru 0x0193b5 (0x7ea bytes) 000517 1786690582.847735 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ea99edd (stale) Mirror 7: 0x4ea99edd (stale) Mirror 8: 0x4ea99edd (stale) Mirror 13: 0x4ea99edd (stale) Mirror 19: 0x4ea99edd (stale) Mirror 20: 0x4ea99edd (stale) Mirror 21: 0x4ea99edd (stale) Mirror 22: 0x4ea99edd (stale) Mirror 24: 0x4ea99edd (stale) Mirror 25: 0x7616ecfa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ea99edd (stale) Mirror 7: 0x4ea99edd (stale) Mirror 8: 0x4ea99edd (stale) Mirror 13: 0x4ea99edd (stale) Mirror 19: 0x4ea99edd (stale) Mirror 20: 0x4ea99edd (stale) Mirror 21: 0x4ea99edd (stale) Mirror 22: 0x4ea99edd (stale) Mirror 24: 0x4ea99edd (stale) Mirror 25: 0x7616ecfa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000518 1786690583.663900 trunc from 0x0290c9 to 0x0381b6 (0xf0ee bytes) 000520 1786690583.671441 mapwrite 0x02d6e0 thru 0x03b760 (0xe081 bytes) 000521 1786690583.728928 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ea99edd (stale) Mirror 7: 0x4ea99edd (stale) Mirror 8: 0x4ea99edd (stale) Mirror 13: 0x4ea99edd (stale) Mirror 19: 0x4ea99edd (stale) Mirror 20: 0x4ea99edd (stale) Mirror 21: 0x4ea99edd (stale) Mirror 24: 0x4ea99edd (stale) Mirror 25: 0x7616ecfa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ea99edd (stale) Mirror 7: 0x4ea99edd (stale) Mirror 8: 0x4ea99edd (stale) Mirror 13: 0x4ea99edd (stale) Mirror 19: 0x4ea99edd (stale) Mirror 20: 0x4ea99edd (stale) Mirror 21: 0x4ea99edd (stale) Mirror 24: 0x4ea99edd (stale) Mirror 25: 0x7616ecfa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000522 1786690584.499507 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ea99edd (stale) Mirror 7: 0x4ea99edd (stale) Mirror 13: 0x4ea99edd (stale) Mirror 19: 0x4ea99edd (stale) Mirror 20: 0x4ea99edd (stale) Mirror 21: 0x4ea99edd (stale) Mirror 24: 0x4ea99edd (stale) Mirror 25: 0x7616ecfa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4ea99edd (stale) Mirror 7: 0x4ea99edd (stale) Mirror 13: 0x4ea99edd (stale) Mirror 19: 0x4ea99edd (stale) Mirror 20: 0x4ea99edd (stale) Mirror 21: 0x4ea99edd (stale) Mirror 24: 0x4ea99edd (stale) Mirror 25: 0x7616ecfa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000523 1786690585.105925 fallocate 0x02fba8 thru 0x031b99 (0x1ff2 bytes) 000524 1786690585.116099 mapwrite 0x02d288 thru 0x035389 (0x8102 bytes) 000526 1786690585.185256 fallocate 0x02d513 thru 0x03a3dd (0xcecb bytes) 000527 1786690585.191690 trunc from 0x002b42 to 0x0038bc (0xd7b bytes) 000528 1786690585.199824 trunc from 0x040000 to 0x0132d4 (0x2cd2b bytes) 000530 1786690585.255213 read 0x011975 thru 0x0132b5 (0x1941 bytes) 000531 1786690585.262139 mapwrite 0x03fcc7 thru 0x03ffff (0x339 bytes) 000532 1786690585.346288 punch from 0x02152a to 0x02733c (0x5e13 bytes) 000533 1786690585.359279 trunc from 0x040000 to 0x02ab18 (0x154e7 bytes) 000535 1786690585.391899 write 0x02af3e thru 0x02b508 (0x5cb bytes) 000536 1786690585.403549 trunc from 0x02159f to 0x028970 (0x73d2 bytes) 000537 1786690585.415973 trunc from 0x0135e7 to 0x01796e (0x4388 bytes) 000539 1786690585.425231 read 0x02240f thru 0x022849 (0x43b bytes) 000540 1786690585.439168 fallocate 0x00c40d thru 0x015bd4 (0x97c8 bytes) 000541 1786690585.444253 mapread 0x017440 thru 0x01a2ba (0x2e7b bytes) 000542 1786690585.446124 punch from 0x019a84 to 0x01faa5 (0x6022 bytes) 000543 1786690585.453626 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2b509): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b509): Mirror 3: 0xc6c8a9f8 (stale) Mirror 7: 0xc6c8a9f8 (stale) Mirror 13: 0xc6c8a9f8 (stale) Mirror 19: 0xc6c8a9f8 (stale) Mirror 20: 0xc6c8a9f8 (stale) Mirror 21: 0xc6c8a9f8 (stale) Mirror 24: 0xc6c8a9f8 (stale) Mirror 25: 0x105f70f5 (stale) CRC-32 checksum value for chunk [0, 0x2b509): Mirror 1: 0x561f76a7 Mirror 26: 0x561f76a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b509: skipped 000544 1786690586.418255 mapwrite 0x036c6a thru 0x03a870 (0x3c07 bytes) 000545 1786690586.567159 trunc from 0x02372d to 0x027319 (0x3bed bytes) 000546 1786690586.575326 read 0x015cda thru 0x016fed (0x1314 bytes) 000547 1786690586.581652 read 0x022fb0 thru 0x029b40 (0x6b91 bytes) 000548 1786690586.592660 read 0x024d77 thru 0x02fb38 (0xadc2 bytes) 000550 1786690586.598598 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a871): Mirror 3: 0x8d992713 (stale) Mirror 13: 0x8d992713 (stale) Mirror 19: 0x8d992713 (stale) Mirror 20: 0x8d992713 (stale) Mirror 21: 0x8d992713 (stale) Mirror 24: 0x8d992713 (stale) Mirror 25: 0xce9f1bc6 (stale) Mirror 26: 0xd994a7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a871): Mirror 3: 0x8d992713 (stale) Mirror 13: 0x8d992713 (stale) Mirror 19: 0x8d992713 (stale) Mirror 20: 0x8d992713 (stale) Mirror 21: 0x8d992713 (stale) Mirror 24: 0x8d992713 (stale) Mirror 25: 0xce9f1bc6 (stale) Mirror 26: 0xd994a7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a871: skipped 000551 1786690587.080838 punch from 0x008a3d to 0x00a9e5 (0x1fa9 bytes) 000552 1786690587.085575 punch from 0x03445e to 0x036799 (0x233c bytes) 000553 1786690587.092553 write 0x00af12 thru 0x013a0f (0x8afe bytes) 000554 1786690587.105970 punch from 0x007b55 to 0x00a5b7 (0x2a63 bytes) 000556 1786690587.120408 trunc from 0x010be1 to 0x01bd21 (0xb141 bytes) 000557 1786690587.125758 fallocate 0x006eb1 thru 0x00d28b (0x63db bytes) 000559 1786690587.137876 mapwrite 0x029eab thru 0x02e695 (0x47eb bytes) 000560 1786690587.181004 trunc from 0x03a871 to 0x01315f (0x27711 bytes) 000561 1786690587.208967 fallocate 0x03203d thru 0x0357b6 (0x377a bytes) 000562 1786690587.214291 fallocate 0x0351bf thru 0x037ad7 (0x2919 bytes) 000563 1786690587.219043 read 0x006267 thru 0x00d49e (0x7238 bytes) 000564 1786690587.228319 trunc from 0x037ad8 to 0x01993d (0x1e19a bytes) 000565 1786690587.243230 write 0x00529e thru 0x00a350 (0x50b3 bytes) 000567 1786690587.250100 punch from 0x011135 to 0x01993d (0x8809 bytes) 000568 1786690587.261142 MIRROR_RESYNC 9 mirrors 000569 1786690589.278234 fallocate 0x010961 thru 0x016a9a (0x613a bytes) 000570 1786690589.333911 write 0x03c210 thru 0x03ee0a (0x2bfb bytes) 000571 1786690589.336576 fallocate 0x02171e thru 0x02a187 (0x8a6a bytes) 000572 1786690589.346937 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ee0b): Mirror 3: (stale - read error) Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee0b): Mirror 3: 0x45262b9 (stale) Mirror 13: 0x45262b9 (stale) Mirror 19: 0x45262b9 (stale) Mirror 20: 0x45262b9 (stale) Mirror 21: 0x45262b9 (stale) Mirror 24: 0x45262b9 (stale) Mirror 25: 0x45262b9 (stale) Mirror 26: 0x45262b9 (stale) CRC-32 checksum value for chunk [0, 0x3ee0b): Mirror 1: 0x15c6ef1d Mirror 27: 0x15c6ef1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee0b: skipped 000573 1786690590.717518 punch from 0x004bd8 to 0x00cfb2 (0x83db bytes) 000574 1786690590.801158 punch from 0x036c5c to 0x03ddd5 (0x717a bytes) 000575 1786690590.809774 punch from 0x007b72 to 0x00e6f3 (0x6b82 bytes) 000576 1786690590.816428 punch from 0x017695 to 0x02290f (0xb27b bytes) 000577 1786690590.834779 fallocate 0x01a57b thru 0x01ba80 (0x1506 bytes) 000578 1786690590.841818 fallocate 0x035ee3 thru 0x03ffff (0xa11d bytes) 000579 1786690590.849209 punch from 0x039d15 to 0x03b2c6 (0x15b2 bytes) 000580 1786690590.857618 mapread 0x01f2f8 thru 0x029ca9 (0xa9b2 bytes) 000582 1786690590.878226 trunc from 0x03ee0b to 0x01e8ed (0x2051d bytes) 000583 1786690590.927782 read 0x006d3a thru 0x00e241 (0x7508 bytes) 000584 1786690590.935940 read 0x0109ac thru 0x019307 (0x895c bytes) 000585 1786690590.943666 trunc from 0x01e8ee to 0x02983d (0xaf50 bytes) 000586 1786690590.961548 mapread 0x025a95 thru 0x027b12 (0x207e bytes) 000588 1786690590.981157 trunc from 0x02983e to 0x02a824 (0xfe7 bytes) 000589 1786690591.003735 mapwrite 0x00c47e thru 0x011572 (0x50f5 bytes) 000590 1786690591.046175 mapwrite 0x03776d thru 0x03ffff (0x8893 bytes) 000591 1786690591.129032 punch from 0x03c989 to 0x03fa86 (0x30fe bytes) 000592 1786690591.142219 punch from 0x03c625 to 0x03ffff (0x39db bytes) 000593 1786690591.153502 trunc from 0x03a149 to 0x03ffff (0x5eb7 bytes) 000595 1786690591.163239 mapread 0x0286f9 thru 0x03762f (0xef37 bytes) 000596 1786690591.165116 read 0x00dd8b thru 0x01bc70 (0xdee6 bytes) 000597 1786690591.168575 fallocate 0x001366 thru 0x007b2c (0x67c7 bytes) 000598 1786690591.181280 mapwrite 0x0104dd thru 0x019b6d (0x9691 bytes) 000599 1786690591.207837 mapread 0x03db28 thru 0x03ffff (0x24d8 bytes) 000600 1786690591.220811 punch from 0x014ab4 to 0x01fc6d (0xb1ba bytes) 000601 1786690591.228664 trunc from 0x0035fd to 0x00d373 (0x9d77 bytes) 000602 1786690591.241612 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x84c8671c (stale) Mirror 13: 0x84c8671c (stale) Mirror 19: 0x84c8671c (stale) Mirror 20: 0x84c8671c (stale) Mirror 21: 0x84c8671c (stale) Mirror 25: 0x84c8671c (stale) Mirror 26: 0x84c8671c (stale) Mirror 27: 0x854748dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x84c8671c (stale) Mirror 13: 0x84c8671c (stale) Mirror 19: 0x84c8671c (stale) Mirror 20: 0x84c8671c (stale) Mirror 21: 0x84c8671c (stale) Mirror 25: 0x84c8671c (stale) Mirror 26: 0x84c8671c (stale) Mirror 27: 0x854748dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000603 1786690591.904586 trunc from 0x040000 to 0x029153 (0x16eac bytes) 000604 1786690591.923944 mapread 0x0152aa thru 0x01ae5d (0x5bb4 bytes) 000606 1786690591.944491 mapread 0x0022b3 thru 0x0032f8 (0x1046 bytes) 000607 1786690591.958618 trunc from 0x021a87 to 0x0247c5 (0x2d3f bytes) 000608 1786690591.963255 trunc from 0x012843 to 0x01a47f (0x7c3d bytes) 000609 1786690591.969419 trunc from 0x01155d to 0x01d25b (0xbcff bytes) 000610 1786690591.974508 mapread 0x025e89 thru 0x029153 (0x32cb bytes) 000612 1786690591.986484 mapread 0x021892 thru 0x029153 (0x78c2 bytes) 000613 1786690591.987658 read 0x014875 thru 0x015cca (0x1456 bytes) 000614 1786690591.995131 trunc from 0x01203a to 0x019a35 (0x79fc bytes) 000616 1786690592.003901 punch from 0x005489 to 0x010bb1 (0xb729 bytes) 000617 1786690592.009509 mapread 0x021ee8 thru 0x0271fc (0x5315 bytes) 000618 1786690592.011469 trunc from 0x015cbb to 0x01d58f (0x78d5 bytes) 000619 1786690592.017982 write 0x01df98 thru 0x0285d0 (0xa639 bytes) 000620 1786690592.033788 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x29154): Mirror 3: (stale - read error) Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29154): Mirror 3: 0xa989111 (stale) Mirror 13: 0xa989111 (stale) Mirror 19: 0xa989111 (stale) Mirror 20: 0xa989111 (stale) Mirror 21: 0xa989111 (stale) Mirror 25: 0xa989111 (stale) Mirror 26: 0xa989111 (stale) Mirror 27: 0xa989111 (stale) CRC-32 checksum value for chunk [0, 0x29154): Mirror 1: 0xf52d7403 Mirror 28: 0xf52d7403 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29154: skipped 000621 1786690593.265642 read 0x021930 thru 0x024175 (0x2846 bytes) 000622 1786690593.307177 mapwrite 0x03bf27 thru 0x03ff6b (0x4045 bytes) 000623 1786690593.556298 fallocate 0x01248a thru 0x0167ce (0x4345 bytes) 000624 1786690593.562305 fallocate 0x0113e1 thru 0x0178f4 (0x6514 bytes) 000625 1786690593.567242 read 0x007d05 thru 0x00f929 (0x7c25 bytes) 000626 1786690593.589320 mapread 0x020ee1 thru 0x024004 (0x3124 bytes) 000627 1786690593.591257 read 0x02d800 thru 0x038842 (0xb043 bytes) 000628 1786690593.592458 fallocate 0x00b0f3 thru 0x00deb3 (0x2dc1 bytes) 000629 1786690593.598538 read 0x00d622 thru 0x018fa5 (0xb984 bytes) 000630 1786690593.604338 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff6c): Mirror 13: 0x9d6a765f (stale) Mirror 19: 0x9d6a765f (stale) Mirror 20: 0x9d6a765f (stale) Mirror 21: 0x9d6a765f (stale) Mirror 25: 0x9d6a765f (stale) Mirror 26: 0x9d6a765f (stale) Mirror 27: 0xf9c48649 (stale) Mirror 28: 0x4bed430b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff6c): Mirror 13: 0x9d6a765f (stale) Mirror 19: 0x9d6a765f (stale) Mirror 20: 0x9d6a765f (stale) Mirror 21: 0x9d6a765f (stale) Mirror 25: 0x9d6a765f (stale) Mirror 26: 0x9d6a765f (stale) Mirror 27: 0xf9c48649 (stale) Mirror 28: 0x4bed430b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff6c: skipped 000631 1786690594.209896 trunc from 0x025b38 to 0x02ad7e (0x5247 bytes) 000632 1786690594.234572 mapwrite 0x01d2bf thru 0x0266ff (0x9441 bytes) 000634 1786690594.318689 trunc from 0x03ff6c to 0x004e48 (0x3b123 bytes) 000636 1786690594.359262 fallocate 0x02089c thru 0x0252cb (0x4a30 bytes) 000637 1786690594.380939 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x252cc): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x252cc): Mirror 13: 0xfacca0cd (stale) Mirror 19: 0xfacca0cd (stale) Mirror 20: 0xfacca0cd (stale) Mirror 21: 0xfacca0cd (stale) Mirror 25: 0xfacca0cd (stale) Mirror 26: 0xfacca0cd (stale) Mirror 27: 0xfacca0cd (stale) Mirror 28: 0xc3288adc (stale) CRC-32 checksum value for chunk [0, 0x252cc): Mirror 1: 0x72e90321 Mirror 29: 0x72e90321 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x252cc: skipped 000638 1786690595.660946 read 0x0081dc thru 0x0150fa (0xcf1f bytes) 000640 1786690595.692729 punch from 0x0131b7 to 0x01f3bb (0xc205 bytes) 000642 1786690595.748165 mapwrite 0x017c7c thru 0x01b007 (0x338c bytes) 000643 1786690595.788544 mapwrite 0x03e2d8 thru 0x03ffff (0x1d28 bytes) 000644 1786690595.842114 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x84c8671c (stale) Mirror 19: 0x84c8671c (stale) Mirror 20: 0x84c8671c (stale) Mirror 21: 0x84c8671c (stale) Mirror 25: 0x84c8671c (stale) Mirror 26: 0x84c8671c (stale) Mirror 27: 0x854748dd (stale) Mirror 28: 0x4162e39a (stale) Mirror 29: 0xa4748b7b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1047759c Mirror 30: 0x1047759c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000645 1786690597.345114 trunc from 0x02871d to 0x0375ff (0xeee3 bytes) 000646 1786690597.402141 fallocate 0x037e4c thru 0x03ffff (0x81b4 bytes) 000647 1786690597.416210 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x84c8671c (stale) Mirror 19: 0x84c8671c (stale) Mirror 20: 0x84c8671c (stale) Mirror 21: 0x84c8671c (stale) Mirror 25: 0x84c8671c (stale) Mirror 26: 0x84c8671c (stale) Mirror 27: 0x854748dd (stale) Mirror 28: 0x4162e39a (stale) Mirror 30: 0x1047759c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x84c8671c (stale) Mirror 19: 0x84c8671c (stale) Mirror 20: 0x84c8671c (stale) Mirror 21: 0x84c8671c (stale) Mirror 25: 0x84c8671c (stale) Mirror 26: 0x84c8671c (stale) Mirror 27: 0x854748dd (stale) Mirror 28: 0x4162e39a (stale) Mirror 30: 0x1047759c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000648 1786690598.134651 punch from 0x036112 to 0x03ffff (0x9eee bytes) 000649 1786690598.140531 mapwrite 0x022143 thru 0x02acb6 (0x8b74 bytes) 000650 1786690598.193525 MIRROR_RESYNC 10 mirrors 000651 1786690602.217780 mapread 0x01470c thru 0x018074 (0x3969 bytes) 000652 1786690602.242687 punch from 0x02c6a9 to 0x03c68d (0xffe5 bytes) 000653 1786690602.297364 mapwrite 0x028351 thru 0x02ce11 (0x4ac1 bytes) 000654 1786690602.333855 fallocate 0x01fbd4 thru 0x029965 (0x9d92 bytes) 000655 1786690602.340182 mapwrite 0x02c504 thru 0x0349a2 (0x849f bytes) 000656 1786690602.382487 read 0x00f2da thru 0x013b67 (0x488e bytes) 000657 1786690602.388850 trunc from 0x01e741 to 0x0271b6 (0x8a76 bytes) 000658 1786690602.394225 mapwrite 0x01d7a2 thru 0x023c00 (0x645f bytes) 000659 1786690602.499895 mapread 0x01599b thru 0x016033 (0x699 bytes) 000660 1786690602.522185 mapread 0x0134b6 thru 0x01b959 (0x84a4 bytes) 000661 1786690602.526333 read 0x014f8c thru 0x0219eb (0xca60 bytes) 000662 1786690602.528269 punch from 0x004ced to 0x014206 (0xf51a bytes) 000665 1786690602.538820 write 0x0262fb thru 0x02ca35 (0x673b bytes) 000666 1786690602.556028 trunc from 0x040000 to 0x02e287 (0x11d78 bytes) 000667 1786690602.581649 mapwrite 0x022387 thru 0x026662 (0x42dc bytes) 000668 1786690602.615223 fallocate 0x0388a5 thru 0x03ffff (0x775b bytes) 000669 1786690602.620901 mapwrite 0x03838e thru 0x03ffff (0x7c72 bytes) 000670 1786690602.730860 punch from 0x004ac6 to 0x00532e (0x869 bytes) 000671 1786690602.736945 fallocate 0x020037 thru 0x025a78 (0x5a42 bytes) 000672 1786690602.744112 mapwrite 0x00b292 thru 0x00df8e (0x2cfd bytes) 000674 1786690602.761075 mapwrite 0x007eb8 thru 0x01712a (0xf273 bytes) 000675 1786690602.789680 MIRROR_RESYNC 10 mirrors 000676 1786690606.086396 punch from 0x03c98d to 0x03ffff (0x3673 bytes) 000677 1786690606.147894 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x1d005d6e (stale) Mirror 19: 0x1d005d6e (stale) Mirror 20: 0x1d005d6e (stale) Mirror 21: 0x1d005d6e (stale) Mirror 25: 0x1d005d6e (stale) Mirror 26: 0x1d005d6e (stale) Mirror 27: 0x1d005d6e (stale) Mirror 28: 0x1d005d6e (stale) Mirror 30: 0x1d005d6e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaad7ac42 Mirror 31: 0xaad7ac42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000678 1786690607.517196 read 0x01e0c3 thru 0x02b7ea (0xd728 bytes) 000679 1786690607.549287 trunc from 0x018b3c to 0x02067f (0x7b44 bytes) 000680 1786690607.646571 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x1d005d6e (stale) Mirror 19: 0x1d005d6e (stale) Mirror 20: 0x1d005d6e (stale) Mirror 21: 0x1d005d6e (stale) Mirror 25: 0x1d005d6e (stale) Mirror 26: 0x1d005d6e (stale) Mirror 27: 0x1d005d6e (stale) Mirror 28: 0x1d005d6e (stale) Mirror 30: 0x1d005d6e (stale) Mirror 31: 0xaad7ac42 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7883df08 Mirror 32: 0x7883df08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000681 1786690609.192869 trunc from 0x040000 to 0x00960d (0x369f2 bytes) 000682 1786690609.266975 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x960e): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x960e): Mirror 13: 0xa4bca779 (stale) Mirror 19: 0xa4bca779 (stale) Mirror 20: 0xa4bca779 (stale) Mirror 21: 0xa4bca779 (stale) Mirror 25: 0xa4bca779 (stale) Mirror 26: 0xa4bca779 (stale) Mirror 27: 0xa4bca779 (stale) Mirror 28: 0xa4bca779 (stale) Mirror 30: 0xa4bca779 (stale) Mirror 31: 0xa4bca779 (stale) Mirror 32: 0xa4bca779 (stale) CRC-32 checksum value for chunk [0, 0x960e): Mirror 1: 0xa4bca779 Mirror 33: 0xa4bca779 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x960e: skipped 000683 1786690610.604627 mapwrite 0x039886 thru 0x03ffff (0x677a bytes) 000684 1786690610.698497 fallocate 0x03356c thru 0x037774 (0x4209 bytes) 000685 1786690610.704578 mapwrite 0x020d4e thru 0x024c95 (0x3f48 bytes) 000686 1786690610.731987 trunc from 0x040000 to 0x03dabd (0x2542 bytes) 000687 1786690610.756294 fallocate 0x033554 thru 0x03ffff (0xcaac bytes) 000688 1786690610.761006 punch from 0x0337b2 to 0x03dabd (0xa30c bytes) 000689 1786690610.767099 trunc from 0x03dabe to 0x0108c5 (0x2d1f8 bytes) 000691 1786690610.787495 trunc from 0x0108c6 to 0x03d52c (0x2cc67 bytes) 000692 1786690610.815312 punch from 0x0374d3 to 0x03d52c (0x605a bytes) 000693 1786690610.821455 trunc from 0x02b355 to 0x030fbb (0x5c67 bytes) 000695 1786690610.844028 write 0x024df0 thru 0x0336f5 (0xe906 bytes) 000697 1786690610.869998 trunc from 0x002343 to 0x007c4c (0x590a bytes) 000699 1786690610.884120 mapread 0x02f6c4 thru 0x033195 (0x3ad2 bytes) 000700 1786690610.890311 punch from 0x011f30 to 0x014a2b (0x2afc bytes) 000701 1786690610.902178 mapwrite 0x00e2d3 thru 0x01d60b (0xf339 bytes) 000702 1786690610.977100 read 0x0293a1 thru 0x02c6a6 (0x3306 bytes) 000703 1786690610.978087 punch from 0x038940 to 0x03c45d (0x3b1e bytes) 000704 1786690610.982910 punch from 0x0260ba to 0x02b57a (0x54c1 bytes) 000705 1786690610.992306 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d52d): Mirror 13: 0xcec4775c (stale) Mirror 19: 0xcec4775c (stale) Mirror 20: 0xcec4775c (stale) Mirror 21: 0xcec4775c (stale) Mirror 25: 0xcec4775c (stale) Mirror 26: 0xcec4775c (stale) Mirror 27: 0xcec4775c (stale) Mirror 28: 0xcec4775c (stale) Mirror 30: 0xcec4775c (stale) Mirror 32: 0x8840d842 (stale) Mirror 33: 0x5daa6434 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d52d): Mirror 13: 0xcec4775c (stale) Mirror 19: 0xcec4775c (stale) Mirror 20: 0xcec4775c (stale) Mirror 21: 0xcec4775c (stale) Mirror 25: 0xcec4775c (stale) Mirror 26: 0xcec4775c (stale) Mirror 27: 0xcec4775c (stale) Mirror 28: 0xcec4775c (stale) Mirror 30: 0xcec4775c (stale) Mirror 32: 0x8840d842 (stale) Mirror 33: 0x5daa6434 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d52d: skipped 000707 1786690611.949234 write 0x031847 thru 0x034ddb (0x3595 bytes) 000708 1786690611.982251 trunc from 0x003903 to 0x00c44c (0x8b4a bytes) 000709 1786690611.995121 mapwrite 0x005235 thru 0x013b0b (0xe8d7 bytes) 000710 1786690612.052253 punch from 0x0199fc to 0x01c36a (0x296f bytes) 000712 1786690612.060434 read 0x013146 thru 0x01b110 (0x7fcb bytes) 000713 1786690612.068139 trunc from 0x02a0b5 to 0x0347fc (0xa748 bytes) 000714 1786690612.075067 trunc from 0x03d52d to 0x0078cc (0x35c60 bytes) 000715 1786690612.093753 mapwrite 0x003e5f thru 0x00e4be (0xa660 bytes) 000716 1786690612.135700 punch from 0x00c8f3 to 0x00e4be (0x1bcc bytes) 000718 1786690612.142178 trunc from 0x0090e9 to 0x00e4be (0x53d6 bytes) 000719 1786690612.148173 write 0x03ad85 thru 0x03ffff (0x527b bytes) 000720 1786690612.151107 read 0x037abf thru 0x03e37a (0x68bc bytes) 000721 1786690612.162236 punch from 0x00080b to 0x007e67 (0x765d bytes) 000722 1786690612.168330 mapwrite 0x004d87 thru 0x005b3f (0xdb9 bytes) 000723 1786690612.194700 punch from 0x03dfd2 to 0x03f0d0 (0x10ff bytes) 000724 1786690612.200811 mapread 0x03985b thru 0x03ed78 (0x551e bytes) 000725 1786690612.213162 trunc from 0x00437f to 0x013c51 (0xf8d3 bytes) 000726 1786690612.219435 mapread 0x01f555 thru 0x02c199 (0xcc45 bytes) 000727 1786690612.221007 mapread 0x009933 thru 0x00d77f (0x3e4d bytes) 000728 1786690612.247316 mapwrite 0x02bb44 thru 0x03aef3 (0xf3b0 bytes) 000729 1786690612.267980 trunc from 0x040000 to 0x02dcf3 (0x1230c bytes) 000730 1786690612.288338 mapwrite 0x03452f thru 0x03b03d (0x6b0f bytes) 000731 1786690612.362552 mapread 0x015d4a thru 0x025088 (0xf33f bytes) 000732 1786690612.369443 MIRROR_RESYNC 12 mirrors 000733 1786690615.655008 fallocate 0x022d37 thru 0x02ced6 (0xa1a0 bytes) 000734 1786690615.691931 mapread 0x036a5d thru 0x039a41 (0x2fe5 bytes) 000735 1786690615.711894 punch from 0x00cc80 to 0x010463 (0x37e4 bytes) 000736 1786690615.720598 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b03e): Mirror 13: 0x42dc094d (stale) Mirror 19: 0x42dc094d (stale) Mirror 20: 0x42dc094d (stale) Mirror 21: 0x42dc094d (stale) Mirror 25: 0x42dc094d (stale) Mirror 26: 0x42dc094d (stale) Mirror 27: 0x42dc094d (stale) Mirror 28: 0x42dc094d (stale) Mirror 32: 0x42dc094d (stale) Mirror 33: 0x42dc094d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b03e): Mirror 13: 0x42dc094d (stale) Mirror 19: 0x42dc094d (stale) Mirror 20: 0x42dc094d (stale) Mirror 21: 0x42dc094d (stale) Mirror 25: 0x42dc094d (stale) Mirror 26: 0x42dc094d (stale) Mirror 27: 0x42dc094d (stale) Mirror 28: 0x42dc094d (stale) Mirror 32: 0x42dc094d (stale) Mirror 33: 0x42dc094d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b03e: skipped 000737 1786690616.495620 write 0x02458a thru 0x028baf (0x4626 bytes) 000739 1786690616.520928 write 0x022bed thru 0x027c7b (0x508f bytes) 000740 1786690616.528978 mapread 0x000f4b thru 0x00a47b (0x9531 bytes) 000741 1786690616.564205 read 0x0277e5 thru 0x02b0e0 (0x38fc bytes) 000742 1786690616.576134 write 0x023f11 thru 0x02ec0e (0xacfe bytes) 000743 1786690616.595675 mapread 0x0080ec thru 0x014104 (0xc019 bytes) 000744 1786690616.615823 fallocate 0x0216be thru 0x02d8c4 (0xc207 bytes) 000745 1786690616.630767 read 0x039476 thru 0x03b03d (0x1bc8 bytes) 000746 1786690616.673645 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b03e): Mirror 13: 0x42dc094d (stale) Mirror 19: 0x42dc094d (stale) Mirror 20: 0x42dc094d (stale) Mirror 21: 0x42dc094d (stale) Mirror 25: 0x42dc094d (stale) Mirror 26: 0x42dc094d (stale) Mirror 28: 0x42dc094d (stale) Mirror 32: 0x42dc094d (stale) Mirror 33: 0x42dc094d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b03e): Mirror 13: 0x42dc094d (stale) Mirror 19: 0x42dc094d (stale) Mirror 20: 0x42dc094d (stale) Mirror 21: 0x42dc094d (stale) Mirror 25: 0x42dc094d (stale) Mirror 26: 0x42dc094d (stale) Mirror 28: 0x42dc094d (stale) Mirror 32: 0x42dc094d (stale) Mirror 33: 0x42dc094d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b03e: skipped 000747 1786690617.297991 read 0x0305ca thru 0x03b03d (0xaa74 bytes) 000748 1786690617.308412 trunc from 0x0084c6 to 0x012616 (0xa151 bytes) 000749 1786690617.313236 mapread 0x0061f1 thru 0x00f806 (0x9616 bytes) 000750 1786690617.333051 write 0x01c0fa thru 0x028c90 (0xcb97 bytes) 000751 1786690617.350841 fallocate 0x036ec7 thru 0x03d314 (0x644e bytes) 000752 1786690617.361031 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3d315): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d315): Mirror 13: 0xf8fe7ec8 (stale) Mirror 19: 0xf8fe7ec8 (stale) Mirror 20: 0xf8fe7ec8 (stale) Mirror 21: 0xf8fe7ec8 (stale) Mirror 25: 0xf8fe7ec8 (stale) Mirror 26: 0xf8fe7ec8 (stale) Mirror 28: 0xf8fe7ec8 (stale) Mirror 32: 0xf8fe7ec8 (stale) Mirror 33: 0xf8fe7ec8 (stale) CRC-32 checksum value for chunk [0, 0x3d315): Mirror 1: 0x8b7287d1 Mirror 34: 0x8b7287d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d315: skipped 000753 1786690618.696962 trunc from 0x0287ba to 0x02a4e3 (0x1d2a bytes) 000754 1786690618.759097 write 0x03670f thru 0x0377aa (0x109c bytes) 000755 1786690618.779080 read 0x0073d9 thru 0x0099f9 (0x2621 bytes) 000756 1786690618.788089 write 0x028ef8 thru 0x0303cf (0x74d8 bytes) 000757 1786690618.803253 read 0x01667e thru 0x0222b4 (0xbc37 bytes) 000758 1786690618.822964 write 0x03bd70 thru 0x03ffff (0x4290 bytes) 000759 1786690618.826080 mapread 0x016254 thru 0x023f17 (0xdcc4 bytes) 000760 1786690618.831247 fallocate 0x01186a thru 0x01392b (0x20c2 bytes) 000762 1786690618.841146 mapwrite 0x03e450 thru 0x03ffff (0x1bb0 bytes) 000763 1786690618.871980 mapwrite 0x0186e3 thru 0x01b61d (0x2f3b bytes) 000764 1786690618.906151 mapwrite 0x018c52 thru 0x01ff6c (0x731b bytes) 000765 1786690618.936552 punch from 0x0319ff to 0x03846e (0x6a70 bytes) 000766 1786690618.945237 mapwrite 0x010621 thru 0x01725d (0x6c3d bytes) 000767 1786690618.966948 trunc from 0x03e738 to 0x03ffff (0x18c8 bytes) 000768 1786690618.973289 write 0x020ddf thru 0x022066 (0x1288 bytes) 000769 1786690618.977043 mapread 0x02a721 thru 0x02d56f (0x2e4f bytes) 000770 1786690618.978464 write 0x017048 thru 0x01d052 (0x600b bytes) 000771 1786690618.984760 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xebfb6124 (stale) Mirror 19: 0xebfb6124 (stale) Mirror 20: 0xebfb6124 (stale) Mirror 21: 0xebfb6124 (stale) Mirror 25: 0xebfb6124 (stale) Mirror 26: 0xebfb6124 (stale) Mirror 28: 0xebfb6124 (stale) Mirror 32: 0xebfb6124 (stale) Mirror 33: 0xebfb6124 (stale) Mirror 34: 0xbd36b1ff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe281279b Mirror 35: 0xe281279b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000772 1786690620.498742 read 0x022333 thru 0x0246b6 (0x2384 bytes) 000773 1786690620.530823 read 0x035068 thru 0x03ffff (0xaf98 bytes) 000774 1786690620.555426 mapwrite 0x017c67 thru 0x0221f4 (0xa58e bytes) 000775 1786690620.903750 write 0x0185ca thru 0x0269f7 (0xe42e bytes) 000776 1786690620.925754 mapread 0x0386a6 thru 0x03ffff (0x795a bytes) 000777 1786690620.993996 write 0x00a4ba thru 0x00a929 (0x470 bytes) 000778 1786690621.003213 punch from 0x023308 to 0x03202f (0xed28 bytes) 000779 1786690621.049182 read 0x00216d thru 0x005e5c (0x3cf0 bytes) 000780 1786690621.073018 MIRROR_RESYNC 12 mirrors 000782 1786690625.616611 trunc from 0x03a953 to 0x03ffff (0x56ad bytes) 000783 1786690625.660276 trunc from 0x03e44f to 0x03ffff (0x1bb1 bytes) 000784 1786690625.666548 trunc from 0x040000 to 0x003111 (0x3ceee bytes) 000785 1786690625.690707 trunc from 0x003112 to 0x00902d (0x5f1c bytes) 000786 1786690625.707079 trunc from 0x00732d to 0x00874c (0x1420 bytes) 000787 1786690625.713025 mapread 0x002ad4 thru 0x005c81 (0x31ae bytes) 000788 1786690625.732794 mapwrite 0x015789 thru 0x01e35e (0x8bd6 bytes) 000789 1786690625.783776 mapread 0x00efa7 thru 0x017294 (0x82ee bytes) 000790 1786690625.787122 punch from 0x0105d4 to 0x01a4fc (0x9f29 bytes) 000791 1786690625.794055 trunc from 0x006b7f to 0x00dfef (0x7471 bytes) 000792 1786690625.800062 trunc from 0x018f0f to 0x01e35e (0x5450 bytes) 000793 1786690625.806048 fallocate 0x00abd8 thru 0x0192dc (0xe705 bytes) 000794 1786690625.811281 trunc from 0x01e35f to 0x00cac2 (0x1189c bytes) 000795 1786690625.828423 write 0x03e6af thru 0x03ffff (0x1951 bytes) 000796 1786690625.834014 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x647aa3c1 (stale) Mirror 19: 0x647aa3c1 (stale) Mirror 20: 0x647aa3c1 (stale) Mirror 21: 0x647aa3c1 (stale) Mirror 25: 0x647aa3c1 (stale) Mirror 26: 0x647aa3c1 (stale) Mirror 28: 0x647aa3c1 (stale) Mirror 32: 0x647aa3c1 (stale) Mirror 33: 0x647aa3c1 (stale) Mirror 34: 0x647aa3c1 (stale) Mirror 35: 0x647aa3c1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xec1f857a Mirror 36: 0xec1f857a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000797 1786690627.278180 mapwrite 0x008888 thru 0x00a956 (0x20cf bytes) 000798 1786690627.422230 trunc from 0x040000 to 0x001b04 (0x3e4fb bytes) 000799 1786690627.469580 MIRROR_RESYNC 13 mirrors 000800 1786690629.275056 MIRROR_RESYNC 13 mirrors 000801 1786690629.846721 MIRROR_RESYNC 13 mirrors 000802 1786690630.271458 trunc from 0x001b05 to 0x030db6 (0x2f2b2 bytes) 000803 1786690630.351339 read 0x01b7f9 thru 0x01f660 (0x3e68 bytes) 000804 1786690630.364005 mapread 0x01d28b thru 0x029851 (0xc5c7 bytes) 000805 1786690630.386685 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x30db7): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30db7): Mirror 13: 0x5765ced7 (stale) Mirror 19: 0x5765ced7 (stale) Mirror 20: 0x5765ced7 (stale) Mirror 21: 0x5765ced7 (stale) Mirror 25: 0x5765ced7 (stale) Mirror 26: 0x5765ced7 (stale) Mirror 28: 0x5765ced7 (stale) Mirror 32: 0x5765ced7 (stale) Mirror 33: 0x5765ced7 (stale) Mirror 34: 0x5765ced7 (stale) Mirror 35: 0x5765ced7 (stale) Mirror 36: 0x5765ced7 (stale) CRC-32 checksum value for chunk [0, 0x30db7): Mirror 1: 0x5765ced7 Mirror 37: 0x5765ced7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30db7: skipped 000807 1786690631.736214 trunc from 0x00a6cf to 0x015d26 (0xb658 bytes) 000809 1786690631.841586 punch from 0x00eed9 to 0x01925f (0xa387 bytes) 000810 1786690631.848843 read 0x007688 thru 0x00dfae (0x6927 bytes) 000811 1786690631.883289 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x30db7): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30db7): Mirror 13: 0x5765ced7 (stale) Mirror 19: 0x5765ced7 (stale) Mirror 20: 0x5765ced7 (stale) Mirror 21: 0x5765ced7 (stale) Mirror 25: 0x5765ced7 (stale) Mirror 26: 0x5765ced7 (stale) Mirror 28: 0x5765ced7 (stale) Mirror 32: 0x5765ced7 (stale) Mirror 33: 0x5765ced7 (stale) Mirror 34: 0x5765ced7 (stale) Mirror 35: 0x5765ced7 (stale) Mirror 36: 0x5765ced7 (stale) Mirror 37: 0x5765ced7 (stale) CRC-32 checksum value for chunk [0, 0x30db7): Mirror 1: 0x5765ced7 Mirror 38: 0x5765ced7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30db7: skipped 000812 1786690632.980048 fallocate 0x02f439 thru 0x033619 (0x41e1 bytes) 000813 1786690633.070270 fallocate 0x037170 thru 0x03ffff (0x8e90 bytes) 000814 1786690633.080996 punch from 0x0228da to 0x030097 (0xd7be bytes) 000815 1786690633.111751 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x5de72620 (stale) Mirror 19: 0x5de72620 (stale) Mirror 21: 0x5de72620 (stale) Mirror 25: 0x5de72620 (stale) Mirror 26: 0x5de72620 (stale) Mirror 28: 0x5de72620 (stale) Mirror 32: 0x5de72620 (stale) Mirror 33: 0x5de72620 (stale) Mirror 34: 0x5de72620 (stale) Mirror 35: 0x5de72620 (stale) Mirror 36: 0x5de72620 (stale) Mirror 37: 0x5de72620 (stale) Mirror 38: 0x5de72620 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x5de72620 (stale) Mirror 19: 0x5de72620 (stale) Mirror 21: 0x5de72620 (stale) Mirror 25: 0x5de72620 (stale) Mirror 26: 0x5de72620 (stale) Mirror 28: 0x5de72620 (stale) Mirror 32: 0x5de72620 (stale) Mirror 33: 0x5de72620 (stale) Mirror 34: 0x5de72620 (stale) Mirror 35: 0x5de72620 (stale) Mirror 36: 0x5de72620 (stale) Mirror 37: 0x5de72620 (stale) Mirror 38: 0x5de72620 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000817 1786690633.814199 trunc from 0x040000 to 0x000479 (0x3fb86 bytes) 000818 1786690633.843665 fallocate 0x015021 thru 0x01a6f5 (0x56d5 bytes) 000820 1786690633.853018 trunc from 0x014789 to 0x01a6f5 (0x5f6d bytes) 000821 1786690633.865078 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a6f6): Mirror 13: 0x5869665a (stale) Mirror 19: 0x5869665a (stale) Mirror 21: 0x5869665a (stale) Mirror 25: 0x5869665a (stale) Mirror 26: 0x5869665a (stale) Mirror 28: 0x5869665a (stale) Mirror 32: 0x5869665a (stale) Mirror 33: 0x5869665a (stale) Mirror 34: 0x5869665a (stale) Mirror 35: 0x5869665a (stale) Mirror 37: 0x5869665a (stale) Mirror 38: 0x5869665a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a6f6): Mirror 13: 0x5869665a (stale) Mirror 19: 0x5869665a (stale) Mirror 21: 0x5869665a (stale) Mirror 25: 0x5869665a (stale) Mirror 26: 0x5869665a (stale) Mirror 28: 0x5869665a (stale) Mirror 32: 0x5869665a (stale) Mirror 33: 0x5869665a (stale) Mirror 34: 0x5869665a (stale) Mirror 35: 0x5869665a (stale) Mirror 37: 0x5869665a (stale) Mirror 38: 0x5869665a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a6f6: skipped 000822 1786690634.468269 mapread 0x00aeb3 thru 0x010de5 (0x5f33 bytes) 000824 1786690634.496601 write 0x03cd8f thru 0x03ffff (0x3271 bytes) 000825 1786690634.503045 punch from 0x025759 to 0x033c5b (0xe503 bytes) 000826 1786690634.508125 write 0x02070e thru 0x027032 (0x6925 bytes) 000827 1786690634.516596 fallocate 0x03234e thru 0x0392c4 (0x6f77 bytes) 000828 1786690634.524946 MIRROR_RESYNC 13 mirrors 000829 1786690637.651139 trunc from 0x021496 to 0x02a52d (0x9098 bytes) 000830 1786690637.719119 trunc from 0x040000 to 0x027d9e (0x18261 bytes) 000831 1786690637.760179 mapwrite 0x01c6b4 thru 0x026456 (0x9da3 bytes) 000832 1786690637.816020 trunc from 0x02221d to 0x027d9e (0x5b82 bytes) 000833 1786690637.829134 mapread 0x026995 thru 0x027d9e (0x140a bytes) 000834 1786690637.844145 trunc from 0x027d9f to 0x02e757 (0x69b9 bytes) 000835 1786690637.862810 trunc from 0x01a905 to 0x024b2b (0xa227 bytes) 000836 1786690637.868099 mapwrite 0x018735 thru 0x0272ba (0xeb86 bytes) 000837 1786690637.976155 mapwrite 0x01ca63 thru 0x01efac (0x254a bytes) 000838 1786690638.016728 trunc from 0x00b8ba to 0x016f82 (0xb6c9 bytes) 000839 1786690638.024950 trunc from 0x02e758 to 0x02d890 (0xec7 bytes) 000840 1786690638.071998 write 0x017297 thru 0x01f057 (0x7dc1 bytes) 000842 1786690638.081298 punch from 0x023f23 to 0x02d890 (0x996e bytes) 000844 1786690638.098931 mapwrite 0x00426a thru 0x0140b8 (0xfe4f bytes) 000845 1786690638.240825 mapread 0x02c9aa thru 0x02d890 (0xee7 bytes) 000846 1786690638.264015 fallocate 0x001ae4 thru 0x00a177 (0x8694 bytes) 000847 1786690638.274746 mapwrite 0x009154 thru 0x012fbc (0x9e69 bytes) 000848 1786690638.324155 fallocate 0x007687 thru 0x00b1e5 (0x3b5f bytes) 000849 1786690638.336309 punch from 0x00d4f5 to 0x01407d (0x6b89 bytes) 000850 1786690638.353966 write 0x002a31 thru 0x00c481 (0x9a51 bytes) 000851 1786690638.355450 read 0x009280 thru 0x01665f (0xd3e0 bytes) 000852 1786690638.373068 mapread 0x02af7e thru 0x02d890 (0x2913 bytes) 000853 1786690638.374218 read 0x00a4a2 thru 0x00eb44 (0x46a3 bytes) 000854 1786690638.377903 trunc from 0x01b46d to 0x01ec0b (0x379f bytes) 000855 1786690638.415922 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d891): Mirror 13: 0x26aa226a (stale) Mirror 19: 0x26aa226a (stale) Mirror 21: 0x26aa226a (stale) Mirror 25: 0x26aa226a (stale) Mirror 26: 0x26aa226a (stale) Mirror 28: 0x26aa226a (stale) Mirror 32: 0x26aa226a (stale) Mirror 33: 0x26aa226a (stale) Mirror 34: 0x26aa226a (stale) Mirror 35: 0x26aa226a (stale) Mirror 37: 0x26aa226a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d891): Mirror 13: 0x26aa226a (stale) Mirror 19: 0x26aa226a (stale) Mirror 21: 0x26aa226a (stale) Mirror 25: 0x26aa226a (stale) Mirror 26: 0x26aa226a (stale) Mirror 28: 0x26aa226a (stale) Mirror 32: 0x26aa226a (stale) Mirror 33: 0x26aa226a (stale) Mirror 34: 0x26aa226a (stale) Mirror 35: 0x26aa226a (stale) Mirror 37: 0x26aa226a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d891: skipped 000856 1786690639.251860 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d891): Mirror 13: 0x26aa226a (stale) Mirror 19: 0x26aa226a (stale) Mirror 25: 0x26aa226a (stale) Mirror 26: 0x26aa226a (stale) Mirror 28: 0x26aa226a (stale) Mirror 32: 0x26aa226a (stale) Mirror 33: 0x26aa226a (stale) Mirror 34: 0x26aa226a (stale) Mirror 35: 0x26aa226a (stale) Mirror 37: 0x26aa226a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d891): Mirror 13: 0x26aa226a (stale) Mirror 19: 0x26aa226a (stale) Mirror 25: 0x26aa226a (stale) Mirror 26: 0x26aa226a (stale) Mirror 28: 0x26aa226a (stale) Mirror 32: 0x26aa226a (stale) Mirror 33: 0x26aa226a (stale) Mirror 34: 0x26aa226a (stale) Mirror 35: 0x26aa226a (stale) Mirror 37: 0x26aa226a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d891: skipped 000857 1786690640.056084 trunc from 0x025faf to 0x02943f (0x3491 bytes) 000858 1786690640.072111 MIRROR_RESYNC 11 mirrors 000859 1786690644.248318 write 0x037a54 thru 0x03f412 (0x79bf bytes) 000860 1786690644.296148 MIRROR_RESYNC 11 mirrors 000861 1786690649.021625 mapwrite 0x007521 thru 0x00b193 (0x3c73 bytes) 000862 1786690649.119878 fallocate 0x02068c thru 0x022b1c (0x2491 bytes) 000863 1786690649.128496 punch from 0x02daa6 to 0x035502 (0x7a5d bytes) 000864 1786690649.140476 write 0x01e89c thru 0x027c6c (0x93d1 bytes) 000865 1786690649.155063 trunc from 0x03f413 to 0x036ec1 (0x8551 bytes) 000866 1786690649.180469 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x36ec2): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ec2): Mirror 13: 0xe3fa37a0 (stale) Mirror 19: 0xe3fa37a0 (stale) Mirror 25: 0xe3fa37a0 (stale) Mirror 26: 0xe3fa37a0 (stale) Mirror 28: 0xe3fa37a0 (stale) Mirror 32: 0xe3fa37a0 (stale) Mirror 33: 0xe3fa37a0 (stale) Mirror 34: 0xe3fa37a0 (stale) Mirror 35: 0xe3fa37a0 (stale) Mirror 37: 0xe3fa37a0 (stale) CRC-32 checksum value for chunk [0, 0x36ec2): Mirror 1: 0xc7f8cd9 Mirror 38: 0xc7f8cd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ec2: skipped 000869 1786690650.670277 fallocate 0x000d7d thru 0x00d39b (0xc61f bytes) 000870 1786690650.732029 trunc from 0x02797a to 0x02fbdc (0x8263 bytes) 000871 1786690650.737860 fallocate 0x001ea5 thru 0x004cbe (0x2e1a bytes) 000872 1786690650.744194 trunc from 0x036ec2 to 0x028c4e (0xe273 bytes) 000874 1786690650.768190 trunc from 0x028c4f to 0x02ce05 (0x41b7 bytes) 000875 1786690650.786266 mapread 0x00c896 thru 0x0186db (0xbe46 bytes) 000876 1786690650.825604 trunc from 0x02ce06 to 0x003eb4 (0x28f51 bytes) 000877 1786690650.842957 trunc from 0x00018d to 0x00271f (0x2593 bytes) 000878 1786690650.847903 read 0x001cc4 thru 0x003eb4 (0x21f1 bytes) 000880 1786690650.856887 trunc from 0x003eb5 to 0x00b22a (0x7376 bytes) 000881 1786690650.876208 read 0x0018a6 thru 0x00a3e1 (0x8b3c bytes) 000882 1786690650.889013 trunc from 0x004789 to 0x00b22a (0x6aa2 bytes) 000883 1786690650.895113 mapwrite 0x014795 thru 0x021263 (0xcacf bytes) 000884 1786690650.946529 trunc from 0x007821 to 0x00aced (0x34cd bytes) 000885 1786690650.951756 mapread 0x01b1fb thru 0x021263 (0x6069 bytes) 000886 1786690650.953822 mapread 0x00ca5f thru 0x01bbfd (0xf19f bytes) 000887 1786690650.958933 write 0x018ec7 thru 0x01c52e (0x3668 bytes) 000888 1786690650.964058 mapwrite 0x032795 thru 0x03e176 (0xb9e2 bytes) 000889 1786690651.104868 fallocate 0x02b520 thru 0x034f8e (0x9a6f bytes) 000890 1786690651.123798 mapwrite 0x02e2e7 thru 0x034867 (0x6581 bytes) 000891 1786690651.165779 punch from 0x0217d9 to 0x02e0be (0xc8e6 bytes) 000892 1786690651.176462 mapread 0x00d438 thru 0x00e8dd (0x14a6 bytes) 000893 1786690651.177667 trunc from 0x024cb2 to 0x028191 (0x34e0 bytes) 000894 1786690651.191085 mapread 0x01f77e thru 0x023c51 (0x44d4 bytes) 000895 1786690651.224034 write 0x023057 thru 0x03243d (0xf3e7 bytes) 000896 1786690651.229300 trunc from 0x03e177 to 0x02c978 (0x117fe bytes) 000897 1786690651.273200 mapwrite 0x006d05 thru 0x007311 (0x60d bytes) 000898 1786690651.331136 trunc from 0x02c979 to 0x0061b2 (0x267c6 bytes) 000899 1786690651.350580 mapwrite 0x01c6b6 thru 0x01d6fc (0x1047 bytes) 000900 1786690651.433947 mapwrite 0x0350f4 thru 0x0386b0 (0x35bd bytes) 000901 1786690651.580004 read 0x03085d thru 0x036587 (0x5d2b bytes) 000902 1786690651.621497 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x386b1): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x386b1): Mirror 13: 0x9188f39a (stale) Mirror 19: 0x9188f39a (stale) Mirror 25: 0x9188f39a (stale) Mirror 26: 0x9188f39a (stale) Mirror 28: 0x9188f39a (stale) Mirror 32: 0x9188f39a (stale) Mirror 33: 0x9188f39a (stale) Mirror 34: 0x9188f39a (stale) Mirror 35: 0x9188f39a (stale) Mirror 37: 0x9188f39a (stale) Mirror 38: 0x304545e1 (stale) CRC-32 checksum value for chunk [0, 0x386b1): Mirror 1: 0xcdbb346c Mirror 39: 0xcdbb346c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x386b1: skipped 000904 1786690653.386060 trunc from 0x021d26 to 0x02bd7f (0xa05a bytes) 000905 1786690653.487441 punch from 0x001254 to 0x002cb7 (0x1a64 bytes) 000907 1786690653.496322 mapread 0x028036 thru 0x035ed3 (0xde9e bytes) 000908 1786690653.563734 read 0x033f71 thru 0x0386b0 (0x4740 bytes) 000910 1786690653.567937 read 0x008732 thru 0x016810 (0xe0df bytes) 000911 1786690653.585224 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x386b1): Mirror 13: 0x9188f39a (stale) Mirror 19: 0x9188f39a (stale) Mirror 25: 0x9188f39a (stale) Mirror 26: 0x9188f39a (stale) Mirror 32: 0x9188f39a (stale) Mirror 33: 0x9188f39a (stale) Mirror 34: 0x9188f39a (stale) Mirror 35: 0x9188f39a (stale) Mirror 37: 0x9188f39a (stale) Mirror 38: 0x304545e1 (stale) Mirror 39: 0xcdbb346c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x386b1): Mirror 13: 0x9188f39a (stale) Mirror 19: 0x9188f39a (stale) Mirror 25: 0x9188f39a (stale) Mirror 26: 0x9188f39a (stale) Mirror 32: 0x9188f39a (stale) Mirror 33: 0x9188f39a (stale) Mirror 34: 0x9188f39a (stale) Mirror 35: 0x9188f39a (stale) Mirror 37: 0x9188f39a (stale) Mirror 38: 0x304545e1 (stale) Mirror 39: 0xcdbb346c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x386b1: skipped 000912 1786690654.622395 punch from 0x003647 to 0x004898 (0x1252 bytes) 000913 1786690654.633648 mapwrite 0x0268b9 thru 0x031220 (0xa968 bytes) 000914 1786690654.691908 mapwrite 0x026368 thru 0x033133 (0xcdcc bytes) 000915 1786690654.721497 mapread 0x0286de thru 0x0328f3 (0xa216 bytes) 000916 1786690654.723732 mapwrite 0x0125aa thru 0x014767 (0x21be bytes) 000917 1786690654.745830 trunc from 0x01b09c to 0x022b56 (0x7abb bytes) 000918 1786690654.752161 fallocate 0x029d6d thru 0x02ad4b (0xfdf bytes) 000920 1786690654.758810 punch from 0x02ca55 to 0x0386b0 (0xbc5c bytes) 000921 1786690654.769490 MIRROR_RESYNC 12 mirrors 000923 1786690659.831763 fallocate 0x0019c6 thru 0x00cac2 (0xb0fd bytes) 000924 1786690659.875736 mapwrite 0x00b920 thru 0x018487 (0xcb68 bytes) 000925 1786690659.959989 fallocate 0x02d999 thru 0x031946 (0x3fae bytes) 000926 1786690659.968085 fallocate 0x018d96 thru 0x01c5da (0x3845 bytes) 000927 1786690659.980939 mapread 0x0383d9 thru 0x038649 (0x271 bytes) 000928 1786690660.003930 trunc from 0x00c23c to 0x01047e (0x4243 bytes) 000929 1786690660.009948 write 0x0331f1 thru 0x03f284 (0xc094 bytes) 000930 1786690660.027916 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3f285): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f285): Mirror 13: 0x38b9ef04 (stale) Mirror 19: 0x38b9ef04 (stale) Mirror 25: 0x38b9ef04 (stale) Mirror 26: 0x38b9ef04 (stale) Mirror 32: 0x38b9ef04 (stale) Mirror 33: 0x38b9ef04 (stale) Mirror 34: 0x38b9ef04 (stale) Mirror 35: 0x38b9ef04 (stale) Mirror 37: 0x38b9ef04 (stale) Mirror 38: 0x38b9ef04 (stale) Mirror 39: 0x38b9ef04 (stale) CRC-32 checksum value for chunk [0, 0x3f285): Mirror 1: 0x2da1471 Mirror 40: 0x2da1471 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f285: skipped 000931 1786690661.928884 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f285): Mirror 13: 0x38b9ef04 (stale) Mirror 19: 0x38b9ef04 (stale) Mirror 25: 0x38b9ef04 (stale) Mirror 26: 0x38b9ef04 (stale) Mirror 32: 0x38b9ef04 (stale) Mirror 33: 0x38b9ef04 (stale) Mirror 34: 0x38b9ef04 (stale) Mirror 35: 0x38b9ef04 (stale) Mirror 37: 0x38b9ef04 (stale) Mirror 38: 0x38b9ef04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f285): Mirror 13: 0x38b9ef04 (stale) Mirror 19: 0x38b9ef04 (stale) Mirror 25: 0x38b9ef04 (stale) Mirror 26: 0x38b9ef04 (stale) Mirror 32: 0x38b9ef04 (stale) Mirror 33: 0x38b9ef04 (stale) Mirror 34: 0x38b9ef04 (stale) Mirror 35: 0x38b9ef04 (stale) Mirror 37: 0x38b9ef04 (stale) Mirror 38: 0x38b9ef04 (stale) CRC-32 checksum value for chunk [0, 0x3f285): Mirror 1: 0x2da1471 Mirror 40: 0x2da1471 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f285: skipped 000932 1786690663.120787 fallocate 0x037443 thru 0x03a506 (0x30c4 bytes) 000933 1786690663.192153 read 0x03b289 thru 0x03f284 (0x3ffc bytes) 000934 1786690663.203612 mapwrite 0x03d00b thru 0x03de5e (0xe54 bytes) 000935 1786690663.216626 fallocate 0x005b21 thru 0x00f690 (0x9b70 bytes) 000936 1786690663.222539 trunc from 0x01181a to 0x017854 (0x603b bytes) 000938 1786690663.229224 read 0x0351c0 thru 0x0396c1 (0x4502 bytes) 000939 1786690663.240890 punch from 0x03489a to 0x0355cf (0xd36 bytes) 000940 1786690663.246628 trunc from 0x03f285 to 0x00ba39 (0x3384b bytes) 000941 1786690663.280268 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xba3a): Mirror 13: 0x4817c0a9 (stale) Mirror 19: 0x4817c0a9 (stale) Mirror 25: 0x4817c0a9 (stale) Mirror 26: 0x4817c0a9 (stale) Mirror 32: 0x4817c0a9 (stale) Mirror 33: 0x4817c0a9 (stale) Mirror 34: 0x4817c0a9 (stale) Mirror 35: 0x4817c0a9 (stale) Mirror 37: 0x4817c0a9 (stale) Mirror 40: 0x21f49677 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xba3a): Mirror 13: 0x4817c0a9 (stale) Mirror 19: 0x4817c0a9 (stale) Mirror 25: 0x4817c0a9 (stale) Mirror 26: 0x4817c0a9 (stale) Mirror 32: 0x4817c0a9 (stale) Mirror 33: 0x4817c0a9 (stale) Mirror 34: 0x4817c0a9 (stale) Mirror 35: 0x4817c0a9 (stale) Mirror 37: 0x4817c0a9 (stale) Mirror 40: 0x21f49677 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xba3a: skipped 000942 1786690663.766803 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xba3a): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 37: (stale - read error) Mirror 40: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xba3a): Mirror 13: 0x4817c0a9 (stale) Mirror 19: 0x4817c0a9 (stale) Mirror 25: 0x4817c0a9 (stale) Mirror 26: 0x4817c0a9 (stale) Mirror 32: 0x4817c0a9 (stale) Mirror 33: 0x4817c0a9 (stale) Mirror 34: 0x4817c0a9 (stale) Mirror 35: 0x4817c0a9 (stale) Mirror 37: 0x4817c0a9 (stale) Mirror 40: 0x21f49677 (stale) CRC-32 checksum value for chunk [0, 0xba3a): Mirror 1: 0x21f49677 Mirror 41: 0x21f49677 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xba3a: skipped 000943 1786690665.169129 trunc from 0x0014f1 to 0x0091ad (0x7cbd bytes) 000944 1786690665.215837 mapwrite 0x021b41 thru 0x02afb0 (0x9470 bytes) 000945 1786690665.326224 mapwrite 0x016b22 thru 0x017a21 (0xf00 bytes) 000946 1786690665.356115 mapread 0x0246c6 thru 0x02afb0 (0x68eb bytes) 000947 1786690665.358917 write 0x01de2e thru 0x021a76 (0x3c49 bytes) 000948 1786690665.361710 mapwrite 0x0105f1 thru 0x01c599 (0xbfa9 bytes) 000949 1786690665.398100 MIRROR_RESYNC 12 mirrors 000950 1786690668.335718 trunc from 0x0261b0 to 0x02afb0 (0x4e01 bytes) 000951 1786690668.380165 trunc from 0x02afb1 to 0x033a0a (0x8a5a bytes) 000952 1786690668.401136 mapread 0x006b73 thru 0x013afc (0xcf8a bytes) 000953 1786690668.444555 mapwrite 0x03aa14 thru 0x03e39d (0x398a bytes) 000955 1786690668.507977 mapwrite 0x02268b thru 0x02abee (0x8564 bytes) 000956 1786690668.555405 read 0x03812b thru 0x03e39d (0x6273 bytes) 000957 1786690668.556092 mapread 0x017920 thru 0x01e024 (0x6705 bytes) 000959 1786690668.568678 trunc from 0x02f98f to 0x030c61 (0x12d3 bytes) 000960 1786690668.578842 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e39e): Mirror 13: 0x62ae2026 (stale) Mirror 19: 0x62ae2026 (stale) Mirror 25: 0x62ae2026 (stale) Mirror 26: 0x62ae2026 (stale) Mirror 32: 0x62ae2026 (stale) Mirror 33: 0x62ae2026 (stale) Mirror 34: 0x62ae2026 (stale) Mirror 35: 0x62ae2026 (stale) Mirror 37: 0x62ae2026 (stale) Mirror 41: 0x62ae2026 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e39e): Mirror 13: 0x62ae2026 (stale) Mirror 19: 0x62ae2026 (stale) Mirror 25: 0x62ae2026 (stale) Mirror 26: 0x62ae2026 (stale) Mirror 32: 0x62ae2026 (stale) Mirror 33: 0x62ae2026 (stale) Mirror 34: 0x62ae2026 (stale) Mirror 35: 0x62ae2026 (stale) Mirror 37: 0x62ae2026 (stale) Mirror 41: 0x62ae2026 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e39e: skipped 000961 1786690669.406059 mapwrite 0x01dc9d thru 0x023fc7 (0x632b bytes) 000962 1786690669.455605 write 0x03b27e thru 0x03d9f8 (0x277b bytes) 000963 1786690669.498456 punch from 0x00fcd3 to 0x013cc0 (0x3fee bytes) 000964 1786690669.511015 mapread 0x00783b thru 0x009a14 (0x21da bytes) 000966 1786690669.529418 fallocate 0x02aef6 thru 0x032e52 (0x7f5d bytes) 000967 1786690669.534757 trunc from 0x004b0b to 0x007fc9 (0x34bf bytes) 000968 1786690669.539036 mapread 0x03a688 thru 0x03e39d (0x3d16 bytes) 000969 1786690669.548137 punch from 0x00bed2 to 0x00dd4b (0x1e7a bytes) 000970 1786690669.552798 punch from 0x008cfb to 0x012b58 (0x9e5e bytes) 000971 1786690669.557896 read 0x02ccac thru 0x02ef76 (0x22cb bytes) 000972 1786690669.563496 write 0x03450c thru 0x03ffff (0xbaf4 bytes) 000973 1786690669.570582 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xda30263b (stale) Mirror 19: 0xda30263b (stale) Mirror 25: 0xda30263b (stale) Mirror 32: 0xda30263b (stale) Mirror 33: 0xda30263b (stale) Mirror 34: 0xda30263b (stale) Mirror 35: 0xda30263b (stale) Mirror 37: 0xda30263b (stale) Mirror 41: 0xda30263b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xda30263b (stale) Mirror 19: 0xda30263b (stale) Mirror 25: 0xda30263b (stale) Mirror 32: 0xda30263b (stale) Mirror 33: 0xda30263b (stale) Mirror 34: 0xda30263b (stale) Mirror 35: 0xda30263b (stale) Mirror 37: 0xda30263b (stale) Mirror 41: 0xda30263b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000975 1786690670.265321 mapwrite 0x00a2ec thru 0x0128a3 (0x85b8 bytes) 000976 1786690670.307716 mapread 0x03c904 thru 0x03ffff (0x36fc bytes) 000977 1786690670.320833 mapwrite 0x02cd99 thru 0x0323bf (0x5627 bytes) 000978 1786690670.357966 fallocate 0x01d42d thru 0x02a846 (0xd41a bytes) 000979 1786690670.367983 read 0x0363c8 thru 0x039150 (0x2d89 bytes) 000980 1786690670.382434 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 25: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 37: (stale - read error) Mirror 41: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xda30263b (stale) Mirror 19: 0xda30263b (stale) Mirror 25: 0xda30263b (stale) Mirror 32: 0xda30263b (stale) Mirror 33: 0xda30263b (stale) Mirror 34: 0xda30263b (stale) Mirror 35: 0xda30263b (stale) Mirror 37: 0xda30263b (stale) Mirror 41: 0xda30263b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x869ff41a Mirror 42: 0x869ff41a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000981 1786690672.022767 mapread 0x019ff3 thru 0x02764f (0xd65d bytes) 000982 1786690672.075466 write 0x014814 thru 0x01a35d (0x5b4a bytes) 000984 1786690672.134146 mapread 0x0390b6 thru 0x03ffff (0x6f4a bytes) 000985 1786690672.144899 read 0x009452 thru 0x01018f (0x6d3e bytes) 000986 1786690672.155840 mapwrite 0x0078e6 thru 0x01399f (0xc0ba bytes) 000987 1786690672.208564 trunc from 0x040000 to 0x021bca (0x1e435 bytes) 000988 1786690672.234858 trunc from 0x021bcb to 0x02ef92 (0xd3c8 bytes) 000989 1786690672.249418 write 0x0370f2 thru 0x03af15 (0x3e24 bytes) 000990 1786690672.253408 mapwrite 0x00a661 thru 0x0187c8 (0xe168 bytes) 000991 1786690672.281611 fallocate 0x01f4b2 thru 0x02d348 (0xde97 bytes) 000992 1786690672.287051 read 0x039b16 thru 0x03a052 (0x53d bytes) 000993 1786690672.290715 trunc from 0x03af16 to 0x009c56 (0x312bf bytes) 000994 1786690672.316758 write 0x03af97 thru 0x03ed4d (0x3db7 bytes) 000995 1786690672.323539 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ed4e): Mirror 13: 0xa2a2a377 (stale) Mirror 25: 0xa2a2a377 (stale) Mirror 32: 0xa2a2a377 (stale) Mirror 33: 0xa2a2a377 (stale) Mirror 34: 0xa2a2a377 (stale) Mirror 35: 0xa2a2a377 (stale) Mirror 37: 0xa2a2a377 (stale) Mirror 41: 0xa2a2a377 (stale) Mirror 42: 0xec8a617f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed4e): Mirror 13: 0xa2a2a377 (stale) Mirror 25: 0xa2a2a377 (stale) Mirror 32: 0xa2a2a377 (stale) Mirror 33: 0xa2a2a377 (stale) Mirror 34: 0xa2a2a377 (stale) Mirror 35: 0xa2a2a377 (stale) Mirror 37: 0xa2a2a377 (stale) Mirror 41: 0xa2a2a377 (stale) Mirror 42: 0xec8a617f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed4e: skipped 000996 1786690673.086172 read 0x008f43 thru 0x017546 (0xe604 bytes) 000997 1786690673.103062 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ed4e): Mirror 13: (stale - read error) Mirror 25: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 37: (stale - read error) Mirror 41: (stale - read error) Mirror 42: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ed4e): Mirror 13: 0xa2a2a377 (stale) Mirror 25: 0xa2a2a377 (stale) Mirror 32: 0xa2a2a377 (stale) Mirror 33: 0xa2a2a377 (stale) Mirror 34: 0xa2a2a377 (stale) Mirror 35: 0xa2a2a377 (stale) Mirror 37: 0xa2a2a377 (stale) Mirror 41: 0xa2a2a377 (stale) Mirror 42: 0xec8a617f (stale) CRC-32 checksum value for chunk [0, 0x3ed4e): Mirror 1: 0xa81fc847 Mirror 43: 0xa81fc847 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ed4e: skipped 000999 1786690674.290841 read 0x00c7cf thru 0x01c124 (0xf956 bytes) 001000 1786690674.311669 fallocate 0x00d92b thru 0x014799 (0x6e6f bytes) All operations completed A-OK! PASS 70a (195s) == sanity-flr test 71: check mirror extend parent fid ==== 02:57:59 (1786690679) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x71:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 02:58:10 (1786690690) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.249887 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144358 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ce:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 02:58:19 (1786690699) Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre2 Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1070656 .. Extending file size to 7921888 .. done Extending file size to 4996128 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9162208 .. Extending file size to 5214176 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7009632 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8254464 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8465696 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61431 61432 61433 61434 61435 done 192.168.206.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg604-client.virtnet /mnt/lustre2 (opts:) 192.168.206.104@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg604-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:59:38 (1786690778) Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre2 Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8754784 .. Extending file size to 5761408 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8789568 .. Extending file size to 7731136 .. Extending file size to 8537760 .. Extending file size to 8946688 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8983680 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62982 62983 62984 62985 62988 done 192.168.206.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg604-client.virtnet /mnt/lustre2 (opts:) 192.168.206.104@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg604-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:00:53 (1786690853) Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1268864 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2212 1269092 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18728 3588292 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24348 3582672 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43076 7170964 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1268864 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2212 1269092 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18728 3588292 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24348 3582672 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43076 7170964 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0744938 s, 28.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.38623 s, 5.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb30fc2c4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg604-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:01:13 (1786690873) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121563 s, 17.3 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 03:01:22 (1786690882) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1bb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x306:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:01:30 (1786690890) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115411 s, 18.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:01:38 (1786690898) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0838111 s, 25.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x307:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x307:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x308:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:01:44 (1786690904) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0960862 s, 21.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5f:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:02:15 (1786690935) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg604-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726346 s, 14.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x60:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x61:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x60:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5d:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:03:00 (1786690980) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg604-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.090475 s, 23.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:03:47 (1786691027) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg604-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg604-server: Pool lustre.test_204f created Adding targets to pool oleg604-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg604-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106053 s, 19.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg604-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg604-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg604-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:04:57 (1786691097) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:05:03 (1786691103) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:05:10 (1786691110) oleg604-server: Pool lustre.test_206 created Adding targets to pool oleg604-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg604-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg604-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg604-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg604-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:05:37 (1786691137) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457257 s, 22.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439113 s, 23.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdf6317b7 Mirror 3: 0xdf6317b7 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:05:46 (1786691146) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:05:49 (1786691149) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:05:52 (1786691152) Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0859893 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xf80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2230000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc80000 Read this (4 bytes): BBBB 192.168.206.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg604-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:06:26 (1786691186) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0329811 s, 248 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:06:32 (1786691192) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054517 s, 19.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1da lmm_fid: [0x200000405:0x1da:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1da lmm_fid: [0x200000405:0x1da:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x315:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611610 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 03:06:33.000000000 -0400 Modify: 2026-08-14 03:06:34.000000000 -0400 Change: 2026-08-14 03:06:35.000000000 -0400 Birth: 2026-08-14 03:06:33.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:06:39 (1786691199) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033137 s, 31.6 MB/s oleg604-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg604-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg604-server: O/280000401/d21/2f5: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:07:00 (1786691220) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.230367 s, 178 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0191871 s, 213 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 03:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1df:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 03:07 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:07:09 (1786691229) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:07:16 (1786691236) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1843 sec =========== 03:07:25 (1786691245) === sanity-flr: start cleanup 03:07:27 (1786691247) === === sanity-flr: finish cleanup 03:07:36 (1786691256) ===