-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 21:30:32 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg218-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:30:52 (1781227852) === oleg218-client.virtnet: executing check_config_client /mnt/lustre oleg218-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg218-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98ee6107f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98ee6107f800.idle_timeout=debug disable quota as required oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:31:28 (1781227888) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8209] == sanity-flr test 0a: lfs mirror create with -N option == 21:31:30 (1781227890) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:31:44 (1781227904) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:31:47 (1781227907) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:31:50 (1781227910) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:32:13 (1781227933) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:32:17 (1781227937) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 21:32:20 (1781227940) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98ee6107f800.stats=clear osc.lustre-OST0001-osc-ffff98ee6107f800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:32:27 (1781227947) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:32:36 (1781227956) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:32:43 (1781227963) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:32:47 (1781227967) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:32:54 (1781227974) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 21:33:03 (1781227983) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:33:10 (1781227990) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:33:18 (1781227998) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:33:27 (1781228007) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:33:37 (1781228017) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:33:45 (1781228025) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.192961 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949347 s, 11.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:33:55 (1781228035) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0959506 s, 21.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:34:03 (1781228043) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:34:14 (1781228054) Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:35:33 (1781228133) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.333601 s, 15.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:36:17 (1781228177) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:37:36 (1781228256) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.24168 s, 16.2 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 21:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.058s user 0m0.001s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98ee6107f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.485s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg218-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg218-server oleg218-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.109s user 0m0.000s sys 0m0.172s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg218-server: oleg218-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg218-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg218-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98ee6107f800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:39:05 (1781228345) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:39:08 (1781228348) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:39:11 (1781228351) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:39:15 (1781228355) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 21:39:24 (1781228364) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.636761 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:39:37 (1781228377) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331602 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271862 s, 38.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:39:44 (1781228384) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:39:52 (1781228392) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000707304 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000791022 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0135457 s, 75.6 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00324708 s, 315 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 21:40:02 (1781228402) create test files with size 15760811 .. a2ee654a0d4ca53db2be3dd11e8eb8a9 - e8c5bea60db3ee9e0e4ac629e6640a45 - 58baac4cdc81cb150af63cf7c8e12000 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 21:40:23 (1781228423) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:40:36 (1781228436) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:40:45 (1781228445) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120666 s, 8.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243421 s, 43.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 21:40:55 (1781228455) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11405 s, 18.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.367253 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205404 s, 10.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 21:41:11 (1781228471) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 21:41:14 (1781228474) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:41:18 (1781228478) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594974 s, 17.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 21:41:26 (1781228486) llite.lustre-ffff98ee5049f000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 44a: lfs mirror split check =========== 21:41:34 (1781228494) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.358734 s, 8.8 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:41:45 (1781228505) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:41:56 (1781228516) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.22528 s, 46.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:42:10 (1781228530) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff98ee5e1a9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.380339 s, 27.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg218-client.virtnet /mnt/lustre (opts:) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff98ee46465800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:42:27 (1781228547) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:43:07 (1781228587) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:43:48 (1781228628) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 21:43:51 (1781228631) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.213332 s, 14.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.185883 s, 16.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x632d240a (nosync,stale) Mirror 2: 0x632d240a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 39f37c0d74dc47f4ae0fe9d7165d4ce9 - mirror 1 checksum: 47d2eed790ef00442a349b6546e61e9d - mirror 2 checksum: 39f37c0d74dc47f4ae0fe9d7165d4ce9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:43:53 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:43:53 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 39f37c0d74dc47f4ae0fe9d7165d4ce9 - mirror 1 checksum: 47d2eed790ef00442a349b6546e61e9d - mirror 2 checksum: 47d2eed790ef00442a349b6546e61e9d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:43:53 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:43:53 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd8:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:44:01 (1781228641) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196921 s, 53.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:44:08 (1781228648) Failing mds1 on oleg218-server Stopping /mnt/lustre-mds1 (opts:) on oleg218-server 21:44:20 (1781228660) shut down facet: mds1 facet_host: oleg218-server facet_failover_host: oleg218-server Failover mds1 to oleg218-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 21:44:43 (1781228683) targets are mounted 21:44:43 (1781228683) facet_failover done oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393651 s, 26.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xda:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:44:58 (1781228698) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.028231 s, 725 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0111111 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:45:08 (1781228708) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.037736 s, 543 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00812772 s, 2.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:45:16 (1781228716) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00978609 s, 1.7 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:45:25 (1781228725) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0278951 s, 734 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:45:33 (1781228733) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0071456 s, 573 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0215757 s, 190 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:45:42 (1781228742) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:46:01 (1781228761) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:46:30 (1781228790) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:46:56 (1781228816) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:47:22 (1781228842) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 21:47:31 (1781228851) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57776 mirror split pid 57777 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57776 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57777 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 21:48:39 (1781228919) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 8922 truncating to largest ever: 0x19f7e 000001 1781228922.622113 trunc from 00000000 to 0x019f7d (0x19f7e bytes) 000002 1781228922.645845 read 0x012298 thru 0x019f7d (0x7ce6 bytes) truncating to largest ever: 0x1b2b5 000003 1781228922.661091 trunc from 0x019f7e to 0x01b2b4 (0x1337 bytes) 000004 1781228922.681814 punch from 0x0075fe to 0x014715 (0xd118 bytes) 000008 1781228922.700250 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3d5f6 000009 1781228923.511201 trunc from 0x01b2b5 to 0x03d5f5 (0x22341 bytes) 000010 1781228923.559888 read 0x01ef0d thru 0x02158a (0x267e bytes) 000011 1781228923.570739 punch from 0x01238c to 0x013bd5 (0x184a bytes) 000012 1781228923.576113 mapread 0x0061f3 thru 0x00a423 (0x4231 bytes) 000013 1781228923.597288 punch from 0x00587b to 0x00cd30 (0x74b6 bytes) 000014 1781228923.602776 fallocate 0x000320 thru 0x008c95 (0x8976 bytes) 000015 1781228923.608753 trunc from 0x03d5f6 to 0x02fd9e (0xd857 bytes) fallocating to largest ever: 0x40000 000016 1781228923.643750 fallocate 0x03f586 thru 0x03ffff (0xa7a bytes) 000017 1781228923.651169 trunc from 0x040000 to 0x00a3c2 (0x35c3d bytes) 000018 1781228923.690149 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000019 1781228923.934956 fallocate 0x0210d0 thru 0x023656 (0x2587 bytes) 000020 1781228923.949189 punch from 0x00db75 to 0x00f789 (0x1c15 bytes) 000023 1781228923.981689 trunc from 0x023657 to 0x0025ae (0x210a8 bytes) 000024 1781228924.011264 mapread 0x0003db thru 0x0025ae (0x21d4 bytes) 000025 1781228924.035449 trunc from 0x00151f to 0x0025ae (0x1090 bytes) 000026 1781228924.042956 trunc from 0x0025af to 0x01741f (0x14e71 bytes) 000027 1781228924.085758 punch from 0x00af09 to 0x00d8c1 (0x29b9 bytes) 000028 1781228924.091378 punch from 0x011f72 to 0x015af6 (0x3b85 bytes) 000029 1781228924.103368 write 0x006833 thru 0x00dafa (0x72c8 bytes) 000031 1781228924.122206 write 0x003183 thru 0x00e84c (0xb6ca bytes) 000032 1781228924.123753 mapread 0x00f57a thru 0x01741f (0x7ea6 bytes) 000033 1781228924.140712 trunc from 0x015a68 to 0x01741f (0x19b8 bytes) 000034 1781228924.154358 mapwrite 0x008bb1 thru 0x0090df (0x52f bytes) 000035 1781228924.203878 read 0x00da82 thru 0x01741f (0x999e bytes) 000036 1781228924.215101 mapread 0x00dd36 thru 0x00fa02 (0x1ccd bytes) 000037 1781228924.224810 mapwrite 0x008714 thru 0x01556d (0xce5a bytes) 000038 1781228924.249095 trunc from 0x017420 to 0x018c45 (0x1826 bytes) 000039 1781228924.264378 read 0x0152f8 thru 0x018c45 (0x394e bytes) 000041 1781228924.270497 trunc from 0x018c46 to 0x0253a4 (0xc75f bytes) 000042 1781228924.284211 fallocate 0x0298f2 thru 0x02f3c7 (0x5ad6 bytes) 000043 1781228924.289909 write 0x01d3cb thru 0x02cffd (0xfc33 bytes) 000044 1781228924.308245 mapwrite 0x02fa7e thru 0x033f02 (0x4485 bytes) 000045 1781228924.368222 write 0x03d0af thru 0x03ffff (0x2f51 bytes) 000046 1781228924.370288 write 0x03d5a3 thru 0x03ffff (0x2a5d bytes) 000047 1781228924.373768 mapwrite 0x009c4f thru 0x016a51 (0xce03 bytes) 000048 1781228924.421482 read 0x032415 thru 0x032f4c (0xb38 bytes) 000049 1781228924.429180 mapwrite 0x017b37 thru 0x01f800 (0x7cca bytes) 000052 1781228924.505271 fallocate 0x02862a thru 0x0299ab (0x1382 bytes) 000053 1781228924.509403 trunc from 0x040000 to 0x02489e (0x1b761 bytes) 000054 1781228924.528693 MIRROR_EXTEND to 2 mirrors 000056 1781228925.309728 write 0x017333 thru 0x01aedf (0x3bad bytes) 000057 1781228925.467429 trunc from 0x007d22 to 0x0110be (0x939d bytes) 000058 1781228925.478946 write 0x01f3cd thru 0x024a42 (0x5676 bytes) 000059 1781228925.497196 fallocate 0x02a2a1 thru 0x035d15 (0xba75 bytes) 000060 1781228925.513741 read 0x013755 thru 0x01cac9 (0x9375 bytes) 000061 1781228925.520820 punch from 0x009851 to 0x016a4f (0xd1ff bytes) 000062 1781228925.529223 write 0x003116 thru 0x005cfa (0x2be5 bytes) 000063 1781228925.540844 trunc from 0x024a43 to 0x0195c1 (0xb481 bytes) 000064 1781228925.586925 fallocate 0x0229ca thru 0x0277ca (0x4e01 bytes) 000065 1781228925.593539 punch from 0x00a85b to 0x017fea (0xd790 bytes) 000066 1781228925.610413 fallocate 0x02c591 thru 0x038921 (0xc391 bytes) 000067 1781228925.615683 punch from 0x0029ab to 0x00dfa2 (0xb5f8 bytes) 000069 1781228925.625424 MIRROR_RESYNC 2 mirrors 000071 1781228926.233812 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000072 1781228926.457919 trunc from 0x0277cb to 0x017db5 (0xfa15 bytes) 000075 1781228926.522679 trunc from 0x004804 to 0x006e0e (0x260b bytes) 000076 1781228926.529288 write 0x0076ea thru 0x00a5cc (0x2ee3 bytes) 000077 1781228926.544688 trunc from 0x017db6 to 0x01be3f (0x408a bytes) 000078 1781228926.563248 read 0x00ee69 thru 0x01a884 (0xba1c bytes) 000080 1781228926.577808 mapread 0x00a6d0 thru 0x0179fb (0xd32c bytes) 000083 1781228926.592046 mapread 0x00917c thru 0x011e76 (0x8cfb bytes) 000084 1781228926.593473 write 0x010563 thru 0x0182a3 (0x7d41 bytes) 000085 1781228926.595571 read 0x0012dd thru 0x00b945 (0xa669 bytes) 000087 1781228926.609259 read 0x005e3b thru 0x0118f0 (0xbab6 bytes) 000088 1781228926.614772 MIRROR_EXTEND to 2 mirrors 000089 1781228927.710222 trunc from 0x01be40 to 0x025179 (0x933a bytes) 000090 1781228927.851745 write 0x01fa0c thru 0x02644d (0x6a42 bytes) 000091 1781228927.870090 write 0x022adf thru 0x03077c (0xdc9e bytes) 000092 1781228927.874666 punch from 0x0202c3 to 0x02b889 (0xb5c7 bytes) 000093 1781228927.913740 trunc from 0x03077d to 0x033e57 (0x36db bytes) 000094 1781228927.986243 fallocate 0x0383d8 thru 0x03ba82 (0x36ab bytes) 000095 1781228928.001204 read 0x03365b thru 0x033e57 (0x7fd bytes) 000096 1781228928.017853 read 0x020595 thru 0x02f20c (0xec78 bytes) 000097 1781228928.068817 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000098 1781228928.316627 punch from 0x031d86 to 0x033e57 (0x20d2 bytes) 000100 1781228928.331722 mapwrite 0x007e49 thru 0x011714 (0x98cc bytes) 000101 1781228928.393722 read 0x024526 thru 0x02a33c (0x5e17 bytes) 000102 1781228928.403192 trunc from 0x000481 to 0x004439 (0x3fb9 bytes) 000104 1781228928.410059 mapwrite 0x02b00a thru 0x03921b (0xe212 bytes) 000105 1781228928.529221 mapread 0x00c1c8 thru 0x01aac4 (0xe8fd bytes) 000107 1781228928.547235 trunc from 0x03921c to 0x02d551 (0xbcca bytes) 000108 1781228928.572088 read 0x00864c thru 0x0116e1 (0x9096 bytes) 000109 1781228928.574212 trunc from 0x025097 to 0x02598f (0x8f9 bytes) 000110 1781228928.581699 trunc from 0x0181b1 to 0x01bece (0x3d1e bytes) 000111 1781228928.590023 fallocate 0x0158a5 thru 0x0227ad (0xcf09 bytes) 000112 1781228928.598697 trunc from 0x02d552 to 0x03ed26 (0x117d5 bytes) 000113 1781228928.624606 fallocate 0x02e869 thru 0x03bcbf (0xd457 bytes) 000114 1781228928.633342 trunc from 0x03ed27 to 0x0104ff (0x2e827 bytes) 000115 1781228928.666007 punch from 0x004c50 to 0x0104ff (0xb8b0 bytes) 000116 1781228928.673209 trunc from 0x003672 to 0x00d9e3 (0xa372 bytes) 000117 1781228928.682655 trunc from 0x010500 to 0x00bd3e (0x47c1 bytes) 000118 1781228928.728724 write 0x02843d thru 0x030b03 (0x86c7 bytes) 000119 1781228928.740229 mapwrite 0x02d780 thru 0x031035 (0x38b6 bytes) 000120 1781228928.889266 trunc from 0x031036 to 0x00f71b (0x2191a bytes) 000121 1781228928.920296 write 0x00457f thru 0x0113b5 (0xce37 bytes) 000122 1781228928.927334 mapwrite 0x001d3f thru 0x009799 (0x7a5b bytes) 000123 1781228928.990673 fallocate 0x009591 thru 0x0142b9 (0xad29 bytes) 000124 1781228929.003415 trunc from 0x0113b6 to 0x03260f (0x2125a bytes) 000125 1781228929.034181 fallocate 0x02222d thru 0x030db8 (0xeb8c bytes) 000126 1781228929.044596 write 0x034df2 thru 0x0364d9 (0x16e8 bytes) 000127 1781228929.052261 trunc from 0x0364da to 0x0108fc (0x25bdd bytes) 000128 1781228929.098004 punch from 0x000885 to 0x005e80 (0x55fc bytes) 000129 1781228929.110662 trunc from 0x0108fd to 0x00c8a1 (0x405b bytes) 000130 1781228929.141095 mapread 0x00ac8d thru 0x00c8a1 (0x1c15 bytes) 000131 1781228929.142378 read 0x005304 thru 0x00c8a1 (0x759e bytes) 000132 1781228929.159562 trunc from 0x00c8a2 to 0x009a10 (0x2e91 bytes) 000133 1781228929.250572 trunc from 0x005b3f to 0x009a10 (0x3ed2 bytes) 000135 1781228929.275883 write 0x03b5e6 thru 0x03ffff (0x4a1a bytes) 000137 1781228929.287101 write 0x02da6c thru 0x035fa2 (0x8537 bytes) 000138 1781228929.305133 trunc from 0x040000 to 0x025c12 (0x1a3ed bytes) 000139 1781228929.333646 trunc from 0x02218f to 0x025c12 (0x3a84 bytes) 000140 1781228929.350581 trunc from 0x025c13 to 0x00eaf5 (0x1711d bytes) 000141 1781228929.393214 fallocate 0x0287d9 thru 0x0306a2 (0x7eca bytes) 000142 1781228929.404564 write 0x00697b thru 0x00d09e (0x6724 bytes) 000143 1781228929.428465 mapread 0x0234c3 thru 0x02e122 (0xac60 bytes) 000144 1781228929.593024 fallocate 0x0359ce thru 0x03f1d0 (0x9803 bytes) 000145 1781228929.601778 mapwrite 0x038e3c thru 0x03ffff (0x71c4 bytes) 000146 1781228929.750618 read 0x033213 thru 0x03405a (0xe48 bytes) 000147 1781228929.770094 mapread 0x0145dd thru 0x01676d (0x2191 bytes) 000148 1781228929.772648 write 0x038001 thru 0x03ffff (0x7fff bytes) 000149 1781228929.777419 mapwrite 0x000627 thru 0x00be2e (0xb808 bytes) 000150 1781228929.810676 mapread 0x02c1c5 thru 0x039739 (0xd575 bytes) 000152 1781228929.820206 trunc from 0x026591 to 0x0314fc (0xaf6c bytes) 000153 1781228929.827683 write 0x013e1a thru 0x018403 (0x45ea bytes) 000154 1781228929.831368 write 0x00de17 thru 0x018119 (0xa303 bytes) 000155 1781228929.835290 write 0x0338dc thru 0x039e21 (0x6546 bytes) 000156 1781228929.840094 write 0x001f91 thru 0x00d433 (0xb4a3 bytes) 000157 1781228929.845765 fallocate 0x0008b7 thru 0x003e7a (0x35c4 bytes) 000159 1781228929.884114 write 0x009a89 thru 0x0107e8 (0x6d60 bytes) 000160 1781228929.901860 trunc from 0x040000 to 0x009b6e (0x36491 bytes) 000161 1781228929.954758 mapwrite 0x035202 thru 0x03ad83 (0x5b82 bytes) 000162 1781228930.132784 write 0x00d5f7 thru 0x01550d (0x7f17 bytes) 000163 1781228930.181129 trunc from 0x03ad84 to 0x019fa3 (0x20de0 bytes) 000164 1781228930.215359 write 0x00bb02 thru 0x00d203 (0x1702 bytes) 000165 1781228930.221926 mapread 0x01869e thru 0x019fa3 (0x1906 bytes) 000166 1781228930.234789 read 0x0053b7 thru 0x013c5b (0xe8a5 bytes) 000167 1781228930.244293 mapwrite 0x015e1e thru 0x01f8c1 (0x9aa4 bytes) 000168 1781228930.342620 trunc from 0x01d316 to 0x01f8c1 (0x25ac bytes) 000169 1781228930.383897 MIRROR_EXTEND to 2 mirrors 000170 1781228931.332606 trunc from 0x01b97d to 0x01f8c1 (0x3f45 bytes) 000171 1781228931.423115 read 0x01ab17 thru 0x01f8c1 (0x4dab bytes) 000172 1781228931.430166 mapwrite 0x026925 thru 0x035c19 (0xf2f5 bytes) 000173 1781228931.570507 punch from 0x022bd5 to 0x02913d (0x6569 bytes) 000174 1781228931.578606 trunc from 0x00c4dd to 0x01685e (0xa382 bytes) 000175 1781228931.583956 trunc from 0x01d58e to 0x025892 (0x8305 bytes) 000176 1781228931.593195 write 0x01f070 thru 0x02290f (0x38a0 bytes) 000177 1781228931.611743 trunc from 0x028103 to 0x02c1ae (0x40ac bytes) 000178 1781228931.624601 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000179 1781228931.821658 fallocate 0x018fb2 thru 0x01aa6b (0x1aba bytes) 000180 1781228931.828204 mapread 0x01a519 thru 0x01eb47 (0x462f bytes) 000181 1781228931.881256 mapread 0x01227f thru 0x0190b6 (0x6e38 bytes) 000182 1781228931.891682 fallocate 0x031c89 thru 0x033cd0 (0x2048 bytes) 000185 1781228931.896848 punch from 0x01c95e to 0x022ba6 (0x6249 bytes) 000186 1781228931.903560 trunc from 0x035c1a to 0x00cabd (0x2915c bytes) 000187 1781228931.923337 fallocate 0x02add5 thru 0x035245 (0xa471 bytes) 000188 1781228931.929734 fallocate 0x00f93a thru 0x010dde (0x14a5 bytes) 000189 1781228931.936000 trunc from 0x010ddf to 0x03dd59 (0x2cf7b bytes) 000192 1781228931.962436 MIRROR_EXTEND to 2 mirrors 000193 1781228933.284815 read 0x02cbbe thru 0x03a406 (0xd849 bytes) 000194 1781228933.298362 write 0x030984 thru 0x033130 (0x27ad bytes) 000195 1781228933.378529 punch from 0x01dfe9 to 0x021576 (0x358e bytes) 000196 1781228933.405690 MIRROR_RESYNC 2 mirrors 000197 1781228934.758754 read 0x00f661 thru 0x013c0e (0x45ae bytes) 000198 1781228934.770553 read 0x036e87 thru 0x03dd59 (0x6ed3 bytes) 000199 1781228934.784167 punch from 0x030bbf to 0x034e7e (0x42c0 bytes) 000201 1781228934.863168 punch from 0x0052e6 to 0x013f78 (0xec93 bytes) 000202 1781228934.871837 punch from 0x0250ed to 0x034c11 (0xfb25 bytes) 000203 1781228934.882196 mapread 0x01a638 thru 0x01f5cc (0x4f95 bytes) 000204 1781228934.964761 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3dd5a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd5a): Mirror 2: 0xc3361a4b (stale) CRC-32 checksum value for chunk [0, 0x3dd5a): Mirror 1: 0x10b47e9 Mirror 3: 0x10b47e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd5a: skipped 000205 1781228936.366040 trunc from 0x001b5a to 0x0020df (0x586 bytes) 000206 1781228936.417522 read 0x004569 thru 0x00a9be (0x6456 bytes) 000207 1781228936.431232 punch from 0x032de7 to 0x037f97 (0x51b1 bytes) 000208 1781228936.437102 fallocate 0x02216d thru 0x02ac62 (0x8af6 bytes) 000209 1781228936.449101 mapwrite 0x02d6de thru 0x035ce3 (0x8606 bytes) 000210 1781228936.532896 mapread 0x023107 thru 0x02d9f2 (0xa8ec bytes) 000211 1781228936.561639 read 0x02c042 thru 0x03a814 (0xe7d3 bytes) 000212 1781228936.572160 trunc from 0x03dd5a to 0x03f10e (0x13b5 bytes) 000213 1781228936.609781 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f10f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f10f): Mirror 2: 0x7b5963e9 (stale) Mirror 3: 0x6cb1cc99 (stale) CRC-32 checksum value for chunk [0, 0x3f10f): Mirror 1: 0xe955f8b6 Mirror 4: 0xe955f8b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f10f: skipped 000214 1781228938.385523 punch from 0x022bb7 to 0x0274f2 (0x493c bytes) 000215 1781228938.480846 fallocate 0x035017 thru 0x03d171 (0x815b bytes) 000216 1781228938.498635 mapread 0x01bea4 thru 0x01c9e7 (0xb44 bytes) 000217 1781228938.525043 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f10f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f10f): Mirror 2: 0x7b5963e9 (stale) Mirror 3: 0x6cb1cc99 (stale) Mirror 4: 0xe955f8b6 (stale) CRC-32 checksum value for chunk [0, 0x3f10f): Mirror 1: 0xe955f8b6 Mirror 5: 0xe955f8b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f10f: skipped 000219 1781228940.571232 MIRROR_RESYNC 5 mirrors 000220 1781228942.722688 MIRROR_EXTEND to 6 mirrors 000221 1781228943.770117 trunc from 0x03f10f to 0x00995a (0x357b4 bytes) 000222 1781228943.890521 trunc from 0x00995b to 0x024b1b (0x1b1c1 bytes) 000223 1781228943.922505 mapread 0x00a549 thru 0x00c05e (0x1b16 bytes) 000224 1781228943.962062 write 0x0111c8 thru 0x01ad49 (0x9b82 bytes) 000225 1781228943.979261 write 0x02a0c2 thru 0x02a9ad (0x8ec bytes) 000226 1781228943.989255 fallocate 0x021dae thru 0x02b327 (0x957a bytes) 000227 1781228944.002125 trunc from 0x02963d to 0x02b327 (0x1ceb bytes) 000228 1781228944.029870 punch from 0x00bb90 to 0x01736c (0xb7dd bytes) 000229 1781228944.043524 MIRROR_RESYNC 6 mirrors 000230 1781228946.387091 write 0x027f84 thru 0x030995 (0x8a12 bytes) 000231 1781228946.466134 MIRROR_RESYNC 6 mirrors 000232 1781228948.295910 read 0x02e6b8 thru 0x030995 (0x22de bytes) 000233 1781228948.300793 read 0x0070ea thru 0x00f12f (0x8046 bytes) 000234 1781228948.317614 read 0x026302 thru 0x030995 (0xa694 bytes) 000235 1781228948.320964 fallocate 0x03179f thru 0x034611 (0x2e73 bytes) 000237 1781228948.393036 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34612): Mirror 2: 0x5b2dd39e (stale) Mirror 3: 0x5b2dd39e (stale) Mirror 4: 0x5b2dd39e (stale) Mirror 6: 0x5b2dd39e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34612): Mirror 2: 0x5b2dd39e (stale) Mirror 3: 0x5b2dd39e (stale) Mirror 4: 0x5b2dd39e (stale) Mirror 6: 0x5b2dd39e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34612: skipped 000238 1781228949.026198 mapwrite 0x00c324 thru 0x01173c (0x5419 bytes) 000239 1781228949.096372 fallocate 0x01c042 thru 0x01d770 (0x172f bytes) 000240 1781228949.105236 write 0x00ab96 thru 0x0190b8 (0xe523 bytes) 000242 1781228949.127921 punch from 0x01caab to 0x020397 (0x38ed bytes) 000243 1781228949.173032 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34612): Mirror 2: 0x5b2dd39e (stale) Mirror 3: 0x5b2dd39e (stale) Mirror 4: 0x5b2dd39e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34612): Mirror 2: 0x5b2dd39e (stale) Mirror 3: 0x5b2dd39e (stale) Mirror 4: 0x5b2dd39e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34612: skipped 000244 1781228949.679572 trunc from 0x034612 to 0x037be6 (0x35d5 bytes) 000245 1781228949.713654 MIRROR_RESYNC 4 mirrors 000246 1781228951.176830 mapread 0x03080f thru 0x037be6 (0x73d8 bytes) 000247 1781228951.200280 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000248 1781228951.550462 write 0x01d25b thru 0x02c750 (0xf4f6 bytes) 000250 1781228951.635610 fallocate 0x01a3f0 thru 0x0211b5 (0x6dc6 bytes) 000254 1781228951.644963 trunc from 0x01dda9 to 0x02cb66 (0xedbe bytes) 000255 1781228951.665108 mapread 0x00db58 thru 0x00e433 (0x8dc bytes) 000256 1781228951.675170 punch from 0x00dfc2 to 0x01607b (0x80ba bytes) 000257 1781228951.683829 fallocate 0x013ecd thru 0x01a00f (0x6143 bytes) 000258 1781228951.691125 trunc from 0x037be7 to 0x009264 (0x2e982 bytes) 000259 1781228951.719911 trunc from 0x0048b2 to 0x009264 (0x49b3 bytes) 000260 1781228951.746909 trunc from 0x000a33 to 0x007431 (0x69ff bytes) 000261 1781228951.766882 read 0x008e2d thru 0x009264 (0x438 bytes) 000262 1781228951.777983 mapwrite 0x00e8b7 thru 0x014508 (0x5c52 bytes) 000263 1781228951.847026 write 0x009251 thru 0x011f61 (0x8d11 bytes) 000264 1781228951.849205 mapwrite 0x03007d thru 0x03e0d8 (0xe05c bytes) 000265 1781228951.930446 write 0x028b9f thru 0x0353c4 (0xc826 bytes) 000266 1781228951.962277 read 0x025a43 thru 0x03013d (0xa6fb bytes) 000267 1781228951.965161 write 0x018928 thru 0x024243 (0xb91c bytes) 000268 1781228951.992083 write 0x03720e thru 0x0389af (0x17a2 bytes) 000269 1781228951.995045 fallocate 0x03b995 thru 0x03ffff (0x466b bytes) 000270 1781228952.005949 read 0x0123e8 thru 0x017813 (0x542c bytes) 000271 1781228952.008981 trunc from 0x0361b4 to 0x0398a4 (0x36f1 bytes) 000272 1781228952.049475 trunc from 0x040000 to 0x00f8ab (0x30754 bytes) 000273 1781228952.109375 write 0x01916c thru 0x0224b6 (0x934b bytes) 000274 1781228952.111161 write 0x02e7e5 thru 0x033468 (0x4c84 bytes) 000275 1781228952.123690 write 0x037862 thru 0x03ffff (0x879e bytes) 000276 1781228952.128640 mapwrite 0x02b80c thru 0x035d5e (0xa553 bytes) 000277 1781228952.205815 mapwrite 0x03307f thru 0x03af16 (0x7e98 bytes) 000278 1781228952.263574 trunc from 0x040000 to 0x0374b3 (0x8b4c bytes) 000279 1781228952.298988 fallocate 0x0361e6 thru 0x039589 (0x33a4 bytes) 000281 1781228952.338117 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x374b4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374b4): Mirror 2: 0x56e4cffa (stale) Mirror 4: 0x56e4cffa (stale) CRC-32 checksum value for chunk [0, 0x374b4): Mirror 1: 0x4d1c4eca Mirror 5: 0x4d1c4eca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374b4: skipped 000282 1781228953.375588 write 0x01bca5 thru 0x01e4b8 (0x2814 bytes) 000283 1781228953.431819 trunc from 0x01cac2 to 0x01f184 (0x26c3 bytes) 000284 1781228953.445523 read 0x032449 thru 0x0374b3 (0x506b bytes) 000285 1781228953.458059 mapwrite 0x01757e thru 0x026e44 (0xf8c7 bytes) 000286 1781228953.504449 fallocate 0x018e4a thru 0x021f42 (0x90f9 bytes) 000288 1781228953.509837 mapwrite 0x01a477 thru 0x022cdd (0x8867 bytes) 000289 1781228953.527778 read 0x01b7bb thru 0x026b0d (0xb353 bytes) 000290 1781228953.528987 mapread 0x02ae8a thru 0x02b520 (0x697 bytes) 000291 1781228953.552275 mapwrite 0x02374d thru 0x023983 (0x237 bytes) 000292 1781228953.567274 mapread 0x0366e8 thru 0x0374b3 (0xdcc bytes) 000293 1781228953.573251 MIRROR_RESYNC 4 mirrors 000294 1781228955.033253 mapread 0x006af6 thru 0x016986 (0xfe91 bytes) 000295 1781228955.067650 fallocate 0x03318f thru 0x03ffff (0xce71 bytes) 000296 1781228955.115460 read 0x00b4bc thru 0x01274c (0x7291 bytes) 000297 1781228955.131566 trunc from 0x0374b4 to 0x01b186 (0x1c32d bytes) 000298 1781228955.151356 punch from 0x019266 to 0x01b186 (0x1f21 bytes) 000299 1781228955.157735 mapread 0x00f774 thru 0x01164c (0x1ed9 bytes) 000301 1781228955.160652 read 0x00c3b7 thru 0x01b186 (0xedd0 bytes) 000302 1781228955.170047 trunc from 0x01b187 to 0x035f34 (0x1adae bytes) 000303 1781228955.186693 mapread 0x00fc0c thru 0x01024e (0x643 bytes) 000304 1781228955.187762 write 0x021fc4 thru 0x028303 (0x6340 bytes) 000305 1781228955.210423 mapread 0x00b599 thru 0x014494 (0x8efc bytes) 000306 1781228955.212329 mapwrite 0x026f64 thru 0x0330ae (0xc14b bytes) 000308 1781228955.249277 mapread 0x02f230 thru 0x035f34 (0x6d05 bytes) 000309 1781228955.250779 trunc from 0x035f35 to 0x022336 (0x13bfe bytes) 000310 1781228955.280520 mapread 0x007d57 thru 0x0142f0 (0xc59a bytes) 000311 1781228955.281971 trunc from 0x0144c0 to 0x01a114 (0x5c55 bytes) 000312 1781228955.288070 write 0x027b09 thru 0x033ec1 (0xc3b9 bytes) 000313 1781228955.290739 write 0x01425e thru 0x0159ff (0x17a2 bytes) 000314 1781228955.300408 mapread 0x001a9b thru 0x00284f (0xdb5 bytes) 000315 1781228955.301573 mapwrite 0x03ddbf thru 0x03ffff (0x2241 bytes) 000316 1781228955.347578 fallocate 0x03d64c thru 0x03ffff (0x29b4 bytes) 000317 1781228955.361420 punch from 0x0304c5 to 0x03b25c (0xad98 bytes) 000320 1781228955.376411 MIRROR_RESYNC 4 mirrors 000321 1781228957.274413 MIRROR_EXTEND to 5 mirrors 000322 1781228958.528491 read 0x0138b3 thru 0x01add0 (0x751e bytes) 000323 1781228958.539499 write 0x02d4ab thru 0x036e39 (0x998f bytes) 000324 1781228958.715999 read 0x014b41 thru 0x01bd07 (0x71c7 bytes) 000325 1781228958.745187 trunc from 0x03c8e0 to 0x03ce61 (0x582 bytes) 000326 1781228958.756089 mapread 0x00dbb7 thru 0x01137e (0x37c8 bytes) 000327 1781228958.766940 punch from 0x021e79 to 0x02298f (0xb17 bytes) 000328 1781228958.777029 trunc from 0x03b606 to 0x03ffff (0x49fa bytes) 000329 1781228958.785012 trunc from 0x040000 to 0x02dba9 (0x12456 bytes) 000330 1781228958.830134 fallocate 0x02d1f9 thru 0x02d331 (0x139 bytes) 000331 1781228958.855601 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2dbaa): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dbaa): Mirror 2: 0xcd895706 (stale) Mirror 4: 0xcd895706 (stale) Mirror 5: 0xcd895706 (stale) Mirror 6: 0xcd895706 (stale) CRC-32 checksum value for chunk [0, 0x2dbaa): Mirror 1: 0x8850b128 Mirror 7: 0x8850b128 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dbaa: skipped 000332 1781228960.404706 trunc from 0x01a776 to 0x02190f (0x719a bytes) 000334 1781228960.551669 punch from 0x02aa44 to 0x02b67c (0xc39 bytes) 000335 1781228960.562780 read 0x00ddce thru 0x01c9cd (0xec00 bytes) 000336 1781228960.630766 trunc from 0x01aa1d to 0x029524 (0xeb08 bytes) 000337 1781228960.645776 read 0x01af57 thru 0x026f93 (0xc03d bytes) 000338 1781228960.676542 trunc from 0x02dbaa to 0x033cb6 (0x610d bytes) 000339 1781228960.746362 read 0x02ad1c thru 0x033cb6 (0x8f9b bytes) 000340 1781228960.787529 MIRROR_RESYNC 6 mirrors 000342 1781228962.944581 mapwrite 0x02639e thru 0x026653 (0x2b6 bytes) 000343 1781228963.030578 mapread 0x029d3a thru 0x033cb6 (0x9f7d bytes) 000345 1781228963.075582 mapread 0x031d20 thru 0x033cb6 (0x1f97 bytes) 000346 1781228963.077712 mapwrite 0x02a420 thru 0x02b7bf (0x13a0 bytes) 000347 1781228963.092650 write 0x02441a thru 0x02c9b6 (0x859d bytes) 000349 1781228963.097966 write 0x02a3d8 thru 0x03009a (0x5cc3 bytes) 000350 1781228963.101677 trunc from 0x02271d to 0x02c9b1 (0xa295 bytes) 000351 1781228963.119912 fallocate 0x012cb1 thru 0x01829a (0x55ea bytes) 000352 1781228963.131720 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33cb7): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cb7): Mirror 2: 0xe6c7f8a3 (stale) Mirror 4: 0xe6c7f8a3 (stale) Mirror 5: 0xe6c7f8a3 (stale) Mirror 6: 0xe6c7f8a3 (stale) Mirror 7: 0xe6c7f8a3 (stale) CRC-32 checksum value for chunk [0, 0x33cb7): Mirror 1: 0x1dd24c6c Mirror 8: 0x1dd24c6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cb7: skipped 000353 1781228964.140352 mapread 0x02c030 thru 0x031d05 (0x5cd6 bytes) 000354 1781228964.192484 write 0x02b0bc thru 0x035dec (0xad31 bytes) 000355 1781228964.282213 fallocate 0x01fb1f thru 0x02bc4f (0xc131 bytes) 000356 1781228964.287950 fallocate 0x03f4a8 thru 0x03ffff (0xb58 bytes) 000357 1781228964.294048 fallocate 0x029a44 thru 0x0327c5 (0x8d82 bytes) 000358 1781228964.303160 punch from 0x004689 to 0x0046b9 (0x31 bytes) 000359 1781228964.307083 trunc from 0x040000 to 0x020c09 (0x1f3f6 bytes) 000360 1781228964.403690 write 0x0234ef thru 0x023717 (0x229 bytes) 000362 1781228964.406756 mapwrite 0x01691f thru 0x01c0b6 (0x5798 bytes) 000363 1781228964.436184 write 0x036911 thru 0x03dbfd (0x72ed bytes) 000364 1781228964.437905 trunc from 0x03dbfe to 0x023179 (0x1aa84 bytes) 000365 1781228964.456715 trunc from 0x02317a to 0x03c697 (0x1951e bytes) 000366 1781228964.472347 trunc from 0x035b70 to 0x03c697 (0x6b28 bytes) 000367 1781228964.479233 write 0x023412 thru 0x023b64 (0x753 bytes) 000368 1781228964.480396 fallocate 0x019632 thru 0x026b91 (0xd560 bytes) 000369 1781228964.485275 fallocate 0x0074d4 thru 0x012c3b (0xb768 bytes) 000370 1781228964.489938 trunc from 0x018aaf to 0x0208a6 (0x7df8 bytes) 000372 1781228964.496016 write 0x0019f1 thru 0x0027bb (0xdcb bytes) 000373 1781228964.508955 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c698): Mirror 4: 0x4e9a1673 (stale) Mirror 5: 0x4e9a1673 (stale) Mirror 6: 0x4e9a1673 (stale) Mirror 7: 0x4e9a1673 (stale) Mirror 8: 0x2c529319 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c698): Mirror 4: 0x4e9a1673 (stale) Mirror 5: 0x4e9a1673 (stale) Mirror 6: 0x4e9a1673 (stale) Mirror 7: 0x4e9a1673 (stale) Mirror 8: 0x2c529319 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c698: skipped 000375 1781228965.025079 mapread 0x034df3 thru 0x039ec7 (0x50d5 bytes) 000376 1781228965.073408 mapread 0x00e02d thru 0x0107dc (0x27b0 bytes) 000377 1781228965.116027 trunc from 0x03c698 to 0x016923 (0x25d74 bytes) 000378 1781228965.217895 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x16924): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16924): Mirror 4: 0x84239fc1 (stale) Mirror 5: 0x84239fc1 (stale) Mirror 6: 0x84239fc1 (stale) Mirror 7: 0x84239fc1 (stale) Mirror 8: 0x84239fc1 (stale) CRC-32 checksum value for chunk [0, 0x16924): Mirror 1: 0xe131361a Mirror 9: 0xe131361a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16924: skipped 000379 1781228966.760379 trunc from 0x003cc6 to 0x00743c (0x3777 bytes) 000380 1781228966.848696 trunc from 0x008188 to 0x016923 (0xe79c bytes) 000381 1781228966.855647 trunc from 0x016924 to 0x030489 (0x19b66 bytes) 000382 1781228966.908300 write 0x015950 thru 0x02594f (0x10000 bytes) 000383 1781228966.934192 trunc from 0x027f52 to 0x02c5e0 (0x468f bytes) 000384 1781228966.950876 mapwrite 0x013236 thru 0x015d3e (0x2b09 bytes) 000385 1781228967.054513 fallocate 0x00ba78 thru 0x00bbf0 (0x179 bytes) 000386 1781228967.067044 read 0x0271bc thru 0x02c050 (0x4e95 bytes) 000387 1781228967.093198 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3048a): Mirror 4: 0x8be8858d (stale) Mirror 5: 0x8be8858d (stale) Mirror 7: 0x8be8858d (stale) Mirror 8: 0x4272e783 (stale) Mirror 9: 0x83f47bc9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3048a): Mirror 4: 0x8be8858d (stale) Mirror 5: 0x8be8858d (stale) Mirror 7: 0x8be8858d (stale) Mirror 8: 0x4272e783 (stale) Mirror 9: 0x83f47bc9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3048a: skipped 000388 1781228967.710441 fallocate 0x03f557 thru 0x03ffff (0xaa9 bytes) 000389 1781228967.717039 trunc from 0x02da6e to 0x03ac30 (0xd1c3 bytes) 000390 1781228967.723436 trunc from 0x012e31 to 0x0198f7 (0x6ac7 bytes) 000391 1781228967.729543 punch from 0x001434 to 0x00506a (0x3c37 bytes) 000392 1781228967.736611 mapread 0x001133 thru 0x0065c6 (0x5494 bytes) 000393 1781228967.761697 trunc from 0x040000 to 0x02c945 (0x136ba bytes) 000394 1781228967.804027 punch from 0x005612 to 0x00c5d8 (0x6fc7 bytes) 000395 1781228967.826109 mapwrite 0x01f9e5 thru 0x0242d4 (0x48f0 bytes) 000396 1781228967.874126 punch from 0x013cb1 to 0x01ab57 (0x6ea7 bytes) 000397 1781228967.886790 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c946): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c946): Mirror 4: 0xf95f27ba (stale) Mirror 5: 0xf95f27ba (stale) Mirror 7: 0xf95f27ba (stale) Mirror 8: 0xb2265e66 (stale) Mirror 9: 0x66d9f2e (stale) CRC-32 checksum value for chunk [0, 0x2c946): Mirror 1: 0x234c5a6a Mirror 10: 0x234c5a6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c946: skipped 000398 1781228969.408894 mapwrite 0x01762f thru 0x01e112 (0x6ae4 bytes) 000399 1781228969.547022 read 0x01cdad thru 0x02a505 (0xd759 bytes) 000400 1781228969.560870 read 0x02c42d thru 0x02c945 (0x519 bytes) 000402 1781228969.570528 fallocate 0x029473 thru 0x02e87c (0x540a bytes) 000403 1781228969.578622 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e87d): Mirror 4: 0xf1514475 (stale) Mirror 5: 0xf1514475 (stale) Mirror 7: 0xf1514475 (stale) Mirror 8: 0xac066743 (stale) Mirror 9: 0x3976fdf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e87d): Mirror 4: 0xf1514475 (stale) Mirror 5: 0xf1514475 (stale) Mirror 7: 0xf1514475 (stale) Mirror 8: 0xac066743 (stale) Mirror 9: 0x3976fdf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e87d: skipped 000404 1781228970.209878 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e87d): Mirror 5: 0xf1514475 (stale) Mirror 7: 0xf1514475 (stale) Mirror 8: 0xac066743 (stale) Mirror 9: 0x3976fdf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e87d): Mirror 5: 0xf1514475 (stale) Mirror 7: 0xf1514475 (stale) Mirror 8: 0xac066743 (stale) Mirror 9: 0x3976fdf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e87d: skipped 000405 1781228970.635829 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e87d): Mirror 7: 0xf1514475 (stale) Mirror 8: 0xac066743 (stale) Mirror 9: 0x3976fdf1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e87d): Mirror 7: 0xf1514475 (stale) Mirror 8: 0xac066743 (stale) Mirror 9: 0x3976fdf1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e87d: skipped 000406 1781228971.007126 write 0x014c88 thru 0x016aa5 (0x1e1e bytes) 000407 1781228971.027943 mapwrite 0x038c13 thru 0x038c84 (0x72 bytes) 000408 1781228971.069135 trunc from 0x038c85 to 0x03bebb (0x3237 bytes) 000409 1781228971.091205 trunc from 0x03bebc to 0x03984c (0x266f bytes) 000410 1781228971.108192 read 0x02a1b2 thru 0x032800 (0x864f bytes) 000411 1781228971.123717 MIRROR_RESYNC 4 mirrors 000412 1781228972.331783 mapwrite 0x02473a thru 0x0345ba (0xfe81 bytes) 000413 1781228972.426726 mapread 0x01bbd6 thru 0x020bc7 (0x4ff2 bytes) 000414 1781228972.437650 fallocate 0x006472 thru 0x008dd9 (0x2968 bytes) 000415 1781228972.446863 MIRROR_RESYNC 4 mirrors 000416 1781228973.420854 MIRROR_RESYNC 4 mirrors 000417 1781228974.005263 trunc from 0x017aef to 0x01d73e (0x5c50 bytes) 000418 1781228974.067048 mapread 0x028436 thru 0x032b25 (0xa6f0 bytes) 000421 1781228974.121835 trunc from 0x020f7d to 0x025420 (0x44a4 bytes) 000422 1781228974.142873 write 0x018fd5 thru 0x01e4f6 (0x5522 bytes) 000424 1781228974.167379 write 0x01b19e thru 0x023472 (0x82d5 bytes) 000425 1781228974.178506 mapwrite 0x0362e2 thru 0x03ffff (0x9d1e bytes) 000426 1781228974.279756 punch from 0x0097a0 to 0x00fa38 (0x6299 bytes) 000428 1781228974.289890 read 0x00fe93 thru 0x013cd6 (0x3e44 bytes) 000429 1781228974.304627 MIRROR_RESYNC 4 mirrors 000430 1781228975.599704 mapread 0x03de6e thru 0x03ffff (0x2192 bytes) 000431 1781228975.618188 mapwrite 0x03a2df thru 0x03ffff (0x5d21 bytes) 000432 1781228975.708364 trunc from 0x040000 to 0x0214ec (0x1eb13 bytes) 000433 1781228975.744014 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x214ed): Mirror 8: 0x89bec6da (stale) Mirror 9: 0x89bec6da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214ed): Mirror 8: 0x89bec6da (stale) Mirror 9: 0x89bec6da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214ed: skipped 000434 1781228976.090373 trunc from 0x00577b to 0x011d0d (0xc593 bytes) 000435 1781228976.098392 trunc from 0x0214ed to 0x02bfac (0xaac0 bytes) 000436 1781228976.118691 trunc from 0x02bfad to 0x005913 (0x26699 bytes) 000437 1781228976.132765 mapwrite 0x03fa98 thru 0x03ffff (0x568 bytes) 000438 1781228976.164158 mapread 0x027fb8 thru 0x02cbfc (0x4c45 bytes) 000439 1781228976.183076 fallocate 0x01923f thru 0x025cbd (0xca7f bytes) 000440 1781228976.188169 fallocate 0x03a3b3 thru 0x03ffff (0x5c4d bytes) 000441 1781228976.194188 trunc from 0x040000 to 0x00f788 (0x30877 bytes) 000442 1781228976.208290 trunc from 0x00f789 to 0x002f4e (0xc83a bytes) 000444 1781228976.223073 trunc from 0x001294 to 0x002f4e (0x1cbb bytes) 000445 1781228976.227843 write 0x013f10 thru 0x01fb6b (0xbc5c bytes) 000446 1781228976.229936 trunc from 0x01fb6c to 0x021ce7 (0x217c bytes) 000447 1781228976.247707 punch from 0x010a52 to 0x01f55e (0xeb0d bytes) 000448 1781228976.260874 fallocate 0x02f3b6 thru 0x03a061 (0xacac bytes) 000450 1781228976.267287 write 0x01ec04 thru 0x02b15e (0xc55b bytes) 000451 1781228976.281737 write 0x023d29 thru 0x02ce5e (0x9136 bytes) 000452 1781228976.303879 trunc from 0x02e1ca to 0x035582 (0x73b9 bytes) 000453 1781228976.317660 read 0x0001c1 thru 0x008b8e (0x89ce bytes) 000454 1781228976.328710 write 0x00b837 thru 0x013c23 (0x83ed bytes) 000455 1781228976.336439 punch from 0x005802 to 0x005f85 (0x784 bytes) 000456 1781228976.344167 trunc from 0x03a062 to 0x00c59c (0x2dac5 bytes) 000457 1781228976.366534 punch from 0x0090eb to 0x00c59c (0x34b2 bytes) 000458 1781228976.381325 mapread 0x00a734 thru 0x00c59c (0x1e69 bytes) 000459 1781228976.402115 mapwrite 0x026ba9 thru 0x033fe1 (0xd439 bytes) 000460 1781228976.506444 punch from 0x031d3a to 0x033fe1 (0x22a8 bytes) 000462 1781228976.517679 mapread 0x02dec9 thru 0x02ef1e (0x1056 bytes) 000463 1781228976.519759 read 0x032617 thru 0x033fe1 (0x19cb bytes) 000465 1781228976.539703 fallocate 0x025932 thru 0x030d0a (0xb3d9 bytes) 000466 1781228976.553983 mapread 0x01fd10 thru 0x0286ca (0x89bb bytes) 000467 1781228976.559456 trunc from 0x01f809 to 0x02d4e7 (0xdcdf bytes) 000468 1781228976.576933 fallocate 0x015135 thru 0x02050f (0xb3db bytes) 000469 1781228976.595859 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 8: 0xaaf66d47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 8: 0xaaf66d47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fe2: skipped 000470 1781228977.073635 mapwrite 0x01af13 thru 0x01d03d (0x212b bytes) 000471 1781228977.163497 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 8: 0xaaf66d47 (stale) CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 1: 0x96f4e0ae Mirror 9: 0x96f4e0ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fe2: skipped 000473 1781228978.664895 read 0x01faf2 thru 0x025a72 (0x5f81 bytes) 000474 1781228978.685102 punch from 0x00e8fc to 0x015913 (0x7018 bytes) 000476 1781228978.754701 trunc from 0x00b1b0 to 0x00b549 (0x39a bytes) 000477 1781228978.762987 trunc from 0x033fe2 to 0x0372fc (0x331b bytes) 000478 1781228978.838773 mapread 0x03304a thru 0x0372fc (0x42b3 bytes) 000479 1781228978.859933 MIRROR_RESYNC 3 mirrors 000480 1781228979.862616 trunc from 0x0372fd to 0x027e5d (0xf49f bytes) 000481 1781228979.939681 trunc from 0x024ece to 0x027e5d (0x2f90 bytes) 000482 1781228979.948644 trunc from 0x027e5e to 0x01d5d6 (0xa887 bytes) 000483 1781228979.972652 trunc from 0x0144d2 to 0x0193d2 (0x4f01 bytes) 000484 1781228979.977688 mapread 0x001761 thru 0x002ab3 (0x1353 bytes) 000485 1781228980.004898 trunc from 0x0144b6 to 0x01cf67 (0x8ab2 bytes) 000486 1781228980.009925 fallocate 0x01ff6c thru 0x02aa46 (0xaadb bytes) 000487 1781228980.019785 mapwrite 0x036c25 thru 0x03ffff (0x93db bytes) 000488 1781228980.071120 punch from 0x0062eb to 0x01430b (0xe021 bytes) 000489 1781228980.080500 punch from 0x009004 to 0x012b3d (0x9b3a bytes) 000490 1781228980.085891 punch from 0x03895b to 0x03ffff (0x76a5 bytes) 000491 1781228980.091845 trunc from 0x040000 to 0x027974 (0x1868b bytes) 000492 1781228980.110712 fallocate 0x0091b2 thru 0x0093ff (0x24e bytes) 000493 1781228980.117636 trunc from 0x027975 to 0x02d4df (0x5b6b bytes) 000494 1781228980.134760 mapread 0x01910d thru 0x01f116 (0x600a bytes) 000495 1781228980.152294 trunc from 0x026138 to 0x02d4df (0x73a8 bytes) 000496 1781228980.157801 punch from 0x02c697 to 0x02d4df (0xe49 bytes) 000497 1781228980.162676 punch from 0x00ecc0 to 0x0193fb (0xa73c bytes) 000498 1781228980.168881 mapwrite 0x007235 thru 0x0105c2 (0x938e bytes) 000499 1781228980.225650 MIRROR_RESYNC 3 mirrors 000500 1781228980.920841 mapread 0x016c40 thru 0x01f040 (0x8401 bytes) 000501 1781228980.954393 fallocate 0x0215ea thru 0x02677d (0x5194 bytes) 000503 1781228981.013367 MIRROR_RESYNC 3 mirrors 000504 1781228981.721719 trunc from 0x02d4e0 to 0x007e5d (0x25682 bytes) 000505 1781228981.782502 trunc from 0x007e5e to 0x0244a8 (0x1c64b bytes) 000506 1781228981.817255 mapread 0x01c598 thru 0x0244a8 (0x7f11 bytes) 000507 1781228981.858726 mapread 0x01e24c thru 0x0244a8 (0x625d bytes) 000508 1781228981.860048 mapwrite 0x01900e thru 0x0194bf (0x4b2 bytes) 000509 1781228981.896402 trunc from 0x0244a9 to 0x0291bb (0x4d13 bytes) 000510 1781228981.936754 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x291bc): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x291bc): Mirror 8: 0x28543780 (stale) Mirror 9: 0x28543780 (stale) CRC-32 checksum value for chunk [0, 0x291bc): Mirror 1: 0x314fafb3 Mirror 10: 0x314fafb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x291bc: skipped 000511 1781228983.023046 write 0x02edc6 thru 0x0346b8 (0x58f3 bytes) 000512 1781228983.064718 read 0x00eee1 thru 0x014cdc (0x5dfc bytes) 000513 1781228983.074847 read 0x02232a thru 0x027d07 (0x59de bytes) 000514 1781228983.085692 mapwrite 0x008d7e thru 0x012002 (0x9285 bytes) 000515 1781228983.108785 read 0x0024d2 thru 0x00f2c9 (0xcdf8 bytes) 000516 1781228983.109506 fallocate 0x03e397 thru 0x03ffff (0x1c69 bytes) 000517 1781228983.114813 mapread 0x016587 thru 0x021ea4 (0xb91e bytes) 000518 1781228983.119218 mapwrite 0x03c497 thru 0x03d4e6 (0x1050 bytes) 000519 1781228983.154750 trunc from 0x01bba9 to 0x020bd2 (0x502a bytes) 000520 1781228983.169466 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3c5a2e91 (stale) Mirror 9: 0x3c5a2e91 (stale) Mirror 10: 0x6494ca99 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa1f2194c Mirror 11: 0xa1f2194c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000521 1781228983.923649 fallocate 0x0379c3 thru 0x038657 (0xc95 bytes) 000522 1781228983.974296 punch from 0x0112dc to 0x01c8d8 (0xb5fd bytes) 000523 1781228983.982116 fallocate 0x033671 thru 0x03ffff (0xc98f bytes) 000524 1781228983.989483 fallocate 0x001dc2 thru 0x010cb0 (0xeeef bytes) 000525 1781228983.995070 punch from 0x0352a2 to 0x03d92a (0x8689 bytes) 000530 1781228984.020478 punch from 0x021008 to 0x029b59 (0x8b52 bytes) 000531 1781228984.025882 trunc from 0x03ec94 to 0x03ffff (0x136c bytes) 000532 1781228984.032038 read 0x01cc36 thru 0x01eb74 (0x1f3f bytes) 000533 1781228984.040287 punch from 0x015913 to 0x01b9d7 (0x60c5 bytes) 000535 1781228984.047257 write 0x02401d thru 0x027115 (0x30f9 bytes) 000536 1781228984.068716 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3c5a2e91 (stale) Mirror 9: 0x3c5a2e91 (stale) Mirror 10: 0x6494ca99 (stale) Mirror 11: 0xa1f2194c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe63b2bb1 Mirror 12: 0xe63b2bb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000537 1781228985.226784 punch from 0x039165 to 0x03c269 (0x3105 bytes) 000538 1781228985.281507 trunc from 0x040000 to 0x0364ea (0x9b15 bytes) 000539 1781228985.335398 read 0x003a08 thru 0x008eb8 (0x54b1 bytes) 000540 1781228985.342414 read 0x01dfcf thru 0x0213bb (0x33ed bytes) 000541 1781228985.352226 mapread 0x01506f thru 0x019f18 (0x4eaa bytes) 000542 1781228985.353465 write 0x01263c thru 0x017578 (0x4f3d bytes) 000543 1781228985.357031 trunc from 0x0031d0 to 0x00a5db (0x740c bytes) 000544 1781228985.364673 mapwrite 0x03d64d thru 0x03ffff (0x29b3 bytes) 000545 1781228985.418099 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x3c5a2e91 (stale) Mirror 9: 0x3c5a2e91 (stale) Mirror 10: 0x6494ca99 (stale) Mirror 11: 0xa1f2194c (stale) Mirror 12: 0xe63b2bb1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfb764d03 Mirror 13: 0xfb764d03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000546 1781228986.953811 write 0x007f58 thru 0x00bdb5 (0x3e5e bytes) 000547 1781228987.030656 mapread 0x008c7f thru 0x013bb0 (0xaf32 bytes) 000548 1781228987.053676 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3c5a2e91 (stale) Mirror 10: 0x6494ca99 (stale) Mirror 11: 0xa1f2194c (stale) Mirror 12: 0xe63b2bb1 (stale) Mirror 13: 0xfb764d03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3c5a2e91 (stale) Mirror 10: 0x6494ca99 (stale) Mirror 11: 0xa1f2194c (stale) Mirror 12: 0xe63b2bb1 (stale) Mirror 13: 0xfb764d03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1781228987.660787 mapwrite 0x00aac6 thru 0x00c199 (0x16d4 bytes) 000550 1781228987.699582 read 0x00f751 thru 0x01894a (0x91fa bytes) 000551 1781228987.715657 punch from 0x01e4ff to 0x029fb3 (0xbab5 bytes) 000552 1781228987.725829 mapread 0x003d47 thru 0x007937 (0x3bf1 bytes) 000553 1781228987.735054 fallocate 0x03bf59 thru 0x03ffff (0x40a7 bytes) 000554 1781228987.748085 mapwrite 0x034fdf thru 0x039c61 (0x4c83 bytes) 000555 1781228987.786600 fallocate 0x002b1b thru 0x005fe4 (0x34ca bytes) 000556 1781228987.791770 mapread 0x005d23 thru 0x007e91 (0x216f bytes) 000557 1781228987.793543 write 0x0241b5 thru 0x0277d3 (0x361f bytes) 000558 1781228987.807170 fallocate 0x034823 thru 0x03d0f7 (0x88d5 bytes) 000559 1781228987.811811 mapread 0x0294df thru 0x02a5f8 (0x111a bytes) 000560 1781228987.813250 mapwrite 0x0045d6 thru 0x00cd36 (0x8761 bytes) 000561 1781228987.838260 MIRROR_RESYNC 6 mirrors 000562 1781228989.620148 read 0x007949 thru 0x00de31 (0x64e9 bytes) 000564 1781228989.632298 read 0x014d28 thru 0x023b1b (0xedf4 bytes) 000565 1781228989.661968 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) 000566 1781228989.990963 mapwrite 0x0266c9 thru 0x02f8f1 (0x9229 bytes) 000567 1781228990.067750 mapread 0x020639 thru 0x029f5a (0x9922 bytes) 000568 1781228990.069103 mapread 0x02dd5a thru 0x030570 (0x2817 bytes) 000569 1781228990.078047 fallocate 0x00d2d0 thru 0x0119ac (0x46dd bytes) 000570 1781228990.082604 mapwrite 0x03d29e thru 0x03ffff (0x2d62 bytes) 000571 1781228990.101707 read 0x0196e5 thru 0x022a5b (0x9377 bytes) 000572 1781228990.109230 punch from 0x01afc7 to 0x01b5cf (0x609 bytes) 000573 1781228990.113088 read 0x03fd83 thru 0x03ffff (0x27d bytes) 000574 1781228990.113769 mapwrite 0x026491 thru 0x02ebe1 (0x8751 bytes) 000575 1781228990.129778 punch from 0x036e8a to 0x03ffff (0x9176 bytes) 000576 1781228990.135140 read 0x032ab7 thru 0x03693d (0x3e87 bytes) 000577 1781228990.148805 trunc from 0x040000 to 0x037c89 (0x8376 bytes) 000578 1781228990.172054 write 0x024922 thru 0x02ae86 (0x6565 bytes) 000579 1781228990.174550 write 0x01e354 thru 0x02909d (0xad4a bytes) 000580 1781228990.176652 punch from 0x0179a6 to 0x01f96c (0x7fc7 bytes) 000581 1781228990.193075 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 9: 0x888451a1 (stale) Mirror 11: 0x888451a1 (stale) Mirror 12: 0x888451a1 (stale) Mirror 13: 0x888451a1 (stale) CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 1: 0xdce5f357 Mirror 14: 0xdce5f357 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c8a: skipped 000582 1781228991.184967 punch from 0x02c938 to 0x031726 (0x4def bytes) 000583 1781228991.236446 read 0x0296fb thru 0x036e92 (0xd798 bytes) 000584 1781228991.252792 mapwrite 0x01161b thru 0x0142d3 (0x2cb9 bytes) 000585 1781228991.295758 trunc from 0x0236b8 to 0x027d73 (0x46bc bytes) 000587 1781228991.303997 mapwrite 0x01f8e7 thru 0x02c77a (0xce94 bytes) 000588 1781228991.352471 mapread 0x0334f6 thru 0x035186 (0x1c91 bytes) 000589 1781228991.353578 write 0x02d443 thru 0x031d38 (0x48f6 bytes) 000590 1781228991.360676 write 0x002b69 thru 0x00cfc2 (0xa45a bytes) 000592 1781228991.371133 fallocate 0x008294 thru 0x00b0e4 (0x2e51 bytes) 000593 1781228991.381486 MIRROR_RESYNC 6 mirrors 000594 1781228992.853230 write 0x036ded thru 0x03ffff (0x9213 bytes) 000595 1781228992.897079 mapread 0x00d63a thru 0x01bc4f (0xe616 bytes) 000596 1781228992.941281 trunc from 0x00806a to 0x012aad (0xaa44 bytes) 000597 1781228992.955293 mapread 0x0021c3 thru 0x003741 (0x157f bytes) 000598 1781228992.993297 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x539c8609 (stale) Mirror 11: 0x539c8609 (stale) Mirror 12: 0x539c8609 (stale) Mirror 13: 0x539c8609 (stale) Mirror 14: 0x539c8609 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa1e22db7 Mirror 15: 0xa1e22db7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1781228994.414857 read 0x008b7b thru 0x0148cb (0xbd51 bytes) 000601 1781228994.433389 write 0x00c8ab thru 0x016da7 (0xa4fd bytes) 000602 1781228994.509154 trunc from 0x040000 to 0x00967e (0x36981 bytes) 000603 1781228994.577427 write 0x02e77a thru 0x0327cd (0x4054 bytes) 000604 1781228994.578891 fallocate 0x001b89 thru 0x008685 (0x6afd bytes) 000605 1781228994.593716 fallocate 0x014dfa thru 0x0225c8 (0xd7cf bytes) 000606 1781228994.613216 punch from 0x025444 to 0x0327cd (0xd38a bytes) 000607 1781228994.646168 trunc from 0x0327ce to 0x00e8fe (0x23ecf bytes) 000608 1781228994.668649 fallocate 0x03e5d2 thru 0x03ffff (0x1a2e bytes) 000609 1781228994.682522 read 0x003709 thru 0x003957 (0x24f bytes) 000610 1781228994.709652 read 0x00ccc2 thru 0x00edf4 (0x2133 bytes) 000611 1781228994.710385 punch from 0x01d941 to 0x01d9c3 (0x83 bytes) 000612 1781228994.722165 punch from 0x0284bc to 0x02c04a (0x3b8f bytes) 000613 1781228994.732419 trunc from 0x00e20a to 0x01160d (0x3404 bytes) 000614 1781228994.743662 mapwrite 0x01ceb8 thru 0x01d657 (0x7a0 bytes) 000616 1781228994.794175 write 0x010305 thru 0x01d9bd (0xd6b9 bytes) 000617 1781228994.800527 mapread 0x030cd9 thru 0x033a1c (0x2d44 bytes) 000618 1781228994.803540 trunc from 0x040000 to 0x016835 (0x297ca bytes) 000619 1781228994.820244 write 0x010a19 thru 0x01d610 (0xcbf8 bytes) 000620 1781228994.823057 write 0x02e440 thru 0x0358b3 (0x7474 bytes) 000621 1781228994.826141 fallocate 0x024709 thru 0x02f133 (0xaa2b bytes) 000622 1781228994.831808 trunc from 0x0358b4 to 0x017d8a (0x1db29 bytes) 000623 1781228994.861002 mapwrite 0x03da74 thru 0x03ffff (0x258c bytes) 000624 1781228994.935915 read 0x01deb6 thru 0x02bfe8 (0xe133 bytes) 000625 1781228994.964371 write 0x012e61 thru 0x0142ce (0x146e bytes) 000626 1781228994.966206 read 0x0310d6 thru 0x03ffff (0xef2a bytes) 000627 1781228994.967755 read 0x019b97 thru 0x01a727 (0xb91 bytes) 000629 1781228994.969879 trunc from 0x02fe55 to 0x03c8b1 (0xca5d bytes) 000630 1781228994.974914 trunc from 0x040000 to 0x025ea3 (0x1a15c bytes) 000632 1781228994.992175 trunc from 0x01d3b5 to 0x01e771 (0x13bd bytes) 000633 1781228994.997463 mapwrite 0x01c805 thru 0x027f31 (0xb72d bytes) 000634 1781228995.039583 read 0x020944 thru 0x027f31 (0x75ee bytes) 000635 1781228995.040665 trunc from 0x01f85e to 0x027f31 (0x86d4 bytes) 000636 1781228995.047230 punch from 0x00fa2a to 0x01e0a9 (0xe680 bytes) 000637 1781228995.052946 read 0x00bd20 thru 0x012f59 (0x723a bytes) 000638 1781228995.071508 punch from 0x00ba8d to 0x016a5c (0xafd0 bytes) 000639 1781228995.076623 mapread 0x010670 thru 0x0108c9 (0x25a bytes) 000640 1781228995.082793 read 0x027b84 thru 0x027f31 (0x3ae bytes) 000641 1781228995.083787 trunc from 0x0006fb to 0x001611 (0xf17 bytes) 000642 1781228995.094731 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27f32): Mirror 9: 0xe81595f2 (stale) Mirror 12: 0xe81595f2 (stale) Mirror 13: 0xe81595f2 (stale) Mirror 14: 0xe81595f2 (stale) Mirror 15: 0xa5931e34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f32): Mirror 9: 0xe81595f2 (stale) Mirror 12: 0xe81595f2 (stale) Mirror 13: 0xe81595f2 (stale) Mirror 14: 0xe81595f2 (stale) Mirror 15: 0xa5931e34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f32: skipped 000643 1781228995.654402 trunc from 0x018c7e to 0x01bad1 (0x2e54 bytes) 000644 1781228995.682062 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27f32): Mirror 9: 0xe81595f2 (stale) Mirror 12: 0xe81595f2 (stale) Mirror 14: 0xe81595f2 (stale) Mirror 15: 0xa5931e34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f32): Mirror 9: 0xe81595f2 (stale) Mirror 12: 0xe81595f2 (stale) Mirror 14: 0xe81595f2 (stale) Mirror 15: 0xa5931e34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f32: skipped 000646 1781228996.028839 mapread 0x013622 thru 0x0233a2 (0xfd81 bytes) 000647 1781228996.059712 mapwrite 0x01b224 thru 0x023700 (0x84dd bytes) 000648 1781228996.091067 punch from 0x0081e0 to 0x0143e1 (0xc202 bytes) 000649 1781228996.109951 punch from 0x01cb67 to 0x023120 (0x65ba bytes) 000650 1781228996.120204 write 0x02fb54 thru 0x036cd7 (0x7184 bytes) 000652 1781228996.138165 MIRROR_RESYNC 5 mirrors 000653 1781228998.423654 fallocate 0x016070 thru 0x021c17 (0xbba8 bytes) 000655 1781228998.471495 trunc from 0x034559 to 0x034d0c (0x7b4 bytes) 000656 1781228998.477733 fallocate 0x0026bb thru 0x00c0b1 (0x99f7 bytes) 000657 1781228998.485411 mapwrite 0x014554 thru 0x014d14 (0x7c1 bytes) 000658 1781228998.537978 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36cd8): Mirror 9: 0xa3881459 (stale) Mirror 12: 0xa3881459 (stale) Mirror 15: 0xa3881459 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36cd8): Mirror 9: 0xa3881459 (stale) Mirror 12: 0xa3881459 (stale) Mirror 15: 0xa3881459 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36cd8: skipped 000659 1781228999.106723 mapwrite 0x010c20 thru 0x0160f7 (0x54d8 bytes) 000660 1781228999.195104 mapread 0x032b4d thru 0x036cd7 (0x418b bytes) 000661 1781228999.222550 trunc from 0x036cd8 to 0x01ac5f (0x1c078 bytes) 000662 1781228999.267550 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ac60): Mirror 12: 0x53815a34 (stale) Mirror 15: 0x53815a34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ac60): Mirror 12: 0x53815a34 (stale) Mirror 15: 0x53815a34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ac60: skipped 000663 1781228999.632656 mapwrite 0x001578 thru 0x00e990 (0xd419 bytes) 000664 1781228999.697156 trunc from 0x017205 to 0x01ac5f (0x3a5b bytes) 000665 1781228999.703241 punch from 0x000841 to 0x00d0cb (0xc88b bytes) 000666 1781228999.714221 punch from 0x011754 to 0x012662 (0xf0f bytes) 000667 1781228999.718950 trunc from 0x003f69 to 0x013b35 (0xfbcd bytes) 000668 1781228999.728958 trunc from 0x015b87 to 0x015c8b (0x105 bytes) 000669 1781228999.733878 mapread 0x016243 thru 0x01ac5f (0x4a1d bytes) 000670 1781228999.743187 fallocate 0x026a52 thru 0x031239 (0xa7e8 bytes) 000671 1781228999.748631 write 0x00986c thru 0x0136e2 (0x9e77 bytes) 000672 1781228999.766202 mapread 0x01d9a1 thru 0x023a55 (0x60b5 bytes) 000673 1781228999.774183 trunc from 0x03123a to 0x01898b (0x188ae bytes) 000674 1781228999.792577 read 0x016815 thru 0x01898b (0x2177 bytes) 000675 1781228999.793296 mapwrite 0x02c178 thru 0x0330ba (0x6f43 bytes) 000676 1781228999.849336 MIRROR_RESYNC 3 mirrors 000677 1781229000.942270 mapwrite 0x018dc3 thru 0x01c743 (0x3981 bytes) 000678 1781229001.044412 trunc from 0x00213d to 0x0056ff (0x35c3 bytes) 000679 1781229001.050456 mapwrite 0x018e64 thru 0x0255e1 (0xc77e bytes) 000680 1781229001.101452 mapread 0x01bc53 thru 0x01dddd (0x218b bytes) 000681 1781229001.102615 trunc from 0x0330bb to 0x012e71 (0x20249 bytes) 000682 1781229001.131203 trunc from 0x0100d4 to 0x012e71 (0x2d9e bytes) 000683 1781229001.137661 punch from 0x00fb27 to 0x012e71 (0x334b bytes) 000684 1781229001.144216 mapwrite 0x02c4f5 thru 0x03c130 (0xfc3c bytes) 000685 1781229001.208414 punch from 0x01b56a to 0x025c60 (0xa6f7 bytes) 000686 1781229001.218735 trunc from 0x01f4d2 to 0x02a444 (0xaf73 bytes) 000687 1781229001.237645 trunc from 0x03c131 to 0x029081 (0x130af bytes) 000688 1781229001.270398 read 0x013992 thru 0x0176a5 (0x3d14 bytes) 000689 1781229001.285124 fallocate 0x003621 thru 0x007fce (0x49ae bytes) 000690 1781229001.297092 mapread 0x011008 thru 0x011779 (0x772 bytes) 000691 1781229001.319882 trunc from 0x01a91a to 0x01dbfe (0x32e5 bytes) 000692 1781229001.328488 trunc from 0x029082 to 0x0314cc (0x844b bytes) 000693 1781229001.354249 mapwrite 0x023807 thru 0x02e043 (0xa83d bytes) 000695 1781229001.452137 mapwrite 0x00da58 thru 0x0135a8 (0x5b51 bytes) 000696 1781229001.503136 read 0x002d91 thru 0x00e5ed (0xb85d bytes) 000697 1781229001.535941 mapwrite 0x002727 thru 0x00726d (0x4b47 bytes) 000698 1781229001.573529 read 0x013720 thru 0x0144e4 (0xdc5 bytes) 000699 1781229001.578022 write 0x039a15 thru 0x03ffff (0x65eb bytes) 000700 1781229001.583172 read 0x0311cd thru 0x031bdb (0xa0f bytes) 000701 1781229001.584198 punch from 0x039cdc to 0x03bfa3 (0x22c8 bytes) 000703 1781229001.623354 punch from 0x02d9b3 to 0x035852 (0x7ea0 bytes) 000705 1781229001.632091 trunc from 0x040000 to 0x017979 (0x28686 bytes) 000706 1781229001.652201 write 0x01217b thru 0x01d3b3 (0xb239 bytes) 000707 1781229001.655614 trunc from 0x01d3b4 to 0x022f7f (0x5bcc bytes) 000708 1781229001.672718 write 0x02cb50 thru 0x038d9b (0xc24c bytes) 000709 1781229001.674507 write 0x02c31c thru 0x03126c (0x4f51 bytes) 000710 1781229001.680570 read 0x018a8f thru 0x01a9de (0x1f50 bytes) 000712 1781229001.689187 mapwrite 0x005c6c thru 0x015248 (0xf5dd bytes) 000713 1781229001.738679 mapread 0x02aff5 thru 0x02c797 (0x17a3 bytes) 000714 1781229001.749820 trunc from 0x0386d6 to 0x038d9b (0x6c6 bytes) 000715 1781229001.756797 punch from 0x01ddfa to 0x02c5f7 (0xe7fe bytes) 000716 1781229001.762257 read 0x012be2 thru 0x017f8b (0x53aa bytes) 000717 1781229001.763596 mapwrite 0x0141a9 thru 0x01bdaf (0x7c07 bytes) 000718 1781229001.787696 punch from 0x028e57 to 0x034262 (0xb40c bytes) 000719 1781229001.792702 write 0x025397 thru 0x02e5d6 (0x9240 bytes) 000720 1781229001.804937 write 0x01d7e0 thru 0x01e688 (0xea9 bytes) 000721 1781229001.806489 trunc from 0x038d9c to 0x027dbe (0x10fdd bytes) 000722 1781229001.831617 punch from 0x007430 to 0x00785f (0x430 bytes) 000723 1781229001.837692 fallocate 0x00e484 thru 0x01ba0a (0xd587 bytes) 000724 1781229001.843482 mapwrite 0x00065a thru 0x0057ed (0x5194 bytes) 000725 1781229001.869925 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27dbf): Mirror 12: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27dbf): Mirror 12: 0x1a76aa59 (stale) Mirror 15: 0x1a76aa59 (stale) CRC-32 checksum value for chunk [0, 0x27dbf): Mirror 1: 0xfb733820 Mirror 16: 0xfb733820 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27dbf: skipped 000726 1781229003.058154 MIRROR_RESYNC 4 mirrors 000727 1781229004.144112 mapread 0x01ddfc thru 0x027dbe (0x9fc3 bytes) 000728 1781229004.181092 punch from 0x023271 to 0x0274f4 (0x4284 bytes) 000729 1781229004.265767 fallocate 0x031ccb thru 0x039580 (0x78b6 bytes) 000730 1781229004.278923 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39581): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39581): Mirror 12: 0x272d71ec (stale) Mirror 15: 0x272d71ec (stale) Mirror 16: 0x272d71ec (stale) CRC-32 checksum value for chunk [0, 0x39581): Mirror 1: 0xbf96737 Mirror 17: 0xbf96737 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39581: skipped 000732 1781229005.333291 trunc from 0x039581 to 0x0221ad (0x173d3 bytes) 000733 1781229005.423272 mapwrite 0x012edb thru 0x01c449 (0x956f bytes) 000734 1781229005.500966 punch from 0x01cd72 to 0x0221ad (0x543c bytes) 000736 1781229005.519027 fallocate 0x009b54 thru 0x012d27 (0x91d4 bytes) 000737 1781229005.529827 MIRROR_RESYNC 5 mirrors 000738 1781229006.761935 write 0x024c45 thru 0x024dd1 (0x18d bytes) 000739 1781229006.803331 write 0x03566d thru 0x03ffff (0xa993 bytes) 000740 1781229006.805085 write 0x01efd9 thru 0x0277e2 (0x880a bytes) 000741 1781229006.815930 mapwrite 0x0188c6 thru 0x01cbb6 (0x42f1 bytes) 000742 1781229006.841495 read 0x03f081 thru 0x03ffff (0xf7f bytes) 000743 1781229006.850032 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd58c5db6 (stale) Mirror 15: 0xd58c5db6 (stale) Mirror 16: 0xd58c5db6 (stale) Mirror 17: 0xd58c5db6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x331745f4 Mirror 18: 0x331745f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000744 1781229007.684057 MIRROR_RESYNC 6 mirrors 000745 1781229009.057452 mapread 0x021108 thru 0x0265c8 (0x54c1 bytes) 000746 1781229009.095415 punch from 0x0155f2 to 0x0174c0 (0x1ecf bytes) 000747 1781229009.155789 mapwrite 0x03bd18 thru 0x03ffff (0x42e8 bytes) 000748 1781229009.195674 punch from 0x0095ea to 0x00ff70 (0x6987 bytes) 000749 1781229009.212002 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x331745f4 (stale) Mirror 15: 0x331745f4 (stale) Mirror 17: 0x331745f4 (stale) Mirror 18: 0x331745f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x331745f4 (stale) Mirror 15: 0x331745f4 (stale) Mirror 17: 0x331745f4 (stale) Mirror 18: 0x331745f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000750 1781229009.770848 read 0x014227 thru 0x0168ce (0x26a8 bytes) 000751 1781229009.806766 mapwrite 0x01b8db thru 0x024a9c (0x91c2 bytes) 000752 1781229009.902206 punch from 0x00e93c to 0x019b8d (0xb252 bytes) 000753 1781229009.907776 write 0x035b4f thru 0x03ffff (0xa4b1 bytes) 000754 1781229009.933116 trunc from 0x040000 to 0x017933 (0x286cc bytes) 000755 1781229009.965546 trunc from 0x017934 to 0x03a3ed (0x22aba bytes) 000756 1781229010.011171 trunc from 0x00a975 to 0x0138d5 (0x8f61 bytes) 000757 1781229010.021716 read 0x037235 thru 0x03a3ed (0x31b9 bytes) 000758 1781229010.040872 read 0x0286de thru 0x02aaaf (0x23d2 bytes) 000759 1781229010.041530 punch from 0x018303 to 0x020381 (0x807f bytes) 000761 1781229010.048894 trunc from 0x02d752 to 0x02e379 (0xc28 bytes) 000762 1781229010.061748 write 0x005090 thru 0x005d78 (0xce9 bytes) 000763 1781229010.067206 write 0x0036ae thru 0x0086ce (0x5021 bytes) 000765 1781229010.072848 trunc from 0x00508d to 0x00f707 (0xa67b bytes) 000766 1781229010.113107 punch from 0x01b31f to 0x0211fa (0x5edc bytes) 000767 1781229010.126565 fallocate 0x01c3c1 thru 0x01cf10 (0xb50 bytes) 000768 1781229010.139560 MIRROR_RESYNC 5 mirrors 000769 1781229011.378677 mapread 0x01cdd1 thru 0x022370 (0x55a0 bytes) 000770 1781229011.411282 trunc from 0x03a3ee to 0x02b8b5 (0xeb38 bytes) 000771 1781229011.497090 write 0x039101 thru 0x03ffff (0x6eff bytes) 000772 1781229011.498576 read 0x020499 thru 0x02462f (0x4197 bytes) 000773 1781229011.539785 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x9e768ac7 (stale) Mirror 15: 0x9e768ac7 (stale) Mirror 17: 0x9e768ac7 (stale) Mirror 18: 0x9e768ac7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67a966c4 Mirror 19: 0x67a966c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000774 1781229012.920144 fallocate 0x00be83 thru 0x01a838 (0xe9b6 bytes) 000776 1781229012.987894 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x9e768ac7 (stale) Mirror 15: 0x9e768ac7 (stale) Mirror 17: 0x9e768ac7 (stale) Mirror 18: 0x9e768ac7 (stale) Mirror 19: 0x67a966c4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67a966c4 Mirror 20: 0x67a966c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000777 1781229014.113118 write 0x01d54d thru 0x02a21e (0xccd2 bytes) 000778 1781229014.164037 read 0x01dcbc thru 0x023e82 (0x61c7 bytes) 000779 1781229014.165010 mapwrite 0x01254a thru 0x013807 (0x12be bytes) 000780 1781229014.189608 trunc from 0x011529 to 0x01c01c (0xaaf4 bytes) 000781 1781229014.194493 write 0x032af0 thru 0x03ffff (0xd510 bytes) 000782 1781229014.216175 fallocate 0x028245 thru 0x02b384 (0x3140 bytes) 000783 1781229014.222801 mapread 0x0133d5 thru 0x01e463 (0xb08f bytes) 000784 1781229014.252198 mapread 0x038f68 thru 0x039e37 (0xed0 bytes) 000785 1781229014.253966 read 0x030c03 thru 0x03c3ee (0xb7ec bytes) 000786 1781229014.256083 trunc from 0x0151ed to 0x01e8fb (0x970f bytes) 000787 1781229014.263768 fallocate 0x014770 thru 0x01c216 (0x7aa7 bytes) 000788 1781229014.268729 punch from 0x004fc8 to 0x007766 (0x279f bytes) 000789 1781229014.274047 write 0x02ed11 thru 0x02ff8c (0x127c bytes) 000790 1781229014.275483 punch from 0x03e2ea to 0x03ffff (0x1d16 bytes) 000791 1781229014.288872 mapread 0x015126 thru 0x016c26 (0x1b01 bytes) 000792 1781229014.297642 punch from 0x0160a1 to 0x01d008 (0x6f68 bytes) 000793 1781229014.306741 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x9e768ac7 (stale) Mirror 17: 0x9e768ac7 (stale) Mirror 18: 0x9e768ac7 (stale) Mirror 19: 0x67a966c4 (stale) Mirror 20: 0x67a966c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x9e768ac7 (stale) Mirror 17: 0x9e768ac7 (stale) Mirror 18: 0x9e768ac7 (stale) Mirror 19: 0x67a966c4 (stale) Mirror 20: 0x67a966c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1781229014.863857 mapwrite 0x03ac29 thru 0x03ffff (0x53d7 bytes) 000795 1781229014.904409 punch from 0x010058 to 0x01accc (0xac75 bytes) 000796 1781229014.910571 trunc from 0x03fb8e to 0x03ffff (0x472 bytes) 000797 1781229014.915245 write 0x01274d thru 0x017085 (0x4939 bytes) 000798 1781229014.935009 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x9e768ac7 (stale) Mirror 17: 0x9e768ac7 (stale) Mirror 18: 0x9e768ac7 (stale) Mirror 19: 0x67a966c4 (stale) Mirror 20: 0x67a966c4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x55f9b4a6 Mirror 21: 0x55f9b4a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000799 1781229015.996945 mapread 0x03e2f8 thru 0x03ffff (0x1d08 bytes) 000800 1781229016.007481 punch from 0x022dfa to 0x029c8a (0x6e91 bytes) 000801 1781229016.053564 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x9e768ac7 (stale) Mirror 17: 0x9e768ac7 (stale) Mirror 18: 0x9e768ac7 (stale) Mirror 19: 0x67a966c4 (stale) Mirror 20: 0x67a966c4 (stale) Mirror 21: 0x55f9b4a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8facf4bf Mirror 22: 0x8facf4bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000802 1781229017.627597 trunc from 0x02f5c9 to 0x03a3d4 (0xae0c bytes) 000804 1781229017.689358 MIRROR_RESYNC 8 mirrors 000805 1781229020.372407 write 0x019a0c thru 0x024a7e (0xb073 bytes) 000806 1781229020.439118 write 0x002e27 thru 0x011894 (0xea6e bytes) 000807 1781229020.469825 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x45a81ae1 (stale) Mirror 17: 0x45a81ae1 (stale) Mirror 18: 0x45a81ae1 (stale) Mirror 19: 0x45a81ae1 (stale) Mirror 20: 0x45a81ae1 (stale) Mirror 22: 0x45a81ae1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x45a81ae1 (stale) Mirror 17: 0x45a81ae1 (stale) Mirror 18: 0x45a81ae1 (stale) Mirror 19: 0x45a81ae1 (stale) Mirror 20: 0x45a81ae1 (stale) Mirror 22: 0x45a81ae1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1781229021.265996 fallocate 0x023f98 thru 0x02d0fd (0x9166 bytes) 000811 1781229021.277496 fallocate 0x00a951 thru 0x0172e9 (0xc999 bytes) 000812 1781229021.291559 fallocate 0x023852 thru 0x031b65 (0xe314 bytes) 000813 1781229021.303078 fallocate 0x027230 thru 0x02bb20 (0x48f1 bytes) 000814 1781229021.312658 trunc from 0x040000 to 0x007fa2 (0x3805d bytes) 000815 1781229021.361676 mapwrite 0x0090fa thru 0x00a474 (0x137b bytes) 000816 1781229021.439230 mapwrite 0x018d8d thru 0x027fda (0xf24e bytes) 000817 1781229021.537088 fallocate 0x03a870 thru 0x03d9e9 (0x317a bytes) 000818 1781229021.550505 trunc from 0x027fdb to 0x0089bd (0x1f61d bytes) 000819 1781229021.582747 mapread 0x007e65 thru 0x0089bd (0xb59 bytes) 000820 1781229021.603021 write 0x039932 thru 0x03ac9b (0x136a bytes) 000821 1781229021.604463 mapread 0x0381e3 thru 0x03a31c (0x213a bytes) 000822 1781229021.633107 read 0x01a226 thru 0x02950e (0xf2e9 bytes) 000823 1781229021.644772 fallocate 0x0138ea thru 0x0211dd (0xd8f4 bytes) 000824 1781229021.655404 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac9c): Mirror 17: 0xad2a5eea (stale) Mirror 18: 0xad2a5eea (stale) Mirror 19: 0xad2a5eea (stale) Mirror 20: 0xad2a5eea (stale) Mirror 22: 0xad2a5eea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac9c): Mirror 17: 0xad2a5eea (stale) Mirror 18: 0xad2a5eea (stale) Mirror 19: 0xad2a5eea (stale) Mirror 20: 0xad2a5eea (stale) Mirror 22: 0xad2a5eea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac9c: skipped 000825 1781229022.213713 trunc from 0x009230 to 0x017386 (0xe157 bytes) 000826 1781229022.227632 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ac9c): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac9c): Mirror 17: 0xad2a5eea (stale) Mirror 18: 0xad2a5eea (stale) Mirror 19: 0xad2a5eea (stale) Mirror 20: 0xad2a5eea (stale) Mirror 22: 0xad2a5eea (stale) CRC-32 checksum value for chunk [0, 0x3ac9c): Mirror 1: 0xd3da9fb1 Mirror 23: 0xd3da9fb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac9c: skipped 000827 1781229023.504007 trunc from 0x03905c to 0x03ac9b (0x1c40 bytes) 000828 1781229023.550266 write 0x016fd9 thru 0x01d768 (0x6790 bytes) 000829 1781229023.576960 write 0x0075e0 thru 0x010b33 (0x9554 bytes) 000830 1781229023.601565 MIRROR_RESYNC 7 mirrors 000831 1781229025.801773 mapwrite 0x03dacf thru 0x03ffff (0x2531 bytes) 000832 1781229025.958215 read 0x0284a4 thru 0x037df4 (0xf951 bytes) 000833 1781229025.970765 write 0x018aeb thru 0x0222cb (0x97e1 bytes) 000834 1781229025.989159 trunc from 0x040000 to 0x02a404 (0x15bfb bytes) 000836 1781229026.008091 mapread 0x004c5b thru 0x00976e (0x4b14 bytes) 000838 1781229026.018476 fallocate 0x01678a thru 0x025e72 (0xf6e9 bytes) 000839 1781229026.022889 punch from 0x010272 to 0x018cae (0x8a3d bytes) 000840 1781229026.038066 read 0x00308c thru 0x004024 (0xf99 bytes) 000842 1781229026.039509 read 0x015a64 thru 0x01c110 (0x66ad bytes) 000843 1781229026.052114 trunc from 0x014919 to 0x01ee02 (0xa4ea bytes) 000844 1781229026.057080 write 0x017ea1 thru 0x0200c4 (0x8224 bytes) 000845 1781229026.065371 punch from 0x017dd7 to 0x027a88 (0xfcb2 bytes) 000846 1781229026.082869 punch from 0x025649 to 0x02a404 (0x4dbc bytes) 000847 1781229026.090265 trunc from 0x02a405 to 0x036995 (0xc591 bytes) 000848 1781229026.115130 mapread 0x0003ff thru 0x0013e7 (0xfe9 bytes) 000849 1781229026.117623 mapread 0x02b115 thru 0x02e6e0 (0x35cc bytes) 000850 1781229026.130933 fallocate 0x027b4e thru 0x032e09 (0xb2bc bytes) 000852 1781229026.140467 read 0x011df1 thru 0x016c85 (0x4e95 bytes) 000853 1781229026.154899 mapread 0x0018a1 thru 0x00a1f8 (0x8958 bytes) 000854 1781229026.157401 mapwrite 0x000770 thru 0x003c87 (0x3518 bytes) 000855 1781229026.181784 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36996): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36996): Mirror 17: 0x7cf937ea (stale) Mirror 18: 0x7cf937ea (stale) Mirror 19: 0x7cf937ea (stale) Mirror 20: 0x7cf937ea (stale) Mirror 22: 0x7cf937ea (stale) Mirror 23: 0x7cf937ea (stale) CRC-32 checksum value for chunk [0, 0x36996): Mirror 1: 0x4e6d3c0f Mirror 24: 0x4e6d3c0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36996: skipped 000857 1781229027.241664 punch from 0x000a23 to 0x0027e6 (0x1dc4 bytes) 000859 1781229027.288559 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36996): Mirror 17: 0x7cf937ea (stale) Mirror 18: 0x7cf937ea (stale) Mirror 19: 0x7cf937ea (stale) Mirror 20: 0x7cf937ea (stale) Mirror 22: 0x7cf937ea (stale) Mirror 24: 0x4e6d3c0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36996): Mirror 17: 0x7cf937ea (stale) Mirror 18: 0x7cf937ea (stale) Mirror 19: 0x7cf937ea (stale) Mirror 20: 0x7cf937ea (stale) Mirror 22: 0x7cf937ea (stale) Mirror 24: 0x4e6d3c0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36996: skipped 000860 1781229027.717809 punch from 0x00b769 to 0x00f51b (0x3db3 bytes) 000861 1781229027.723775 mapread 0x00bccc thru 0x016dae (0xb0e3 bytes) 000862 1781229027.750107 mapwrite 0x03d0fd thru 0x03ffff (0x2f03 bytes) 000863 1781229027.813050 mapwrite 0x01a306 thru 0x02863b (0xe336 bytes) 000864 1781229027.888901 mapwrite 0x034c75 thru 0x03b04d (0x63d9 bytes) 000865 1781229027.932400 write 0x021337 thru 0x02dece (0xcb98 bytes) 000866 1781229027.934010 trunc from 0x040000 to 0x026dd2 (0x1922d bytes) 000867 1781229027.964747 punch from 0x00f8c3 to 0x015d0e (0x644c bytes) 000868 1781229027.969828 trunc from 0x016105 to 0x021819 (0xb715 bytes) 000869 1781229027.987588 punch from 0x02230e to 0x026dd2 (0x4ac5 bytes) 000871 1781229027.994023 trunc from 0x026dd3 to 0x01deda (0x8ef8 bytes) 000872 1781229028.010637 fallocate 0x018e62 thru 0x01b646 (0x27e5 bytes) 000873 1781229028.017822 trunc from 0x01dedb to 0x03a1e9 (0x1c30f bytes) 000874 1781229028.050448 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1ea): Mirror 17: 0x9eddbbde (stale) Mirror 18: 0x9eddbbde (stale) Mirror 19: 0x9eddbbde (stale) Mirror 20: 0x9eddbbde (stale) Mirror 22: 0x9eddbbde (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1ea): Mirror 17: 0x9eddbbde (stale) Mirror 18: 0x9eddbbde (stale) Mirror 19: 0x9eddbbde (stale) Mirror 20: 0x9eddbbde (stale) Mirror 22: 0x9eddbbde (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1ea: skipped 000875 1781229028.577189 mapwrite 0x01bf56 thru 0x01d9f5 (0x1aa0 bytes) 000876 1781229028.611390 trunc from 0x03a1ea to 0x00f69c (0x2ab4d bytes) 000877 1781229028.635224 MIRROR_RESYNC 6 mirrors 000878 1781229030.526742 fallocate 0x02942a thru 0x036aa5 (0xd67c bytes) 000879 1781229030.598639 write 0x01a7ed thru 0x026c07 (0xc41b bytes) 000880 1781229030.605878 read 0x025e58 thru 0x026c07 (0xdb0 bytes) 000881 1781229030.607215 mapread 0x01ccc4 thru 0x0200de (0x341b bytes) 000882 1781229030.613389 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26c08): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c08): Mirror 17: 0xc0b31457 (stale) Mirror 18: 0xc0b31457 (stale) Mirror 19: 0xc0b31457 (stale) Mirror 20: 0xc0b31457 (stale) Mirror 22: 0xc0b31457 (stale) CRC-32 checksum value for chunk [0, 0x26c08): Mirror 1: 0xefa31b35 Mirror 23: 0xefa31b35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c08: skipped 000883 1781229031.935349 MIRROR_RESYNC 7 mirrors 000885 1781229033.109906 read 0x012db5 thru 0x01ec51 (0xbe9d bytes) 000886 1781229033.119654 trunc from 0x008b64 to 0x012c83 (0xa120 bytes) 000887 1781229033.160432 punch from 0x020b3f to 0x026c07 (0x60c9 bytes) 000888 1781229033.165987 trunc from 0x026c08 to 0x02e2f0 (0x76e9 bytes) 000889 1781229033.187799 punch from 0x016f48 to 0x01bea6 (0x4f5f bytes) 000890 1781229033.193297 mapwrite 0x019e1f thru 0x027f9f (0xe181 bytes) 000892 1781229033.240614 punch from 0x0103a5 to 0x014eb7 (0x4b13 bytes) 000893 1781229033.245700 write 0x029d9a thru 0x03829f (0xe506 bytes) 000894 1781229033.247817 trunc from 0x006f27 to 0x011af5 (0xabcf bytes) 000895 1781229033.253165 fallocate 0x016fa2 thru 0x01b9e8 (0x4a47 bytes) 000896 1781229033.271252 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x382a0): Mirror 17: 0xe313916 (stale) Mirror 18: 0xe313916 (stale) Mirror 19: 0xe313916 (stale) Mirror 20: 0xe313916 (stale) Mirror 22: 0xe313916 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382a0): Mirror 17: 0xe313916 (stale) Mirror 18: 0xe313916 (stale) Mirror 19: 0xe313916 (stale) Mirror 20: 0xe313916 (stale) Mirror 22: 0xe313916 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382a0: skipped 000897 1781229033.745897 read 0x019528 thru 0x027107 (0xdbe0 bytes) 000898 1781229033.757786 read 0x019990 thru 0x02991e (0xff8f bytes) 000899 1781229033.787528 write 0x02e95b thru 0x035fa2 (0x7648 bytes) 000900 1781229033.793035 mapwrite 0x016b70 thru 0x0263a5 (0xf836 bytes) 000901 1781229033.829813 read 0x001329 thru 0x004b9e (0x3876 bytes) 000902 1781229033.830807 mapwrite 0x0252ee thru 0x03374d (0xe460 bytes) 000903 1781229033.873888 fallocate 0x01164c thru 0x01cabe (0xb473 bytes) 000904 1781229033.891727 punch from 0x00c459 to 0x01833b (0xbee3 bytes) 000905 1781229033.900628 read 0x00fcd6 thru 0x013cd1 (0x3ffc bytes) 000906 1781229033.922238 mapwrite 0x02da72 thru 0x03bc51 (0xe1e0 bytes) 000907 1781229033.990397 read 0x010693 thru 0x017e7f (0x77ed bytes) 000908 1781229033.991137 write 0x013d93 thru 0x021382 (0xd5f0 bytes) 000909 1781229033.992691 read 0x02c8cd thru 0x039e97 (0xd5cb bytes) 000911 1781229033.998564 punch from 0x011d22 to 0x015aa2 (0x3d81 bytes) 000912 1781229034.025566 trunc from 0x0035f8 to 0x005296 (0x1c9f bytes) 000913 1781229034.031016 read 0x033e09 thru 0x0349dd (0xbd5 bytes) 000914 1781229034.031691 fallocate 0x022574 thru 0x02b37a (0x8e07 bytes) 000915 1781229034.043470 punch from 0x012e5d to 0x015704 (0x28a8 bytes) 000916 1781229034.052761 trunc from 0x01a177 to 0x023c27 (0x9ab1 bytes) 000918 1781229034.069884 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3bc52): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc52): Mirror 17: 0x2c7cdb4 (stale) Mirror 18: 0x2c7cdb4 (stale) Mirror 19: 0x2c7cdb4 (stale) Mirror 20: 0x2c7cdb4 (stale) Mirror 22: 0x2c7cdb4 (stale) CRC-32 checksum value for chunk [0, 0x3bc52): Mirror 1: 0x2563889 Mirror 23: 0x2563889 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc52: skipped 000919 1781229035.662783 punch from 0x01503c to 0x024770 (0xf735 bytes) 000920 1781229035.719190 fallocate 0x024864 thru 0x03389b (0xf038 bytes) 000921 1781229035.727594 read 0x01e8d4 thru 0x028865 (0x9f92 bytes) 000922 1781229035.737531 trunc from 0x03bc52 to 0x00fb36 (0x2c11b bytes) 000923 1781229035.785683 write 0x03a7a9 thru 0x03e2c0 (0x3b18 bytes) 000924 1781229035.787092 punch from 0x01a231 to 0x02014b (0x5f1b bytes) 000925 1781229035.802831 mapwrite 0x038c07 thru 0x03ffff (0x73f9 bytes) 000928 1781229035.876573 read 0x0254d5 thru 0x028ea6 (0x39d2 bytes) 000929 1781229035.895998 punch from 0x0181fa to 0x01b0ff (0x2f06 bytes) 000930 1781229035.901696 write 0x0274d7 thru 0x02948f (0x1fb9 bytes) 000931 1781229035.903619 trunc from 0x01113c to 0x0195f1 (0x84b6 bytes) 000932 1781229035.912109 fallocate 0x033120 thru 0x03ffff (0xcee0 bytes) 000933 1781229035.919018 trunc from 0x03b1e4 to 0x03ffff (0x4e1c bytes) 000934 1781229035.923788 punch from 0x02f6f0 to 0x0374d4 (0x7de5 bytes) 000935 1781229035.933862 read 0x0353ee thru 0x03ea52 (0x9665 bytes) 000936 1781229035.960522 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc16eb464 (stale) Mirror 18: 0xc16eb464 (stale) Mirror 19: 0xc16eb464 (stale) Mirror 20: 0xc16eb464 (stale) Mirror 22: 0xc16eb464 (stale) Mirror 23: 0x88d00448 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1629b35b Mirror 24: 0x1629b35b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000937 1781229037.242745 trunc from 0x040000 to 0x02fa25 (0x105da bytes) 000940 1781229037.336793 trunc from 0x02fa26 to 0x0272c3 (0x8762 bytes) 000941 1781229037.369602 trunc from 0x0272c4 to 0x010903 (0x169c0 bytes) 000942 1781229037.390663 trunc from 0x001f71 to 0x010903 (0xe993 bytes) 000943 1781229037.413757 MIRROR_RESYNC 8 mirrors 000944 1781229039.021122 mapwrite 0x034ddc thru 0x03ffff (0xb224 bytes) 000945 1781229039.189972 trunc from 0x00fa80 to 0x017f87 (0x8508 bytes) 000946 1781229039.202670 fallocate 0x026345 thru 0x02beb7 (0x5b73 bytes) 000947 1781229039.214288 mapwrite 0x00be64 thru 0x017b39 (0xbcd6 bytes) 000948 1781229039.299200 mapread 0x0178b4 thru 0x01e5a5 (0x6cf2 bytes) 000949 1781229039.355711 fallocate 0x028ade thru 0x02ecbc (0x61df bytes) 000950 1781229039.373726 trunc from 0x01278a to 0x021ea9 (0xf720 bytes) 000953 1781229039.398453 fallocate 0x03ac21 thru 0x03debc (0x329c bytes) 000954 1781229039.406834 fallocate 0x01236f thru 0x017e96 (0x5b28 bytes) 000955 1781229039.428451 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x99666196 (stale) Mirror 19: 0x99666196 (stale) Mirror 20: 0x99666196 (stale) Mirror 22: 0x99666196 (stale) Mirror 23: 0x99666196 (stale) Mirror 24: 0x99666196 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x99666196 (stale) Mirror 19: 0x99666196 (stale) Mirror 20: 0x99666196 (stale) Mirror 22: 0x99666196 (stale) Mirror 23: 0x99666196 (stale) Mirror 24: 0x99666196 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000956 1781229040.015222 trunc from 0x040000 to 0x028752 (0x178ad bytes) 000957 1781229040.039150 trunc from 0x00c69e to 0x0149ce (0x8331 bytes) 000958 1781229040.046981 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28753): Mirror 17: 0xc4a70cb (stale) Mirror 19: 0xc4a70cb (stale) Mirror 20: 0xc4a70cb (stale) Mirror 23: 0xc4a70cb (stale) Mirror 24: 0xc4a70cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28753): Mirror 17: 0xc4a70cb (stale) Mirror 19: 0xc4a70cb (stale) Mirror 20: 0xc4a70cb (stale) Mirror 23: 0xc4a70cb (stale) Mirror 24: 0xc4a70cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28753: skipped 000959 1781229040.474485 punch from 0x01a0e7 to 0x01d33d (0x3257 bytes) 000961 1781229040.481634 trunc from 0x028753 to 0x007f97 (0x207bb bytes) 000962 1781229040.504714 mapread 0x003af5 thru 0x0076a1 (0x3bad bytes) 000963 1781229040.521732 read 0x0035af thru 0x0067a9 (0x31fb bytes) 000964 1781229040.524528 write 0x0347e2 thru 0x03ffff (0xb81e bytes) 000965 1781229040.527153 fallocate 0x038e7d thru 0x03ffff (0x7183 bytes) 000966 1781229040.533625 mapread 0x004249 thru 0x00da08 (0x97c0 bytes) 000967 1781229040.547150 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x99666196 (stale) Mirror 19: 0x99666196 (stale) Mirror 20: 0x99666196 (stale) Mirror 23: 0x99666196 (stale) Mirror 24: 0x99666196 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4fcf9de0 Mirror 25: 0x4fcf9de0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000968 1781229041.526020 MIRROR_RESYNC 7 mirrors 000969 1781229043.532078 fallocate 0x01576e thru 0x020ea9 (0xb73c bytes) 000970 1781229043.577595 trunc from 0x01bb00 to 0x01ebc1 (0x30c2 bytes) 000971 1781229043.583726 mapread 0x026a08 thru 0x02ea85 (0x807e bytes) 000972 1781229043.609136 mapwrite 0x02f7c5 thru 0x0382d7 (0x8b13 bytes) 000973 1781229043.665329 write 0x027647 thru 0x035c8b (0xe645 bytes) 000974 1781229043.667174 mapwrite 0x0267fc thru 0x02d916 (0x711b bytes) 000975 1781229043.727659 read 0x037495 thru 0x03ffff (0x8b6b bytes) 000976 1781229043.743020 mapread 0x01e0b3 thru 0x01e819 (0x767 bytes) 000977 1781229043.766143 mapwrite 0x0343b9 thru 0x038269 (0x3eb1 bytes) 000978 1781229043.791044 punch from 0x02e364 to 0x03c76c (0xe409 bytes) 000979 1781229043.798109 write 0x0128dd thru 0x0181e4 (0x5908 bytes) 000980 1781229043.813492 mapread 0x034ab6 thru 0x03a632 (0x5b7d bytes) 000981 1781229043.855113 mapwrite 0x025559 thru 0x029171 (0x3c19 bytes) 000982 1781229043.880260 write 0x0298f5 thru 0x02d373 (0x3a7f bytes) 000983 1781229043.881625 mapread 0x02a0ba thru 0x036854 (0xc79b bytes) 000984 1781229043.885694 mapwrite 0x012866 thru 0x018959 (0x60f4 bytes) 000985 1781229043.953163 mapwrite 0x014f37 thru 0x02198d (0xca57 bytes) 000986 1781229044.018977 mapread 0x033f59 thru 0x03ffff (0xc0a7 bytes) 000987 1781229044.026619 mapwrite 0x035a66 thru 0x03b649 (0x5be4 bytes) 000988 1781229044.120009 punch from 0x023ff2 to 0x031be8 (0xdbf7 bytes) 000989 1781229044.154805 trunc from 0x040000 to 0x02a955 (0x156aa bytes) 000990 1781229044.217457 trunc from 0x02a956 to 0x0314f8 (0x6ba3 bytes) 000991 1781229044.251792 mapwrite 0x01e94d thru 0x020e35 (0x24e9 bytes) 000992 1781229044.267581 write 0x0318f4 thru 0x03ffff (0xe70c bytes) 000993 1781229044.287148 punch from 0x0149df to 0x01aff9 (0x661b bytes) 000996 1781229044.308070 trunc from 0x03f669 to 0x03ffff (0x997 bytes) 000997 1781229044.352512 trunc from 0x040000 to 0x01500e (0x2aff1 bytes) 000998 1781229044.388221 trunc from 0x00fbd5 to 0x01500e (0x543a bytes) 000999 1781229044.421131 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1500f): Mirror 19: 0x49f0e033 (stale) Mirror 20: 0x49f0e033 (stale) Mirror 23: 0x49f0e033 (stale) Mirror 24: 0x49f0e033 (stale) Mirror 25: 0x49f0e033 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1500f): Mirror 19: 0x49f0e033 (stale) Mirror 20: 0x49f0e033 (stale) Mirror 23: 0x49f0e033 (stale) Mirror 24: 0x49f0e033 (stale) Mirror 25: 0x49f0e033 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1500f: skipped 001000 1781229044.768689 punch from 0x011f5a to 0x01500e (0x30b5 bytes) All operations completed A-OK! PASS 70a (133s) == sanity-flr test 71: check mirror extend parent fid ==== 21:50:53 (1781229053) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x65:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 21:51:03 (1781229063) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227794 s, 9.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147621 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ff:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 21:51:11 (1781229071) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8366304 .. done Extending file size to 3586944 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9117216 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8737376 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9199520 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9204608 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8976352 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61653 61654 61655 61657 61659 done 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) 192.168.202.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:52:33 (1781229153) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5032768 .. Extending file size to 3716832 .. Extending file size to 7325248 .. Extending file size to 8882880 .. Extending file size to 8753248 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8986432 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8959968 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9001856 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9032256 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63313 63314 63315 63318 63320 done 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) 192.168.202.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:53:52 (1781229232) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2496 1268808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2048 1269256 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24300 3582720 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18728 3588292 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43028 7171012 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2496 1268808 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2048 1269256 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24300 3582720 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18728 3588292 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43028 7171012 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145448 s, 14.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167396 s, 12.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x570bbca (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:54:13 (1781229253) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0794715 s, 26.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 21:54:20 (1781229260) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:54:29 (1781229269) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21125 s, 9.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:54:41 (1781229281) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0786284 s, 26.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:54:51 (1781229291) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0950904 s, 22.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31e:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:55:24 (1781229324) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889915 s, 11.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:56:20 (1781229380) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x320:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118284 s, 17.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x320:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x321:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x320:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x321:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:57:15 (1781229435) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg218-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg218-server: Pool lustre.test_204f created Adding targets to pool oleg218-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg218-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0635 s, 33.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x322:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x322:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg218-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg218-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg218-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:58:26 (1781229506) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:58:35 (1781229515) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:58:43 (1781229523) oleg218-server: Pool lustre.test_206 created Adding targets to pool oleg218-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg218-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg218-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg218-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg218-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:59:12 (1781229552) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723747 s, 14.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348636 s, 30.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x88ae24aa Mirror 3: 0x88ae24aa PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:59:21 (1781229561) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:59:25 (1781229565) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:59:28 (1781229568) Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0412911 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x18e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x950000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x950000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x950000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x920000 Read this (4 bytes): BBBB 192.168.202.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg218-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:00:06 (1781229606) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0382165 s, 214 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:00:14 (1781229614) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504818 s, 20.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f3 lmm_fid: [0x200000405:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f3 lmm_fid: [0x200000405:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x32c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611635 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 22:00:15.000000000 -0400 Modify: 2026-06-11 22:00:16.000000000 -0400 Change: 2026-06-11 22:00:17.000000000 -0400 Birth: 2026-06-11 22:00:15.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:00:22 (1781229622) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407753 s, 25.7 MB/s oleg218-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg218-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg218-server: O/280000401/d12/32c: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:00:45 (1781229645) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.167178 s, 245 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0394533 s, 104 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 22:00 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 22:00 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:00:54 (1781229654) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:01:03 (1781229663) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1839 sec =========== 22:01:13 (1781229673) === sanity-flr: start cleanup 22:01:15 (1781229675) === === sanity-flr: finish cleanup 22:01:22 (1781229682) ===