-----============= acceptance-small: sanity-flr ============----- Fri Aug 21 05:23:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg440-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 05:23:21 (1787304201) === oleg440-client.virtnet: executing check_config_client /mnt/lustre oleg440-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg440-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff899a86043800.idle_timeout=debug osc.lustre-OST0001-osc-ffff899a86043800.idle_timeout=debug disable quota as required oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 05:23:52 (1787304232) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7637] == sanity-flr test 0a: lfs mirror create with -N option == 05:23:54 (1787304234) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:24:13 (1787304253) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:24:17 (1787304257) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:24:20 (1787304260) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:24:50 (1787304290) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:24:55 (1787304295) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 05:24:59 (1787304299) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff899a86043800.stats=clear osc.lustre-OST0001-osc-ffff899a86043800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:25:08 (1787304308) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:25:20 (1787304320) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:25:28 (1787304328) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 05:25:31 (1787304331) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:25:38 (1787304338) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 05:25:48 (1787304348) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:25:55 (1787304355) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:25:58 (1787304358) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:26:07 (1787304367) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:26:17 (1787304377) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:26:24 (1787304384) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.24099 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18431 s, 5.7 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:26:34 (1787304394) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.579451 s, 34.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 05:26:43 (1787304403) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:26:54 (1787304414) Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (140s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:29:14 (1787304554) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.164326 s, 19.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:29:54 (1787304594) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:31:18 (1787304678) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.49415 s, 11.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 21 05:31 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.729s user 0m0.001s sys 0m0.194s Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff899a86043800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.024s user 0m0.000s sys 0m0.183s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.890s user 0m0.001s sys 0m0.215s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff899a86043800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:32:55 (1787304775) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:32:59 (1787304779) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:33:02 (1787304782) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 05:33:05 (1787304785) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 05:33:12 (1787304792) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.532609 s, 23.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:33:26 (1787304806) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238582 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285329 s, 36.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:33:33 (1787304813) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:33:40 (1787304820) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00268282 s, 382 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0024827 s, 412 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000738784 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00369095 s, 277 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 05:33:48 (1787304828) create test files with size 15737893 .. bc87eae1f22156b0ba4367af1928a769 - c1435951b907c7c7c44d858007b942ba - 5f25120236f10310891a8d645d773375 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 05:34:08 (1787304848) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:34:20 (1787304860) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:34:26 (1787304866) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106458 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340944 s, 30.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 05:34:34 (1787304874) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227673 s, 9.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.305604 s, 13.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259357 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 05:34:52 (1787304892) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 05:34:57 (1787304897) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:35:00 (1787304900) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034984 s, 30.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 05:35:09 (1787304909) llite.lustre-ffff899a90a7a800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 05:35:18 (1787304918) llite.lustre-ffff899a90a7a800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586495 s, 17.9 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518309 s, 20.2 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 05:35:27 (1787304927) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:35:30 (1787304930) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe0:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:35:41 (1787304941) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.312675 s, 33.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:35:52 (1787304952) llite.lustre-ffff899a87f0d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.313244 s, 33.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff899a87f21800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:36:07 (1787304967) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:36:43 (1787305003) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:37:21 (1787305041) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 05:37:25 (1787305045) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.238658 s, 13.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.202385 s, 15.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfb72a054 (nosync,stale) Mirror 2: 0xfb72a054 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 38cdbcacdbd72b810cee18da7d7cabd0 - mirror 1 checksum: 2d7dd801da0bce980e356d3ef42d3a16 - mirror 2 checksum: 38cdbcacdbd72b810cee18da7d7cabd0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T05:37:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T05:37:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 38cdbcacdbd72b810cee18da7d7cabd0 - mirror 1 checksum: 2d7dd801da0bce980e356d3ef42d3a16 - mirror 2 checksum: 2d7dd801da0bce980e356d3ef42d3a16 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T05:37:28-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T05:37:28-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:37:37 (1787305057) Failing mds1 on oleg440-server Stopping /mnt/lustre-mds1 (opts:) on oleg440-server 05:37:45 (1787305065) shut down facet: mds1 facet_host: oleg440-server facet_failover_host: oleg440-server Failover mds1 to oleg440-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 05:38:15 (1787305095) targets are mounted 05:38:15 (1787305095) facet_failover done oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514931 s, 20.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:38:33 (1787305113) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:38:37 (1787305117) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:38:40 (1787305120) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00623411 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:38:48 (1787305128) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:38:51 (1787305131) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00639206 s, 641 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.123832 s, 33.1 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:39:00 (1787305140) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:39:22 (1787305162) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:39:51 (1787305191) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:40:13 (1787305213) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:40:37 (1787305237) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 05:40:44 (1787305244) mirror create pid 55571 mirror split pid 55572 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 55572 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55571 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 05:41:51 (1787305311) Seed set to 5313 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2a8ad 000001 1787305313.870342 trunc from 00000000 to 0x02a8ac (0x2a8ad bytes) 000002 1787305313.897464 mapwrite 0x008666 thru 0x01817e (0xfb19 bytes) 000004 1787305314.049934 write 0x00d24d thru 0x014c76 (0x7a2a bytes) 000006 1787305314.054524 mapwrite 0x01eb79 thru 0x01ee62 (0x2ea bytes) 000008 1787305314.176821 write 0x031236 thru 0x03c431 (0xb1fc bytes) 000009 1787305314.180929 write 0x0124ec thru 0x01cb03 (0xa618 bytes) 000010 1787305314.184896 trunc from 0x03c432 to 0x00ee19 (0x2d618 bytes) 000011 1787305314.220093 write 0x027047 thru 0x02ea2d (0x79e7 bytes) 000013 1787305314.224856 read 0x01952b thru 0x024b81 (0xb657 bytes) 000014 1787305314.234943 mapread 0x026357 thru 0x02ea2d (0x86d7 bytes) 000016 1787305314.244148 read 0x011ba9 thru 0x0176d3 (0x5b2b bytes) 000017 1787305314.254669 read 0x01191d thru 0x012617 (0xcfb bytes) 000019 1787305314.256046 write 0x02929b thru 0x037451 (0xe1b7 bytes) 000021 1787305314.261531 mapwrite 0x0068c8 thru 0x011f16 (0xb64f bytes) 000022 1787305314.338596 trunc from 0x037452 to 0x027b92 (0xf8bf bytes) 000023 1787305314.363859 write 0x00ded8 thru 0x013141 (0x526a bytes) 000028 1787305314.368047 mapread 0x017f11 thru 0x021dd6 (0x9ec6 bytes) 000029 1787305314.369108 read 0x016ada thru 0x020b96 (0xa0bd bytes) 000031 1787305314.370238 trunc from 0x027b93 to 0x00476e (0x23424 bytes) truncating to largest ever: 0x38c62 000032 1787305314.411999 trunc from 0x00476f to 0x038c61 (0x344f3 bytes) 000034 1787305314.452444 trunc from 0x038c62 to 0x0262e5 (0x1297c bytes) 000037 1787305314.490787 trunc from 0x0262e6 to 0x018cd5 (0xd610 bytes) 000038 1787305314.533045 MIRROR_EXTEND to 2 mirrors 000040 1787305315.812940 mapwrite 0x02a327 thru 0x03464b (0xa325 bytes) 000041 1787305316.107465 read 0x02b40b thru 0x033ade (0x86d4 bytes) 000043 1787305316.108626 mapwrite 0x02f707 thru 0x030b46 (0x1440 bytes) 000044 1787305316.179494 read 0x02853c thru 0x0308cf (0x8394 bytes) 000045 1787305316.180550 read 0x00e68c thru 0x010b4b (0x24c0 bytes) 000046 1787305316.193205 mapwrite 0x00eee7 thru 0x01ebcc (0xfce6 bytes) 000047 1787305316.258895 mapread 0x003928 thru 0x005188 (0x1861 bytes) 000048 1787305316.266832 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3464c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3464c): Mirror 2: 0xdecba61c (stale) CRC-32 checksum value for chunk [0, 0x3464c): Mirror 1: 0x6b4decd7 Mirror 3: 0x6b4decd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3464c: skipped 000049 1787305317.413160 mapread 0x021536 thru 0x02ef32 (0xd9fd bytes) 000050 1787305317.456958 mapread 0x030f5a thru 0x03464b (0x36f2 bytes) 000051 1787305317.460056 write 0x0167ed thru 0x01b8f5 (0x5109 bytes) 000052 1787305317.568209 trunc from 0x03464c to 0x00cb5b (0x27af0 bytes) 000053 1787305317.615028 write 0x02414a thru 0x03076b (0xc622 bytes) 000054 1787305317.618479 mapwrite 0x03d4da thru 0x03ffff (0x2b26 bytes) 000056 1787305317.715256 mapwrite 0x03ccac thru 0x03ffff (0x3354 bytes) 000057 1787305317.756150 write 0x002253 thru 0x00c8a6 (0xa654 bytes) 000061 1787305317.774718 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0x85ad2394 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7bc5e47e Mirror 4: 0x7bc5e47e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000062 1787305318.956951 mapwrite 0x02cd87 thru 0x034536 (0x77b0 bytes) 000065 1787305319.220124 mapwrite 0x037df8 thru 0x03ffff (0x8208 bytes) 000066 1787305319.287667 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 4: 0x7bc5e47e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 4: 0x7bc5e47e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000067 1787305319.597190 mapwrite 0x02f446 thru 0x03cb5c (0xd717 bytes) 000068 1787305319.773666 trunc from 0x040000 to 0x034756 (0xb8a9 bytes) 000070 1787305319.830658 mapwrite 0x01d7da thru 0x01e179 (0x9a0 bytes) 000071 1787305320.053253 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34757): Mirror 4: 0x60663b62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34757): Mirror 4: 0x60663b62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34757: skipped 000072 1787305320.499580 mapwrite 0x03cbcb thru 0x03ffff (0x3435 bytes) 000073 1787305320.652626 write 0x01f27b thru 0x027fa3 (0x8d29 bytes) 000074 1787305320.679968 read 0x00370d thru 0x01191e (0xe212 bytes) 000075 1787305320.705994 mapread 0x0002b1 thru 0x00920a (0x8f5a bytes) 000076 1787305320.717170 trunc from 0x040000 to 0x02341e (0x1cbe1 bytes) 000077 1787305320.739246 mapread 0x01d921 thru 0x02341e (0x5afe bytes) truncating to largest ever: 0x3d42a 000079 1787305320.757837 trunc from 0x02341f to 0x03d429 (0x1a00b bytes) 000080 1787305320.804495 mapread 0x034cd1 thru 0x03d429 (0x8759 bytes) 000082 1787305320.819069 mapread 0x00d13d thru 0x0187ba (0xb67e bytes) 000084 1787305320.830625 mapread 0x012bbe thru 0x01cb28 (0x9f6b bytes) 000085 1787305320.836979 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d42a): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d42a): Mirror 4: 0x819a055c (stale) CRC-32 checksum value for chunk [0, 0x3d42a): Mirror 1: 0xd3c311c8 Mirror 5: 0xd3c311c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d42a: skipped 000086 1787305322.062113 write 0x03070b thru 0x03e965 (0xe25b bytes) 000087 1787305322.178098 MIRROR_RESYNC 3 mirrors 000088 1787305323.537567 trunc from 0x03e966 to 0x00dfb8 (0x309ad bytes) 000089 1787305323.793424 write 0x01f75a thru 0x02d4ff (0xdda6 bytes) 000090 1787305323.804382 trunc from 0x02d500 to 0x009aaf (0x23a50 bytes) 000093 1787305323.871390 write 0x02c11c thru 0x037692 (0xb577 bytes) 000095 1787305323.880004 mapwrite 0x02cbd4 thru 0x0330a4 (0x64d1 bytes) 000096 1787305324.046387 read 0x01d3fe thru 0x025467 (0x806a bytes) 000099 1787305324.062688 mapread 0x029f4b thru 0x02df46 (0x3ffc bytes) 000101 1787305324.108164 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37693): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37693): Mirror 4: 0xc4281c26 (stale) Mirror 5: 0xc4281c26 (stale) CRC-32 checksum value for chunk [0, 0x37693): Mirror 1: 0x64148b24 Mirror 6: 0x64148b24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37693: skipped 000104 1787305325.577957 mapread 0x036284 thru 0x037692 (0x140f bytes) 000105 1787305325.595999 write 0x02a45e thru 0x031eee (0x7a91 bytes) 000108 1787305325.687699 write 0x011f90 thru 0x015401 (0x3472 bytes) 000110 1787305325.699091 trunc from 0x037693 to 0x03550d (0x2185 bytes) 000111 1787305325.720219 mapread 0x0118e9 thru 0x01f498 (0xdbb0 bytes) 000112 1787305325.742343 trunc from 0x03550e to 0x039494 (0x3f87 bytes) 000113 1787305325.762352 write 0x00dfc1 thru 0x01ab0c (0xcb4c bytes) 000116 1787305325.778777 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39495): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39495): Mirror 4: 0xc87c16eb (stale) Mirror 5: 0xc87c16eb (stale) Mirror 6: 0x99a813b0 (stale) CRC-32 checksum value for chunk [0, 0x39495): Mirror 1: 0x7d782cfb Mirror 7: 0x7d782cfb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39495: skipped 000117 1787305327.187018 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39495): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39495): Mirror 4: 0xc87c16eb (stale) Mirror 5: 0xc87c16eb (stale) Mirror 6: 0x99a813b0 (stale) CRC-32 checksum value for chunk [0, 0x39495): Mirror 1: 0x7d782cfb Mirror 7: 0x7d782cfb Mirror 8: 0x7d782cfb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39495: skipped 000119 1787305328.652688 MIRROR_RESYNC 6 mirrors 000121 1787305329.763264 MIRROR_EXTEND to 7 mirrors 000122 1787305330.791944 mapwrite 0x034612 thru 0x03ffff (0xb9ee bytes) 000123 1787305331.227916 mapwrite 0x037e24 thru 0x03ffff (0x81dc bytes) 000124 1787305331.332047 write 0x01ff59 thru 0x0290dd (0x9185 bytes) 000126 1787305331.357550 mapwrite 0x03a169 thru 0x03ccd3 (0x2b6b bytes) 000127 1787305331.475130 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb0b98f9b (stale) Mirror 5: 0xb0b98f9b (stale) Mirror 6: 0xb0b98f9b (stale) Mirror 7: 0xb0b98f9b (stale) Mirror 8: 0xb0b98f9b (stale) Mirror 9: 0xb0b98f9b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3864946 Mirror 10: 0xc3864946 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000128 1787305332.685606 mapread 0x039346 thru 0x03ffff (0x6cba bytes) 000130 1787305332.710398 write 0x00ecb8 thru 0x01dc5e (0xefa7 bytes) 000131 1787305332.829556 mapwrite 0x02f216 thru 0x03d0ec (0xded7 bytes) 000135 1787305332.960960 trunc from 0x040000 to 0x031e5d (0xe1a2 bytes) 000136 1787305333.081065 trunc from 0x031e5e to 0x02e9a1 (0x34bc bytes) 000137 1787305333.120706 write 0x03a9e6 thru 0x03ffff (0x561a bytes) 000139 1787305333.127606 mapread 0x029e44 thru 0x030c39 (0x6df6 bytes) 000140 1787305333.153557 mapread 0x021143 thru 0x02549f (0x435d bytes) 000141 1787305333.179726 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb0b98f9b (stale) Mirror 5: 0xb0b98f9b (stale) Mirror 6: 0xb0b98f9b (stale) Mirror 7: 0xb0b98f9b (stale) Mirror 8: 0xb0b98f9b (stale) Mirror 9: 0xb0b98f9b (stale) Mirror 10: 0xc3864946 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x917ebd60 Mirror 11: 0x917ebd60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000143 1787305334.551327 write 0x0275b9 thru 0x03223f (0xac87 bytes) 000145 1787305334.659917 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb0b98f9b (stale) Mirror 5: 0xb0b98f9b (stale) Mirror 6: 0xb0b98f9b (stale) Mirror 7: 0xb0b98f9b (stale) Mirror 8: 0xb0b98f9b (stale) Mirror 9: 0xb0b98f9b (stale) Mirror 10: 0xc3864946 (stale) Mirror 11: 0x917ebd60 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8c90829c Mirror 12: 0x8c90829c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1787305336.528559 mapread 0x00806f thru 0x014312 (0xc2a4 bytes) 000151 1787305336.612087 mapwrite 0x000378 thru 0x002a80 (0x2709 bytes) 000155 1787305336.986822 read 0x013c72 thru 0x021c82 (0xe011 bytes) 000157 1787305337.016927 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb0b98f9b (stale) Mirror 5: 0xb0b98f9b (stale) Mirror 6: 0xb0b98f9b (stale) Mirror 7: 0xb0b98f9b (stale) Mirror 8: 0xb0b98f9b (stale) Mirror 9: 0xb0b98f9b (stale) Mirror 10: 0xc3864946 (stale) Mirror 11: 0x917ebd60 (stale) Mirror 12: 0x8c90829c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcc2a92f Mirror 13: 0xcc2a92f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000158 1787305338.917153 trunc from 0x040000 to 0x033583 (0xca7c bytes) 000159 1787305339.046798 mapread 0x02de5b thru 0x033583 (0x5729 bytes) 000160 1787305339.115452 trunc from 0x033584 to 0x00c979 (0x26c0a bytes) 000161 1787305339.180184 read 0x0025c9 thru 0x00c979 (0xa3b1 bytes) 000167 1787305339.209498 trunc from 0x00c97a to 0x0324ed (0x25b74 bytes) 000168 1787305339.261431 read 0x008709 thru 0x00d958 (0x5250 bytes) 000169 1787305339.292738 write 0x02eeb0 thru 0x03414d (0x529e bytes) 000172 1787305339.295025 mapread 0x000d0b thru 0x00979f (0x8a95 bytes) 000174 1787305339.306762 mapwrite 0x005bec thru 0x015770 (0xfb85 bytes) 000175 1787305339.411530 MIRROR_RESYNC 11 mirrors 000178 1787305342.388279 write 0x038b25 thru 0x03ffff (0x74db bytes) 000179 1787305342.528202 mapwrite 0x0013ac thru 0x007659 (0x62ae bytes) 000180 1787305342.682141 mapwrite 0x03c418 thru 0x03ffff (0x3be8 bytes) 000181 1787305342.814087 mapread 0x033ca0 thru 0x03e389 (0xa6ea bytes) 000183 1787305342.854137 trunc from 0x040000 to 0x039632 (0x69cd bytes) 000187 1787305342.920202 mapwrite 0x001b42 thru 0x00bada (0x9f99 bytes) 000188 1787305343.047380 mapwrite 0x022573 thru 0x02a5ff (0x808d bytes) 000190 1787305343.218037 mapread 0x031d1e thru 0x039632 (0x7915 bytes) 000192 1787305343.221049 mapwrite 0x03b8db thru 0x03ffff (0x4725 bytes) 000193 1787305343.312830 write 0x02d95c thru 0x03404d (0x66f2 bytes) 000196 1787305343.321939 read 0x03e244 thru 0x03ffff (0x1dbc bytes) 000197 1787305343.340119 MIRROR_RESYNC 11 mirrors 000199 1787305345.888381 write 0x008342 thru 0x00a6e7 (0x23a6 bytes) 000200 1787305346.001811 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd0b58a51 (stale) Mirror 5: 0xd0b58a51 (stale) Mirror 6: 0xd0b58a51 (stale) Mirror 7: 0xd0b58a51 (stale) Mirror 8: 0xd0b58a51 (stale) Mirror 9: 0xd0b58a51 (stale) Mirror 10: 0xd0b58a51 (stale) Mirror 11: 0xd0b58a51 (stale) Mirror 12: 0xd0b58a51 (stale) Mirror 13: 0xd0b58a51 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf96affa3 Mirror 14: 0xf96affa3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000201 1787305347.867574 trunc from 0x040000 to 0x0081d4 (0x37e2b bytes) 000205 1787305348.026016 trunc from 0x0081d5 to 0x03800c (0x2fe38 bytes) 000206 1787305348.077577 MIRROR_RESYNC 12 mirrors 000207 1787305351.224737 mapwrite 0x03661c thru 0x039a0d (0x33f2 bytes) 000208 1787305351.461769 read 0x0067a2 thru 0x015b7c (0xf3db bytes) 000214 1787305351.481229 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a0e): Mirror 4: 0xe6a20291 (stale) Mirror 5: 0xe6a20291 (stale) Mirror 6: 0xe6a20291 (stale) Mirror 7: 0xe6a20291 (stale) Mirror 9: 0xe6a20291 (stale) Mirror 10: 0xe6a20291 (stale) Mirror 11: 0xe6a20291 (stale) Mirror 12: 0xe6a20291 (stale) Mirror 13: 0xe6a20291 (stale) Mirror 14: 0xe6a20291 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a0e): Mirror 4: 0xe6a20291 (stale) Mirror 5: 0xe6a20291 (stale) Mirror 6: 0xe6a20291 (stale) Mirror 7: 0xe6a20291 (stale) Mirror 9: 0xe6a20291 (stale) Mirror 10: 0xe6a20291 (stale) Mirror 11: 0xe6a20291 (stale) Mirror 12: 0xe6a20291 (stale) Mirror 13: 0xe6a20291 (stale) Mirror 14: 0xe6a20291 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a0e: skipped 000215 1787305352.259267 mapread 0x02e294 thru 0x03317f (0x4eec bytes) 000216 1787305352.284978 trunc from 0x039a0e to 0x022d35 (0x16cd8 bytes) 000218 1787305352.302066 trunc from 0x022d36 to 0x02c510 (0x97db bytes) 000219 1787305352.327463 mapread 0x02b9ef thru 0x02c510 (0xb22 bytes) 000221 1787305352.349904 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c511): Mirror 4: 0x906f352e (stale) Mirror 5: 0x906f352e (stale) Mirror 6: 0x906f352e (stale) Mirror 7: 0x906f352e (stale) Mirror 10: 0x906f352e (stale) Mirror 11: 0x906f352e (stale) Mirror 12: 0x906f352e (stale) Mirror 13: 0x906f352e (stale) Mirror 14: 0x906f352e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c511): Mirror 4: 0x906f352e (stale) Mirror 5: 0x906f352e (stale) Mirror 6: 0x906f352e (stale) Mirror 7: 0x906f352e (stale) Mirror 10: 0x906f352e (stale) Mirror 11: 0x906f352e (stale) Mirror 12: 0x906f352e (stale) Mirror 13: 0x906f352e (stale) Mirror 14: 0x906f352e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c511: skipped 000222 1787305352.976649 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c511): Mirror 4: 0x906f352e (stale) Mirror 5: 0x906f352e (stale) Mirror 7: 0x906f352e (stale) Mirror 10: 0x906f352e (stale) Mirror 11: 0x906f352e (stale) Mirror 12: 0x906f352e (stale) Mirror 13: 0x906f352e (stale) Mirror 14: 0x906f352e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c511): Mirror 4: 0x906f352e (stale) Mirror 5: 0x906f352e (stale) Mirror 7: 0x906f352e (stale) Mirror 10: 0x906f352e (stale) Mirror 11: 0x906f352e (stale) Mirror 12: 0x906f352e (stale) Mirror 13: 0x906f352e (stale) Mirror 14: 0x906f352e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c511: skipped 000223 1787305353.663184 mapread 0x023e6b thru 0x023f53 (0xe9 bytes) 000225 1787305353.670117 write 0x039bfc thru 0x03ffff (0x6404 bytes) 000227 1787305353.672871 write 0x038262 thru 0x03b47d (0x321c bytes) 000228 1787305353.678422 trunc from 0x040000 to 0x016a05 (0x295fa bytes) 000239 1787305353.709925 trunc from 0x016a06 to 0x00945e (0xd5a7 bytes) 000240 1787305353.741274 trunc from 0x00945f to 0x021ba4 (0x18746 bytes) 000241 1787305353.763857 MIRROR_RESYNC 9 mirrors 000243 1787305356.271690 mapread 0x01480f thru 0x01fb3b (0xb32d bytes) 000245 1787305356.317564 mapread 0x015ffb thru 0x017270 (0x1276 bytes) 000246 1787305356.319019 mapread 0x01ab5f thru 0x021858 (0x6cfa bytes) 000248 1787305356.320034 mapread 0x01e65b thru 0x021ba4 (0x354a bytes) 000251 1787305356.321791 mapwrite 0x01af4d thru 0x02071a (0x57ce bytes) 000252 1787305356.583706 read 0x01dc1b thru 0x021ba4 (0x3f8a bytes) 000253 1787305356.592001 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x21ba5): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ba5): Mirror 4: 0xab6736fb (stale) Mirror 5: 0xab6736fb (stale) Mirror 7: 0xab6736fb (stale) Mirror 10: 0xab6736fb (stale) Mirror 11: 0xab6736fb (stale) Mirror 12: 0xab6736fb (stale) Mirror 13: 0xab6736fb (stale) Mirror 14: 0xab6736fb (stale) CRC-32 checksum value for chunk [0, 0x21ba5): Mirror 1: 0xf69fb28d Mirror 15: 0xf69fb28d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ba5: skipped 000259 1787305358.316783 mapwrite 0x023106 thru 0x028b3d (0x5a38 bytes) 000261 1787305358.607197 mapread 0x001a96 thru 0x00d4e2 (0xba4d bytes) 000262 1787305358.622726 MIRROR_RESYNC 10 mirrors 000264 1787305360.571439 MIRROR_EXTEND to 11 mirrors 000265 1787305361.769655 write 0x030cfd thru 0x031cf8 (0xffc bytes) 000267 1787305361.890753 mapwrite 0x00729b thru 0x014bd9 (0xd93f bytes) 000268 1787305362.006741 mapwrite 0x03c5b2 thru 0x03f3d4 (0x2e23 bytes) 000269 1787305362.179101 MIRROR_RESYNC 11 mirrors 000276 1787305364.123355 trunc from 0x03f3d5 to 0x0346a6 (0xad2e bytes) 000280 1787305364.230017 trunc from 0x0346a7 to 0x023fcf (0x106d7 bytes) 000281 1787305364.258553 read 0x01fb13 thru 0x023fcf (0x44bd bytes) 000282 1787305364.274719 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23fd0): Mirror 5: 0x84d785b6 (stale) Mirror 7: 0x84d785b6 (stale) Mirror 10: 0x84d785b6 (stale) Mirror 11: 0x84d785b6 (stale) Mirror 12: 0x84d785b6 (stale) Mirror 13: 0x84d785b6 (stale) Mirror 14: 0x84d785b6 (stale) Mirror 15: 0x84d785b6 (stale) Mirror 16: 0x84d785b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23fd0): Mirror 5: 0x84d785b6 (stale) Mirror 7: 0x84d785b6 (stale) Mirror 10: 0x84d785b6 (stale) Mirror 11: 0x84d785b6 (stale) Mirror 12: 0x84d785b6 (stale) Mirror 13: 0x84d785b6 (stale) Mirror 14: 0x84d785b6 (stale) Mirror 15: 0x84d785b6 (stale) Mirror 16: 0x84d785b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23fd0: skipped 000285 1787305364.832830 trunc from 0x023fd0 to 0x0000d5 (0x23efa bytes) 000286 1787305364.853886 mapwrite 0x028e5e thru 0x02bc68 (0x2e0b bytes) 000287 1787305364.951732 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bc69): Mirror 5: 0x4b078c55 (stale) Mirror 7: 0x4b078c55 (stale) Mirror 10: 0x4b078c55 (stale) Mirror 11: 0x4b078c55 (stale) Mirror 12: 0x4b078c55 (stale) Mirror 14: 0x4b078c55 (stale) Mirror 15: 0x4b078c55 (stale) Mirror 16: 0x4b078c55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc69): Mirror 5: 0x4b078c55 (stale) Mirror 7: 0x4b078c55 (stale) Mirror 10: 0x4b078c55 (stale) Mirror 11: 0x4b078c55 (stale) Mirror 12: 0x4b078c55 (stale) Mirror 14: 0x4b078c55 (stale) Mirror 15: 0x4b078c55 (stale) Mirror 16: 0x4b078c55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc69: skipped 000290 1787305365.833992 mapwrite 0x02ab8b thru 0x02c633 (0x1aa9 bytes) 000291 1787305365.940783 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c634): Mirror 7: 0x6c130adf (stale) Mirror 10: 0x6c130adf (stale) Mirror 11: 0x6c130adf (stale) Mirror 12: 0x6c130adf (stale) Mirror 14: 0x6c130adf (stale) Mirror 15: 0x6c130adf (stale) Mirror 16: 0x6c130adf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c634): Mirror 7: 0x6c130adf (stale) Mirror 10: 0x6c130adf (stale) Mirror 11: 0x6c130adf (stale) Mirror 12: 0x6c130adf (stale) Mirror 14: 0x6c130adf (stale) Mirror 15: 0x6c130adf (stale) Mirror 16: 0x6c130adf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c634: skipped 000293 1787305366.441218 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2c634): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c634): Mirror 7: 0x6c130adf (stale) Mirror 10: 0x6c130adf (stale) Mirror 11: 0x6c130adf (stale) Mirror 12: 0x6c130adf (stale) Mirror 14: 0x6c130adf (stale) Mirror 15: 0x6c130adf (stale) Mirror 16: 0x6c130adf (stale) CRC-32 checksum value for chunk [0, 0x2c634): Mirror 1: 0xc28da419 Mirror 17: 0xc28da419 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c634: skipped 000294 1787305367.776292 MIRROR_RESYNC 9 mirrors 000295 1787305369.952748 write 0x024998 thru 0x02ce91 (0x84fa bytes) 000296 1787305370.161618 mapread 0x02258e thru 0x02ce91 (0xa904 bytes) 000299 1787305370.184072 mapwrite 0x01e74e thru 0x027fc7 (0x987a bytes) 000300 1787305370.307913 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2ce92): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce92): Mirror 7: 0xbcd69d9a (stale) Mirror 10: 0xbcd69d9a (stale) Mirror 11: 0xbcd69d9a (stale) Mirror 12: 0xbcd69d9a (stale) Mirror 14: 0xbcd69d9a (stale) Mirror 15: 0xbcd69d9a (stale) Mirror 16: 0xbcd69d9a (stale) Mirror 17: 0xbcd69d9a (stale) CRC-32 checksum value for chunk [0, 0x2ce92): Mirror 1: 0xd675b661 Mirror 18: 0xd675b661 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce92: skipped 000301 1787305371.814011 trunc from 0x02ce92 to 0x02531d (0x7b74 bytes) 000302 1787305372.042720 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2531e): Mirror 7: 0xd144e3a8 (stale) Mirror 10: 0xd144e3a8 (stale) Mirror 11: 0xd144e3a8 (stale) Mirror 12: 0xd144e3a8 (stale) Mirror 14: 0xd144e3a8 (stale) Mirror 15: 0xd144e3a8 (stale) Mirror 17: 0xd144e3a8 (stale) Mirror 18: 0xaebc75be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2531e): Mirror 7: 0xd144e3a8 (stale) Mirror 10: 0xd144e3a8 (stale) Mirror 11: 0xd144e3a8 (stale) Mirror 12: 0xd144e3a8 (stale) Mirror 14: 0xd144e3a8 (stale) Mirror 15: 0xd144e3a8 (stale) Mirror 17: 0xd144e3a8 (stale) Mirror 18: 0xaebc75be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2531e: skipped 000303 1787305372.676192 read 0x01401e thru 0x02105e (0xd041 bytes) 000304 1787305372.698214 read 0x01d234 thru 0x020cdc (0x3aa9 bytes) 000305 1787305372.699071 mapwrite 0x034f7a thru 0x03d499 (0x8520 bytes) 000308 1787305372.795907 mapread 0x004f4d thru 0x013679 (0xe72d bytes) 000309 1787305372.798163 read 0x021737 thru 0x02faf7 (0xe3c1 bytes) 000311 1787305372.799763 mapwrite 0x00de2d thru 0x013134 (0x5308 bytes) 000315 1787305372.857544 write 0x00cb26 thru 0x01a147 (0xd622 bytes) 000316 1787305372.861232 mapwrite 0x01a48a thru 0x027a5b (0xd5d2 bytes) 000317 1787305372.955926 read 0x0264b9 thru 0x02f829 (0x9371 bytes) 000318 1787305372.957008 read 0x01cc9f thru 0x020bf1 (0x3f53 bytes) 000319 1787305372.960197 mapread 0x0189d2 thru 0x02116f (0x879e bytes) 000320 1787305372.967465 mapread 0x0085e1 thru 0x017b61 (0xf581 bytes) 000322 1787305372.969185 read 0x0083d9 thru 0x012190 (0x9db8 bytes) 000323 1787305372.969843 read 0x03a50c thru 0x03d499 (0x2f8e bytes) 000327 1787305372.978569 mapwrite 0x03535d thru 0x038124 (0x2dc8 bytes) 000329 1787305373.048125 trunc from 0x03d49a to 0x031f35 (0xb564 bytes) 000333 1787305373.081090 mapread 0x02ea5d thru 0x031f35 (0x34d9 bytes) 000334 1787305373.083757 mapwrite 0x006521 thru 0x012f92 (0xca72 bytes) 000336 1787305373.175381 read 0x025b73 thru 0x0304ad (0xa93b bytes) 000340 1787305373.180654 write 0x0232b3 thru 0x02e9ca (0xb718 bytes) 000341 1787305373.182719 write 0x020c58 thru 0x026c80 (0x6029 bytes) 000346 1787305373.189661 mapread 0x030a6e thru 0x031f35 (0x14c8 bytes) 000347 1787305373.191712 trunc from 0x031f36 to 0x02969e (0x8897 bytes) 000349 1787305373.227537 write 0x036bf7 thru 0x03ffff (0x9409 bytes) 000350 1787305373.230805 mapread 0x03d33e thru 0x03ffff (0x2cc2 bytes) 000351 1787305373.232774 mapread 0x024329 thru 0x030026 (0xbcfe bytes) 000352 1787305373.243176 read 0x02bd80 thru 0x036bd7 (0xae58 bytes) 000353 1787305373.244337 trunc from 0x040000 to 0x027646 (0x189b9 bytes) 000354 1787305373.267173 read 0x007145 thru 0x0166c2 (0xf57e bytes) 000355 1787305373.268199 write 0x034e09 thru 0x03ffff (0xb1f7 bytes) 000356 1787305373.271698 mapwrite 0x01de4f thru 0x02dbb5 (0xfd67 bytes) 000357 1787305373.337794 write 0x0034a0 thru 0x009670 (0x61d1 bytes) 000358 1787305373.339034 trunc from 0x040000 to 0x01890f (0x276f0 bytes) 000359 1787305373.374088 read 0x015a45 thru 0x01890f (0x2ecb bytes) 000361 1787305373.375261 mapread 0x016a10 thru 0x01890f (0x1f00 bytes) 000363 1787305373.387894 write 0x02d10b thru 0x0343e2 (0x72d8 bytes) 000364 1787305373.390277 mapwrite 0x035854 thru 0x03ffff (0xa7ac bytes) 000366 1787305373.563547 trunc from 0x040000 to 0x039c80 (0x637f bytes) 000368 1787305373.594742 mapwrite 0x02d3be thru 0x02f6d5 (0x2318 bytes) 000370 1787305373.668714 trunc from 0x039c81 to 0x01df4c (0x1bd34 bytes) 000371 1787305373.692560 read 0x01b1b9 thru 0x01df4c (0x2d94 bytes) 000373 1787305373.695709 mapwrite 0x03e775 thru 0x03ffff (0x188b bytes) 000374 1787305373.786813 mapread 0x004742 thru 0x005600 (0xebf bytes) 000375 1787305373.788389 read 0x03299e thru 0x03f73d (0xcda0 bytes) 000376 1787305373.789976 write 0x006b63 thru 0x01028b (0x9729 bytes) 000377 1787305373.791705 trunc from 0x040000 to 0x00857b (0x37a84 bytes) 000378 1787305373.816088 write 0x03c632 thru 0x03ffff (0x39ce bytes) 000379 1787305373.818378 trunc from 0x040000 to 0x030240 (0xfdbf bytes) 000381 1787305373.854419 read 0x02f024 thru 0x030240 (0x121d bytes) 000382 1787305373.874844 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30241): Mirror 10: 0x43f427df (stale) Mirror 11: 0x43f427df (stale) Mirror 12: 0x43f427df (stale) Mirror 14: 0x43f427df (stale) Mirror 15: 0x43f427df (stale) Mirror 17: 0x43f427df (stale) Mirror 18: 0xcbfc05d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30241): Mirror 10: 0x43f427df (stale) Mirror 11: 0x43f427df (stale) Mirror 12: 0x43f427df (stale) Mirror 14: 0x43f427df (stale) Mirror 15: 0x43f427df (stale) Mirror 17: 0x43f427df (stale) Mirror 18: 0xcbfc05d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30241: skipped 000383 1787305374.482955 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x30241): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30241): Mirror 10: 0x43f427df (stale) Mirror 11: 0x43f427df (stale) Mirror 12: 0x43f427df (stale) Mirror 14: 0x43f427df (stale) Mirror 15: 0x43f427df (stale) Mirror 17: 0x43f427df (stale) Mirror 18: 0xcbfc05d5 (stale) CRC-32 checksum value for chunk [0, 0x30241): Mirror 1: 0x49e3e80e Mirror 19: 0x49e3e80e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30241: skipped 000388 1787305376.006018 mapread 0x00a820 thru 0x0146f2 (0x9ed3 bytes) 000390 1787305376.032112 read 0x015d99 thru 0x01f504 (0x976c bytes) 000391 1787305376.056988 read 0x00dd6c thru 0x017d71 (0xa006 bytes) 000392 1787305376.058218 mapread 0x02a31f thru 0x02bc24 (0x1906 bytes) 000394 1787305376.074071 trunc from 0x030241 to 0x032dee (0x2bae bytes) 000397 1787305376.199422 mapwrite 0x00b9c5 thru 0x00ecf3 (0x332f bytes) 000398 1787305376.334828 MIRROR_RESYNC 9 mirrors 000400 1787305378.920674 mapread 0x00a83f thru 0x00ae37 (0x5f9 bytes) 000401 1787305378.929964 read 0x02b417 thru 0x032dee (0x79d8 bytes) 000402 1787305378.947836 mapread 0x011126 thru 0x01f035 (0xdf10 bytes) 000403 1787305378.994731 write 0x00a6e4 thru 0x015023 (0xa940 bytes) 000404 1787305379.087644 trunc from 0x032def to 0x010e12 (0x21fdc bytes) 000405 1787305379.123677 trunc from 0x010e13 to 0x0067eb (0xa627 bytes) 000408 1787305379.158999 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x67ec): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x67ec): Mirror 10: 0x817dce80 (stale) Mirror 11: 0x817dce80 (stale) Mirror 12: 0x817dce80 (stale) Mirror 14: 0x817dce80 (stale) Mirror 15: 0x817dce80 (stale) Mirror 17: 0x817dce80 (stale) Mirror 18: 0x817dce80 (stale) Mirror 19: 0x817dce80 (stale) CRC-32 checksum value for chunk [0, 0x67ec): Mirror 1: 0x817dce80 Mirror 20: 0x817dce80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x67ec: skipped 000409 1787305380.432875 trunc from 0x0067ec to 0x0388ad (0x320c2 bytes) 000410 1787305380.554346 read 0x019489 thru 0x01b0b8 (0x1c30 bytes) 000411 1787305380.563563 read 0x02d162 thru 0x0388ad (0xb74c bytes) 000414 1787305380.582747 trunc from 0x0388ae to 0x0153cc (0x234e1 bytes) 000417 1787305380.631070 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x153cd): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x153cd): Mirror 10: 0x8c741f3e (stale) Mirror 11: 0x8c741f3e (stale) Mirror 12: 0x8c741f3e (stale) Mirror 14: 0x8c741f3e (stale) Mirror 15: 0x8c741f3e (stale) Mirror 17: 0x8c741f3e (stale) Mirror 18: 0x8c741f3e (stale) Mirror 19: 0x8c741f3e (stale) Mirror 20: 0xcdef40b2 (stale) CRC-32 checksum value for chunk [0, 0x153cd): Mirror 1: 0xcdef40b2 Mirror 21: 0xcdef40b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x153cd: skipped 000418 1787305381.916599 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x153cd): Mirror 10: 0x8c741f3e (stale) Mirror 11: 0x8c741f3e (stale) Mirror 12: 0x8c741f3e (stale) Mirror 14: 0x8c741f3e (stale) Mirror 15: 0x8c741f3e (stale) Mirror 17: 0x8c741f3e (stale) Mirror 18: 0x8c741f3e (stale) Mirror 19: 0x8c741f3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x153cd): Mirror 10: 0x8c741f3e (stale) Mirror 11: 0x8c741f3e (stale) Mirror 12: 0x8c741f3e (stale) Mirror 14: 0x8c741f3e (stale) Mirror 15: 0x8c741f3e (stale) Mirror 17: 0x8c741f3e (stale) Mirror 18: 0x8c741f3e (stale) Mirror 19: 0x8c741f3e (stale) CRC-32 checksum value for chunk [0, 0x153cd): Mirror 1: 0xcdef40b2 Mirror 21: 0xcdef40b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x153cd: skipped 000420 1787305382.702733 read 0x003bd9 thru 0x00590a (0x1d32 bytes) 000423 1787305382.710955 mapread 0x012977 thru 0x0153cc (0x2a56 bytes) 000424 1787305382.740097 read 0x0103c6 thru 0x0153cc (0x5007 bytes) 000425 1787305382.740799 trunc from 0x0153cd to 0x024a1f (0xf653 bytes) 000428 1787305382.837949 mapwrite 0x0133fe thru 0x01e45e (0xb061 bytes) 000429 1787305382.925634 write 0x02131d thru 0x021882 (0x566 bytes) 000430 1787305382.928411 write 0x031b4f thru 0x03db7a (0xc02c bytes) 000431 1787305382.931633 mapread 0x01b6c8 thru 0x023151 (0x7a8a bytes) 000433 1787305382.934269 read 0x0044a7 thru 0x011035 (0xcb8f bytes) 000436 1787305382.937292 mapwrite 0x01c8a1 thru 0x025c61 (0x93c1 bytes) 000437 1787305382.998951 MIRROR_RESYNC 10 mirrors 000438 1787305384.812002 MIRROR_EXTEND to 11 mirrors 000439 1787305385.884389 mapread 0x01d5bf thru 0x022608 (0x504a bytes) 000440 1787305385.915234 read 0x00b80b thru 0x00f6fa (0x3ef0 bytes) 000441 1787305385.924554 write 0x03954b thru 0x03ffff (0x6ab5 bytes) 000444 1787305386.076968 trunc from 0x040000 to 0x037be1 (0x841e bytes) 000445 1787305386.113526 mapread 0x019784 thru 0x022138 (0x89b5 bytes) 000446 1787305386.135703 write 0x0086f8 thru 0x016438 (0xdd41 bytes) 000448 1787305386.144821 mapwrite 0x037fb1 thru 0x03b20e (0x325e bytes) 000453 1787305386.247238 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b20f): Mirror 10: 0x80433029 (stale) Mirror 11: 0x80433029 (stale) Mirror 12: 0x80433029 (stale) Mirror 14: 0x80433029 (stale) Mirror 15: 0x80433029 (stale) Mirror 18: 0x80433029 (stale) Mirror 19: 0x80433029 (stale) Mirror 21: 0x80433029 (stale) Mirror 22: 0x80433029 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b20f): Mirror 10: 0x80433029 (stale) Mirror 11: 0x80433029 (stale) Mirror 12: 0x80433029 (stale) Mirror 14: 0x80433029 (stale) Mirror 15: 0x80433029 (stale) Mirror 18: 0x80433029 (stale) Mirror 19: 0x80433029 (stale) Mirror 21: 0x80433029 (stale) Mirror 22: 0x80433029 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b20f: skipped 000454 1787305386.947548 mapwrite 0x039e2d thru 0x03ffff (0x61d3 bytes) 000455 1787305387.054732 mapread 0x01a538 thru 0x020855 (0x631e bytes) 000463 1787305387.074048 mapwrite 0x038359 thru 0x03ffff (0x7ca7 bytes) 000469 1787305387.167941 trunc from 0x040000 to 0x02dd47 (0x122b8 bytes) 000470 1787305387.184595 trunc from 0x02dd48 to 0x02c581 (0x17c6 bytes) 000471 1787305387.209697 read 0x02422c thru 0x026106 (0x1edb bytes) 000473 1787305387.222104 mapread 0x024e3e thru 0x02c581 (0x7744 bytes) 000474 1787305387.245249 mapwrite 0x0019eb thru 0x0099f7 (0x800d bytes) 000475 1787305387.313153 trunc from 0x02c582 to 0x034415 (0x7e94 bytes) 000478 1787305387.343537 write 0x01df09 thru 0x027cdf (0x9dd7 bytes) 000479 1787305387.345180 write 0x00ba5d thru 0x0199fd (0xdfa1 bytes) 000482 1787305387.354373 write 0x03242e thru 0x03b6e6 (0x92b9 bytes) 000484 1787305387.367787 mapwrite 0x021876 thru 0x025462 (0x3bed bytes) 000485 1787305387.445537 mapread 0x001fee thru 0x00788a (0x589d bytes) 000486 1787305387.447435 read 0x036d3b thru 0x03b6e6 (0x49ac bytes) 000487 1787305387.448138 mapread 0x0171d7 thru 0x02627f (0xf0a9 bytes) 000489 1787305387.449757 write 0x02dbea thru 0x03b712 (0xdb29 bytes) 000490 1787305387.461726 mapwrite 0x03594d thru 0x037ad1 (0x2185 bytes) 000491 1787305387.528947 trunc from 0x03b713 to 0x0184fb (0x23217 bytes) 000493 1787305387.548247 read 0x001acf thru 0x00ca21 (0xaf53 bytes) 000494 1787305387.549000 read 0x00ee3a thru 0x011483 (0x264a bytes) 000498 1787305387.557669 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x184fc): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x184fc): Mirror 10: 0x4ebe3f5b (stale) Mirror 11: 0x4ebe3f5b (stale) Mirror 12: 0x4ebe3f5b (stale) Mirror 14: 0x4ebe3f5b (stale) Mirror 15: 0x4ebe3f5b (stale) Mirror 18: 0x4ebe3f5b (stale) Mirror 19: 0x4ebe3f5b (stale) Mirror 21: 0x4ebe3f5b (stale) Mirror 22: 0x4ebe3f5b (stale) CRC-32 checksum value for chunk [0, 0x184fc): Mirror 1: 0x846918c7 Mirror 23: 0x846918c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x184fc: skipped 000499 1787305388.600100 mapread 0x01290d thru 0x0164ff (0x3bf3 bytes) 000500 1787305388.638614 write 0x00f2a0 thru 0x01396c (0x46cd bytes) 000502 1787305388.773062 trunc from 0x0184fc to 0x007086 (0x11475 bytes) 000504 1787305388.844602 trunc from 0x007087 to 0x030168 (0x290e2 bytes) 000505 1787305388.867381 read 0x02e5eb thru 0x030168 (0x1b7e bytes) 000508 1787305388.876166 read 0x02f193 thru 0x030168 (0xfd6 bytes) 000509 1787305388.877471 trunc from 0x030169 to 0x03923a (0x90d2 bytes) 000510 1787305388.905200 write 0x00a7cb thru 0x0122d7 (0x7b0d bytes) 000513 1787305388.926761 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3923b): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3923b): Mirror 10: 0x6d3889d1 (stale) Mirror 11: 0x6d3889d1 (stale) Mirror 12: 0x6d3889d1 (stale) Mirror 14: 0x6d3889d1 (stale) Mirror 15: 0x6d3889d1 (stale) Mirror 18: 0x6d3889d1 (stale) Mirror 19: 0x6d3889d1 (stale) Mirror 21: 0x6d3889d1 (stale) Mirror 22: 0x6d3889d1 (stale) Mirror 23: 0x9ffb823e (stale) CRC-32 checksum value for chunk [0, 0x3923b): Mirror 1: 0xe488389d Mirror 24: 0xe488389d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3923b: skipped 000516 1787305390.739346 trunc from 0x03923b to 0x00b334 (0x2df06 bytes) 000517 1787305390.866439 trunc from 0x00b335 to 0x001d6c (0x95c8 bytes) 000518 1787305390.903769 write 0x00dc1c thru 0x0114bd (0x38a2 bytes) 000519 1787305390.913124 MIRROR_RESYNC 12 mirrors 000521 1787305393.110210 trunc from 0x0114be to 0x00fd9f (0x171e bytes) 000523 1787305393.222374 trunc from 0x00fda0 to 0x00e011 (0x1d8e bytes) 000524 1787305393.259799 mapwrite 0x004502 thru 0x0138ac (0xf3ab bytes) 000525 1787305393.435450 read 0x0123cf thru 0x0138ac (0x14de bytes) 000526 1787305393.437230 mapwrite 0x0088d5 thru 0x00e2a2 (0x59ce bytes) 000527 1787305393.512960 trunc from 0x0138ad to 0x00adf9 (0x8ab3 bytes) 000529 1787305393.530117 mapread 0x000100 thru 0x00adf9 (0xacfa bytes) 000530 1787305393.536794 trunc from 0x00adfa to 0x01357d (0x8784 bytes) 000531 1787305393.554289 trunc from 0x01357e to 0x030353 (0x1cdd6 bytes) 000534 1787305393.571855 read 0x00379a thru 0x004ac4 (0x132b bytes) 000535 1787305393.574268 trunc from 0x030354 to 0x01051a (0x1fe39 bytes) 000537 1787305393.601706 trunc from 0x01051b to 0x03a326 (0x29e0c bytes) 000538 1787305393.630729 mapread 0x018417 thru 0x0274fe (0xf0e8 bytes) 000539 1787305393.643057 write 0x0289d8 thru 0x0311a3 (0x87cc bytes) 000541 1787305393.645486 mapread 0x035c44 thru 0x03a326 (0x46e3 bytes) 000543 1787305393.648565 read 0x001796 thru 0x00a98f (0x91fa bytes) 000544 1787305393.654460 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a327): Mirror 10: 0x209aaa49 (stale) Mirror 11: 0x209aaa49 (stale) Mirror 14: 0x209aaa49 (stale) Mirror 15: 0x209aaa49 (stale) Mirror 18: 0x209aaa49 (stale) Mirror 19: 0x209aaa49 (stale) Mirror 21: 0x209aaa49 (stale) Mirror 22: 0x209aaa49 (stale) Mirror 23: 0x209aaa49 (stale) Mirror 24: 0x209aaa49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a327): Mirror 10: 0x209aaa49 (stale) Mirror 11: 0x209aaa49 (stale) Mirror 14: 0x209aaa49 (stale) Mirror 15: 0x209aaa49 (stale) Mirror 18: 0x209aaa49 (stale) Mirror 19: 0x209aaa49 (stale) Mirror 21: 0x209aaa49 (stale) Mirror 22: 0x209aaa49 (stale) Mirror 23: 0x209aaa49 (stale) Mirror 24: 0x209aaa49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a327: skipped 000546 1787305394.366487 MIRROR_RESYNC 11 mirrors 000547 1787305396.464228 MIRROR_RESYNC 11 mirrors 000550 1787305397.129102 read 0x038797 thru 0x03a326 (0x1b90 bytes) 000551 1787305397.134227 write 0x001db2 thru 0x00ecf0 (0xcf3f bytes) 000552 1787305397.242289 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3a327): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a327): Mirror 10: 0xfe253ea7 (stale) Mirror 11: 0xfe253ea7 (stale) Mirror 14: 0xfe253ea7 (stale) Mirror 15: 0xfe253ea7 (stale) Mirror 18: 0xfe253ea7 (stale) Mirror 19: 0xfe253ea7 (stale) Mirror 21: 0xfe253ea7 (stale) Mirror 22: 0xfe253ea7 (stale) Mirror 23: 0xfe253ea7 (stale) Mirror 24: 0xfe253ea7 (stale) CRC-32 checksum value for chunk [0, 0x3a327): Mirror 1: 0xcf43fb33 Mirror 25: 0xcf43fb33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a327: skipped 000553 1787305398.736062 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a327): Mirror 10: 0xfe253ea7 (stale) Mirror 11: 0xfe253ea7 (stale) Mirror 15: 0xfe253ea7 (stale) Mirror 18: 0xfe253ea7 (stale) Mirror 19: 0xfe253ea7 (stale) Mirror 21: 0xfe253ea7 (stale) Mirror 22: 0xfe253ea7 (stale) Mirror 23: 0xfe253ea7 (stale) Mirror 24: 0xfe253ea7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a327): Mirror 10: 0xfe253ea7 (stale) Mirror 11: 0xfe253ea7 (stale) Mirror 15: 0xfe253ea7 (stale) Mirror 18: 0xfe253ea7 (stale) Mirror 19: 0xfe253ea7 (stale) Mirror 21: 0xfe253ea7 (stale) Mirror 22: 0xfe253ea7 (stale) Mirror 23: 0xfe253ea7 (stale) Mirror 24: 0xfe253ea7 (stale) CRC-32 checksum value for chunk [0, 0x3a327): Mirror 1: 0xcf43fb33 Mirror 25: 0xcf43fb33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a327: skipped 000554 1787305399.758206 mapwrite 0x0135f1 thru 0x013637 (0x47 bytes) 000555 1787305399.956078 mapwrite 0x0174ce thru 0x01cfd6 (0x5b09 bytes) 000557 1787305400.084343 trunc from 0x03a327 to 0x03a9fe (0x6d8 bytes) 000558 1787305400.155904 write 0x035e87 thru 0x03aa92 (0x4c0c bytes) 000560 1787305400.180225 trunc from 0x03aa93 to 0x02b337 (0xf75b bytes) 000561 1787305400.206091 mapread 0x0201fc thru 0x0292bb (0x90c0 bytes) 000562 1787305400.232426 write 0x00ca39 thru 0x0110da (0x46a2 bytes) 000563 1787305400.245128 write 0x015b20 thru 0x01cf95 (0x7476 bytes) 000564 1787305400.247625 write 0x02e7e7 thru 0x03a845 (0xc05f bytes) 000565 1787305400.250121 trunc from 0x03a846 to 0x02ec9f (0xbba6 bytes) 000566 1787305400.273471 mapread 0x002c34 thru 0x008971 (0x5d3e bytes) 000569 1787305400.287455 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2eca0): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eca0): Mirror 10: 0x92ac27e5 (stale) Mirror 11: 0x92ac27e5 (stale) Mirror 15: 0x92ac27e5 (stale) Mirror 18: 0x92ac27e5 (stale) Mirror 19: 0x92ac27e5 (stale) Mirror 21: 0x92ac27e5 (stale) Mirror 22: 0x92ac27e5 (stale) Mirror 23: 0x92ac27e5 (stale) Mirror 24: 0x92ac27e5 (stale) Mirror 25: 0x7c06ad4 (stale) CRC-32 checksum value for chunk [0, 0x2eca0): Mirror 1: 0x3c3aac02 Mirror 26: 0x3c3aac02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eca0: skipped 000572 1787305401.410727 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2eca0): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eca0): Mirror 10: 0x92ac27e5 (stale) Mirror 11: 0x92ac27e5 (stale) Mirror 15: 0x92ac27e5 (stale) Mirror 18: 0x92ac27e5 (stale) Mirror 19: 0x92ac27e5 (stale) Mirror 21: 0x92ac27e5 (stale) Mirror 22: 0x92ac27e5 (stale) Mirror 23: 0x92ac27e5 (stale) Mirror 24: 0x92ac27e5 (stale) Mirror 25: 0x7c06ad4 (stale) CRC-32 checksum value for chunk [0, 0x2eca0): Mirror 1: 0x3c3aac02 Mirror 26: 0x3c3aac02 Mirror 27: 0x3c3aac02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eca0: skipped 000573 1787305402.509698 mapread 0x028803 thru 0x02ec9f (0x649d bytes) 000574 1787305402.532753 mapread 0x01d9f1 thru 0x01f3cf (0x19df bytes) 000577 1787305402.548159 write 0x0289f0 thru 0x035075 (0xc686 bytes) 000579 1787305402.673513 mapread 0x02bbd4 thru 0x035075 (0x94a2 bytes) 000580 1787305402.674614 read 0x01347f thru 0x022708 (0xf28a bytes) 000581 1787305402.703483 mapread 0x00db84 thru 0x01cd15 (0xf192 bytes) 000582 1787305402.713531 mapread 0x00d505 thru 0x017e4b (0xa947 bytes) 000583 1787305402.714614 mapread 0x019901 thru 0x02878e (0xee8e bytes) 000585 1787305402.725015 write 0x02db3c thru 0x03b903 (0xddc8 bytes) 000586 1787305402.727757 mapread 0x017c70 thru 0x021365 (0x96f6 bytes) 000587 1787305402.737511 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b904): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b904): Mirror 10: 0xf9533b1f (stale) Mirror 11: 0xf9533b1f (stale) Mirror 15: 0xf9533b1f (stale) Mirror 18: 0xf9533b1f (stale) Mirror 19: 0xf9533b1f (stale) Mirror 21: 0xf9533b1f (stale) Mirror 22: 0xf9533b1f (stale) Mirror 23: 0xf9533b1f (stale) Mirror 24: 0xf9533b1f (stale) Mirror 25: 0xaf3e524f (stale) Mirror 26: 0xab73a3a0 (stale) Mirror 27: 0xab73a3a0 (stale) CRC-32 checksum value for chunk [0, 0x3b904): Mirror 1: 0xfdc0816 Mirror 28: 0xfdc0816 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b904: skipped 000589 1787305404.016142 trunc from 0x03b904 to 0x016749 (0x251ba bytes) 000590 1787305404.106831 read 0x001d06 thru 0x00fb09 (0xde04 bytes) 000591 1787305404.114291 read 0x0061bf thru 0x00661c (0x45e bytes) 000592 1787305404.114980 write 0x02ccb3 thru 0x02fae7 (0x2e35 bytes) 000593 1787305404.116145 write 0x011158 thru 0x017920 (0x67c9 bytes) 000596 1787305404.128282 mapread 0x013971 thru 0x01b880 (0x7f10 bytes) 000597 1787305404.133257 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2fae8): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fae8): Mirror 10: 0x8d4a729a (stale) Mirror 11: 0x8d4a729a (stale) Mirror 15: 0x8d4a729a (stale) Mirror 18: 0x8d4a729a (stale) Mirror 19: 0x8d4a729a (stale) Mirror 21: 0x8d4a729a (stale) Mirror 22: 0x8d4a729a (stale) Mirror 23: 0x8d4a729a (stale) Mirror 24: 0x8d4a729a (stale) Mirror 25: 0x8c624943 (stale) Mirror 26: 0x10c3a056 (stale) Mirror 27: 0x10c3a056 (stale) Mirror 28: 0x1294caeb (stale) CRC-32 checksum value for chunk [0, 0x2fae8): Mirror 1: 0x8a9dc602 Mirror 29: 0x8a9dc602 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fae8: skipped 000599 1787305405.316916 write 0x037c92 thru 0x0390b0 (0x141f bytes) 000600 1787305405.401573 write 0x015dc0 thru 0x018413 (0x2654 bytes) 000602 1787305405.417062 trunc from 0x0390b1 to 0x039bd8 (0xb28 bytes) 000604 1787305405.477454 mapwrite 0x0358da thru 0x0366ef (0xe16 bytes) 000605 1787305405.528933 trunc from 0x039bd9 to 0x006a16 (0x331c2 bytes) 000607 1787305405.553162 MIRROR_RESYNC 15 mirrors 000608 1787305407.886662 write 0x0291ae thru 0x036fa6 (0xddf9 bytes) 000610 1787305407.955543 read 0x026ef2 thru 0x02f0e7 (0x81f6 bytes) 000611 1787305407.972136 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x36fa7): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fa7): Mirror 10: 0x2adb3645 (stale) Mirror 11: 0x2adb3645 (stale) Mirror 15: 0x2adb3645 (stale) Mirror 18: 0x2adb3645 (stale) Mirror 19: 0x2adb3645 (stale) Mirror 21: 0x2adb3645 (stale) Mirror 22: 0x2adb3645 (stale) Mirror 23: 0x2adb3645 (stale) Mirror 24: 0x2adb3645 (stale) Mirror 25: 0x2adb3645 (stale) Mirror 26: 0x2adb3645 (stale) Mirror 27: 0x2adb3645 (stale) Mirror 28: 0x2adb3645 (stale) Mirror 29: 0x2adb3645 (stale) CRC-32 checksum value for chunk [0, 0x36fa7): Mirror 1: 0xdb388932 Mirror 30: 0xdb388932 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fa7: skipped 000614 1787305409.000937 mapwrite 0x019fd2 thru 0x023a84 (0x9ab3 bytes) 000615 1787305409.224979 write 0x00ccd0 thru 0x00e4d9 (0x180a bytes) 000616 1787305409.235265 mapwrite 0x025952 thru 0x03055b (0xac0a bytes) 000618 1787305409.289000 write 0x024a64 thru 0x02bdb7 (0x7354 bytes) 000619 1787305409.290373 trunc from 0x036fa7 to 0x021cfd (0x152a9 bytes) 000624 1787305409.310705 mapwrite 0x024a39 thru 0x03436f (0xf937 bytes) 000625 1787305409.424788 write 0x03832a thru 0x03ffff (0x7cd6 bytes) 000627 1787305409.432350 write 0x0120e6 thru 0x01c454 (0xa36f bytes) 000628 1787305409.440142 mapread 0x013fa5 thru 0x01e268 (0xa2c4 bytes) 000633 1787305409.446145 write 0x01885d thru 0x01ac77 (0x241b bytes) 000634 1787305409.449754 mapread 0x031409 thru 0x038c52 (0x784a bytes) 000636 1787305409.458919 mapread 0x00de17 thru 0x0192c2 (0xb4ac bytes) 000637 1787305409.472531 read 0x02c9ff thru 0x02cfb7 (0x5b9 bytes) 000638 1787305409.473843 read 0x03f4f3 thru 0x03ffff (0xb0d bytes) 000639 1787305409.477671 write 0x00ac79 thru 0x0192d3 (0xe65b bytes) 000642 1787305409.480258 write 0x001858 thru 0x00f0ac (0xd855 bytes) 000643 1787305409.488806 read 0x00b37c thru 0x00ff3a (0x4bbf bytes) 000644 1787305409.489432 read 0x010e2f thru 0x01946e (0x8640 bytes) 000645 1787305409.489985 trunc from 0x040000 to 0x00f859 (0x307a6 bytes) 000646 1787305409.510506 read 0x002f08 thru 0x00e758 (0xb851 bytes) 000649 1787305409.514159 write 0x039e30 thru 0x03ffff (0x61d0 bytes) truncating to largest ever: 0x3fe1f 000651 1787305409.517853 trunc from 0x040000 to 0x03fe1e (0x1e1 bytes) 000652 1787305409.540131 mapwrite 0x03dbd6 thru 0x03ffff (0x242a bytes) 000653 1787305409.610979 read 0x0295a3 thru 0x029b63 (0x5c1 bytes) 000654 1787305409.624152 mapread 0x03d264 thru 0x03ffff (0x2d9c bytes) 000655 1787305409.625525 write 0x01fb30 thru 0x029a8e (0x9f5f bytes) 000659 1787305409.636690 mapwrite 0x017c9f thru 0x02701b (0xf37d bytes) 000662 1787305409.750908 write 0x03b271 thru 0x03ffff (0x4d8f bytes) 000671 1787305409.764178 MIRROR_RESYNC 16 mirrors 000673 1787305412.449369 mapwrite 0x003b60 thru 0x00c2fc (0x879d bytes) 000675 1787305412.732912 mapwrite 0x01dfe2 thru 0x02553c (0x755b bytes) 000676 1787305412.842969 write 0x00b6bd thru 0x01193d (0x6281 bytes) 000678 1787305412.859044 read 0x012ee9 thru 0x013285 (0x39d bytes) 000679 1787305412.867978 trunc from 0x040000 to 0x02ebdb (0x11424 bytes) 000680 1787305412.897428 mapwrite 0x00648d thru 0x01488d (0xe401 bytes) 000684 1787305412.970166 write 0x00f322 thru 0x01c18b (0xce6a bytes) 000685 1787305412.972822 mapread 0x0166ce thru 0x01d590 (0x6ec3 bytes) 000686 1787305412.974216 mapread 0x02ba40 thru 0x02ebdb (0x319c bytes) 000687 1787305412.984617 mapread 0x01614b thru 0x01f87e (0x9734 bytes) 000689 1787305412.985997 read 0x013dad thru 0x01caeb (0x8d3f bytes) 000690 1787305412.986577 mapwrite 0x03a94f thru 0x03c77e (0x1e30 bytes) 000692 1787305413.092000 write 0x02797b thru 0x02c2fe (0x4984 bytes) 000693 1787305413.093058 trunc from 0x03c77f to 0x01cf8c (0x1f7f2 bytes) 000697 1787305413.129367 trunc from 0x01cf8d to 0x00f69e (0xd8ee bytes) 000698 1787305413.155162 trunc from 0x00f69f to 0x0300cb (0x20a2d bytes) 000699 1787305413.173006 trunc from 0x0300cc to 0x014c2c (0x1b49f bytes) 000701 1787305413.192406 mapread 0x00b90f thru 0x014c2c (0x931e bytes) 000702 1787305413.222386 read 0x00684a thru 0x014c2c (0xe3e3 bytes) 000703 1787305413.223103 trunc from 0x014c2d to 0x00e35c (0x68d0 bytes) 000704 1787305413.240912 trunc from 0x00e35d to 0x03355f (0x25203 bytes) 000705 1787305413.259683 trunc from 0x033560 to 0x0088ae (0x2acb1 bytes) 000706 1787305413.289902 mapread 0x002fe9 thru 0x0071fb (0x4213 bytes) 000707 1787305413.303397 MIRROR_RESYNC 16 mirrors 000708 1787305415.899139 read 0x001573 thru 0x0088ae (0x733c bytes) 000709 1787305415.910291 mapread 0x0039ea thru 0x003d0e (0x325 bytes) 000713 1787305415.915602 read 0x006452 thru 0x0088ae (0x245d bytes) 000714 1787305415.916201 read 0x005627 thru 0x0088ae (0x3288 bytes) 000716 1787305415.917805 mapwrite 0x026cf5 thru 0x0293de (0x26ea bytes) 000717 1787305416.147105 mapwrite 0x03c902 thru 0x03ffff (0x36fe bytes) 000719 1787305416.224668 read 0x00050a thru 0x004249 (0x3d40 bytes) 000721 1787305416.226741 read 0x0006f1 thru 0x0024f3 (0x1e03 bytes) 000724 1787305416.230107 read 0x002593 thru 0x00ac9c (0x870a bytes) 000727 1787305416.232712 write 0x024e5c thru 0x02c0c8 (0x726d bytes) 000738 1787305416.243918 trunc from 0x040000 to 0x02856b (0x17a94 bytes) 000740 1787305416.269809 write 0x031698 thru 0x03654d (0x4eb6 bytes) 000742 1787305416.277935 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3654e): Mirror 10: 0xd35f63e3 (stale) Mirror 11: 0xd35f63e3 (stale) Mirror 15: 0xd35f63e3 (stale) Mirror 18: 0xd35f63e3 (stale) Mirror 19: 0xd35f63e3 (stale) Mirror 21: 0xd35f63e3 (stale) Mirror 22: 0xd35f63e3 (stale) Mirror 23: 0xd35f63e3 (stale) Mirror 24: 0xd35f63e3 (stale) Mirror 25: 0xd35f63e3 (stale) Mirror 27: 0xd35f63e3 (stale) Mirror 28: 0xd35f63e3 (stale) Mirror 29: 0xd35f63e3 (stale) Mirror 30: 0xd35f63e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3654e): Mirror 10: 0xd35f63e3 (stale) Mirror 11: 0xd35f63e3 (stale) Mirror 15: 0xd35f63e3 (stale) Mirror 18: 0xd35f63e3 (stale) Mirror 19: 0xd35f63e3 (stale) Mirror 21: 0xd35f63e3 (stale) Mirror 22: 0xd35f63e3 (stale) Mirror 23: 0xd35f63e3 (stale) Mirror 24: 0xd35f63e3 (stale) Mirror 25: 0xd35f63e3 (stale) Mirror 27: 0xd35f63e3 (stale) Mirror 28: 0xd35f63e3 (stale) Mirror 29: 0xd35f63e3 (stale) Mirror 30: 0xd35f63e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3654e: skipped 000743 1787305417.172657 trunc from 0x03654e to 0x00dd92 (0x287bb bytes) 000745 1787305417.199168 mapread 0x00220a thru 0x004ad6 (0x28cd bytes) 000747 1787305417.233620 read 0x008060 thru 0x00dd92 (0x5d33 bytes) 000749 1787305417.235777 write 0x034858 thru 0x03ffff (0xb7a8 bytes) 000750 1787305417.237337 mapwrite 0x020370 thru 0x02e05a (0xdceb bytes) 000759 1787305417.390464 read 0x025df7 thru 0x027577 (0x1781 bytes) 000760 1787305417.391465 mapwrite 0x03b273 thru 0x03ffff (0x4d8d bytes) 000761 1787305417.449112 trunc from 0x040000 to 0x0328fd (0xd702 bytes) 000763 1787305417.469964 write 0x00e5a2 thru 0x013af5 (0x5554 bytes) 000764 1787305417.487009 trunc from 0x0328fe to 0x0004f6 (0x32407 bytes) 000765 1787305417.512587 mapread 0x000391 thru 0x0004f6 (0x166 bytes) 000768 1787305417.517837 trunc from 0x0004f7 to 0x01a321 (0x19e2b bytes) 000769 1787305417.540538 MIRROR_RESYNC 15 mirrors 000770 1787305419.698750 MIRROR_EXTEND to 16 mirrors 000771 1787305421.001128 mapwrite 0x031f79 thru 0x035655 (0x36dd bytes) 000775 1787305421.187417 mapwrite 0x03e947 thru 0x03ffff (0x16b9 bytes) 000776 1787305421.264872 mapwrite 0x00a149 thru 0x014cb7 (0xab6f bytes) 000778 1787305421.340722 read 0x0361c1 thru 0x03ffff (0x9e3f bytes) 000780 1787305421.347004 read 0x0132c0 thru 0x02060d (0xd34e bytes) 000781 1787305421.356831 write 0x004aa1 thru 0x007cad (0x320d bytes) 000782 1787305421.358134 mapread 0x038bba thru 0x038c9e (0xe5 bytes) 000784 1787305421.360615 trunc from 0x040000 to 0x03b461 (0x4b9e bytes) 000786 1787305421.381694 read 0x019529 thru 0x024151 (0xac29 bytes) 000787 1787305421.382900 write 0x02e510 thru 0x0304ec (0x1fdd bytes) 000790 1787305421.390952 mapwrite 0x010ba7 thru 0x0115d1 (0xa2b bytes) 000791 1787305421.487434 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b462): Mirror 10: 0xd6b85ede (stale) Mirror 11: 0xd6b85ede (stale) Mirror 15: 0xd6b85ede (stale) Mirror 18: 0xd6b85ede (stale) Mirror 19: 0xd6b85ede (stale) Mirror 21: 0xd6b85ede (stale) Mirror 22: 0xd6b85ede (stale) Mirror 23: 0xd6b85ede (stale) Mirror 24: 0xd6b85ede (stale) Mirror 25: 0xd6b85ede (stale) Mirror 28: 0xd6b85ede (stale) Mirror 29: 0xd6b85ede (stale) Mirror 30: 0xd6b85ede (stale) Mirror 31: 0xd6b85ede (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b462): Mirror 10: 0xd6b85ede (stale) Mirror 11: 0xd6b85ede (stale) Mirror 15: 0xd6b85ede (stale) Mirror 18: 0xd6b85ede (stale) Mirror 19: 0xd6b85ede (stale) Mirror 21: 0xd6b85ede (stale) Mirror 22: 0xd6b85ede (stale) Mirror 23: 0xd6b85ede (stale) Mirror 24: 0xd6b85ede (stale) Mirror 25: 0xd6b85ede (stale) Mirror 28: 0xd6b85ede (stale) Mirror 29: 0xd6b85ede (stale) Mirror 30: 0xd6b85ede (stale) Mirror 31: 0xd6b85ede (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b462: skipped 000793 1787305422.366544 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3b462): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b462): Mirror 10: 0xd6b85ede (stale) Mirror 11: 0xd6b85ede (stale) Mirror 15: 0xd6b85ede (stale) Mirror 18: 0xd6b85ede (stale) Mirror 19: 0xd6b85ede (stale) Mirror 21: 0xd6b85ede (stale) Mirror 22: 0xd6b85ede (stale) Mirror 23: 0xd6b85ede (stale) Mirror 24: 0xd6b85ede (stale) Mirror 25: 0xd6b85ede (stale) Mirror 28: 0xd6b85ede (stale) Mirror 29: 0xd6b85ede (stale) Mirror 30: 0xd6b85ede (stale) Mirror 31: 0xd6b85ede (stale) CRC-32 checksum value for chunk [0, 0x3b462): Mirror 1: 0xaaaa15df Mirror 32: 0xaaaa15df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b462: skipped 000794 1787305423.925402 read 0x015170 thru 0x020074 (0xaf05 bytes) 000796 1787305423.938342 mapread 0x036dfd thru 0x03887f (0x1a83 bytes) 000799 1787305423.966468 trunc from 0x03b462 to 0x012cdd (0x28784 bytes) 000800 1787305424.072744 read 0x00a0db thru 0x012cdd (0x8c03 bytes) 000801 1787305424.093277 read 0x003f1c thru 0x010758 (0xc83d bytes) 000803 1787305424.095825 mapread 0x00c030 thru 0x012cdd (0x6cae bytes) 000805 1787305424.097772 write 0x0279cb thru 0x02dcb6 (0x62ec bytes) 000807 1787305424.100210 write 0x0072aa thru 0x0106fd (0x9454 bytes) 000808 1787305424.101814 mapwrite 0x0355f8 thru 0x03ffff (0xaa08 bytes) 000810 1787305424.260826 mapread 0x01de7a thru 0x02a39b (0xc522 bytes) 000812 1787305424.265909 trunc from 0x040000 to 0x015a45 (0x2a5ba bytes) 000813 1787305424.286179 mapwrite 0x00a49f thru 0x015b23 (0xb685 bytes) 000816 1787305424.361611 MIRROR_RESYNC 16 mirrors 000818 1787305426.900720 mapread 0x011546 thru 0x015b23 (0x45de bytes) 000820 1787305426.932044 MIRROR_SPLIT mirror 12 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000822 1787305427.551648 mapwrite 0x0012b9 thru 0x00e7b0 (0xd4f8 bytes) 000823 1787305428.053061 write 0x033d4e thru 0x03ffff (0xc2b2 bytes) 000824 1787305428.060703 trunc from 0x040000 to 0x01e2ab (0x21d54 bytes) 000828 1787305428.101855 mapwrite 0x00356d thru 0x00e6ca (0xb15e bytes) 000830 1787305428.195365 mapread 0x0149f8 thru 0x01ca2e (0x8037 bytes) 000831 1787305428.217532 mapread 0x013c70 thru 0x01a597 (0x6928 bytes) 000832 1787305428.222985 MIRROR_RESYNC 15 mirrors 000833 1787305431.199708 write 0x039e7e thru 0x03ffff (0x6182 bytes) 000835 1787305431.303039 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xff80f6a4 (stale) Mirror 15: 0xff80f6a4 (stale) Mirror 18: 0xff80f6a4 (stale) Mirror 19: 0xff80f6a4 (stale) Mirror 21: 0xff80f6a4 (stale) Mirror 22: 0xff80f6a4 (stale) Mirror 23: 0xff80f6a4 (stale) Mirror 24: 0xff80f6a4 (stale) Mirror 25: 0xff80f6a4 (stale) Mirror 28: 0xff80f6a4 (stale) Mirror 30: 0xff80f6a4 (stale) Mirror 31: 0xff80f6a4 (stale) Mirror 32: 0xff80f6a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xff80f6a4 (stale) Mirror 15: 0xff80f6a4 (stale) Mirror 18: 0xff80f6a4 (stale) Mirror 19: 0xff80f6a4 (stale) Mirror 21: 0xff80f6a4 (stale) Mirror 22: 0xff80f6a4 (stale) Mirror 23: 0xff80f6a4 (stale) Mirror 24: 0xff80f6a4 (stale) Mirror 25: 0xff80f6a4 (stale) Mirror 28: 0xff80f6a4 (stale) Mirror 30: 0xff80f6a4 (stale) Mirror 31: 0xff80f6a4 (stale) Mirror 32: 0xff80f6a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000838 1787305432.181411 mapwrite 0x03bbf2 thru 0x03ffff (0x440e bytes) 000839 1787305432.305000 mapread 0x0066d6 thru 0x00cb6f (0x649a bytes) 000841 1787305432.326025 trunc from 0x040000 to 0x0373b6 (0x8c49 bytes) 000842 1787305432.370356 trunc from 0x0373b7 to 0x01620d (0x211a9 bytes) 000843 1787305432.426923 mapread 0x00ea82 thru 0x013c41 (0x51c0 bytes) 000844 1787305432.439238 mapwrite 0x00eda4 thru 0x013552 (0x47af bytes) 000845 1787305432.520042 mapwrite 0x00e36a thru 0x01b752 (0xd3e9 bytes) 000846 1787305432.617702 mapwrite 0x00b1f6 thru 0x00d323 (0x212e bytes) 000847 1787305432.662609 write 0x00c82d thru 0x0176d6 (0xaeaa bytes) 000848 1787305432.664075 read 0x019fb3 thru 0x01b752 (0x17a0 bytes) 000850 1787305432.674232 mapwrite 0x00d4a9 thru 0x010eee (0x3a46 bytes) 000851 1787305432.750368 mapread 0x015c9a thru 0x01b381 (0x56e8 bytes) 000854 1787305432.762678 mapwrite 0x029ee7 thru 0x02ee72 (0x4f8c bytes) 000858 1787305432.893798 trunc from 0x02ee73 to 0x02d73f (0x1733 bytes) 000861 1787305432.929882 trunc from 0x02d740 to 0x014850 (0x18eef bytes) 000862 1787305432.967279 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14851): Mirror 11: 0xa2e2e2f8 (stale) Mirror 15: 0xa2e2e2f8 (stale) Mirror 19: 0xa2e2e2f8 (stale) Mirror 21: 0xa2e2e2f8 (stale) Mirror 22: 0xa2e2e2f8 (stale) Mirror 23: 0xa2e2e2f8 (stale) Mirror 24: 0xa2e2e2f8 (stale) Mirror 25: 0xa2e2e2f8 (stale) Mirror 28: 0xa2e2e2f8 (stale) Mirror 30: 0xa2e2e2f8 (stale) Mirror 31: 0xa2e2e2f8 (stale) Mirror 32: 0xa2e2e2f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14851): Mirror 11: 0xa2e2e2f8 (stale) Mirror 15: 0xa2e2e2f8 (stale) Mirror 19: 0xa2e2e2f8 (stale) Mirror 21: 0xa2e2e2f8 (stale) Mirror 22: 0xa2e2e2f8 (stale) Mirror 23: 0xa2e2e2f8 (stale) Mirror 24: 0xa2e2e2f8 (stale) Mirror 25: 0xa2e2e2f8 (stale) Mirror 28: 0xa2e2e2f8 (stale) Mirror 30: 0xa2e2e2f8 (stale) Mirror 31: 0xa2e2e2f8 (stale) Mirror 32: 0xa2e2e2f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14851: skipped 000865 1787305433.792281 mapread 0x009bb3 thru 0x00a807 (0xc55 bytes) 000866 1787305433.813566 write 0x03871c thru 0x03ffff (0x78e4 bytes) 000867 1787305433.816571 mapread 0x00d563 thru 0x0136d4 (0x6172 bytes) 000868 1787305433.834455 read 0x02f188 thru 0x039b09 (0xa982 bytes) 000869 1787305433.846840 mapwrite 0x001fbe thru 0x00f7fa (0xd83d bytes) 000871 1787305434.032475 mapwrite 0x038807 thru 0x03ffff (0x77f9 bytes) 000872 1787305434.157621 mapwrite 0x036a20 thru 0x03e1e3 (0x77c4 bytes) 000873 1787305434.296230 mapwrite 0x00363f thru 0x00e733 (0xb0f5 bytes) 000874 1787305434.425825 mapwrite 0x00de22 thru 0x017fdd (0xa1bc bytes) 000877 1787305434.616582 write 0x01c7c3 thru 0x02bf58 (0xf796 bytes) 000878 1787305434.625476 trunc from 0x040000 to 0x028306 (0x17cf9 bytes) 000880 1787305434.680036 MIRROR_RESYNC 13 mirrors 000881 1787305436.925523 write 0x036102 thru 0x03ffff (0x9efe bytes) 000882 1787305437.072793 mapwrite 0x037fcd thru 0x03ffff (0x8033 bytes) 000884 1787305437.173120 mapwrite 0x03eacd thru 0x03ffff (0x1533 bytes) 000885 1787305437.280636 mapread 0x015dde thru 0x017fe3 (0x2206 bytes) 000889 1787305437.318937 mapwrite 0x02c3ef thru 0x039cc1 (0xd8d3 bytes) 000890 1787305437.390923 mapread 0x026065 thru 0x0315ca (0xb566 bytes) 000891 1787305437.392420 mapread 0x014aab thru 0x02185c (0xcdb2 bytes) 000893 1787305437.401642 mapwrite 0x0179f7 thru 0x0220cb (0xa6d5 bytes) 000895 1787305437.463263 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7f5f6b60 (stale) Mirror 15: 0x7f5f6b60 (stale) Mirror 19: 0x7f5f6b60 (stale) Mirror 21: 0x7f5f6b60 (stale) Mirror 22: 0x7f5f6b60 (stale) Mirror 23: 0x7f5f6b60 (stale) Mirror 24: 0x7f5f6b60 (stale) Mirror 25: 0x7f5f6b60 (stale) Mirror 28: 0x7f5f6b60 (stale) Mirror 30: 0x7f5f6b60 (stale) Mirror 31: 0x7f5f6b60 (stale) Mirror 32: 0x7f5f6b60 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaff3ea68 Mirror 33: 0xaff3ea68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000900 1787305438.742197 read 0x025fba thru 0x030dd6 (0xae1d bytes) 000903 1787305438.755136 read 0x008a13 thru 0x011d2f (0x931d bytes) 000908 1787305438.778227 read 0x01a9d7 thru 0x02199e (0x6fc8 bytes) 000912 1787305438.780802 read 0x001345 thru 0x00bd74 (0xaa30 bytes) 000915 1787305438.782456 mapread 0x0200c6 thru 0x02710c (0x7047 bytes) 000916 1787305438.783695 mapwrite 0x00d08b thru 0x0106d5 (0x364b bytes) 000917 1787305439.055479 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7f5f6b60 (stale) Mirror 15: 0x7f5f6b60 (stale) Mirror 19: 0x7f5f6b60 (stale) Mirror 21: 0x7f5f6b60 (stale) Mirror 22: 0x7f5f6b60 (stale) Mirror 23: 0x7f5f6b60 (stale) Mirror 24: 0x7f5f6b60 (stale) Mirror 25: 0x7f5f6b60 (stale) Mirror 28: 0x7f5f6b60 (stale) Mirror 30: 0x7f5f6b60 (stale) Mirror 31: 0x7f5f6b60 (stale) Mirror 32: 0x7f5f6b60 (stale) Mirror 33: 0xaff3ea68 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4a82682e Mirror 34: 0x4a82682e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000918 1787305440.214335 trunc from 0x040000 to 0x0320dd (0xdf22 bytes) 000921 1787305440.396204 mapread 0x02646a thru 0x0320dd (0xbc74 bytes) 000922 1787305440.430115 read 0x029a78 thru 0x0320dd (0x8666 bytes) 000924 1787305440.431148 mapwrite 0x014746 thru 0x0236eb (0xefa6 bytes) 000925 1787305440.542800 read 0x02c398 thru 0x0320dd (0x5d46 bytes) 000926 1787305440.543441 trunc from 0x0320de to 0x01deb1 (0x1422c bytes) 000927 1787305440.582884 mapread 0x00cea5 thru 0x0116ea (0x4846 bytes) 000928 1787305440.605517 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1deb2): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1deb2): Mirror 11: 0xb16b8928 (stale) Mirror 15: 0xb16b8928 (stale) Mirror 19: 0xb16b8928 (stale) Mirror 21: 0xb16b8928 (stale) Mirror 22: 0xb16b8928 (stale) Mirror 23: 0xb16b8928 (stale) Mirror 24: 0xb16b8928 (stale) Mirror 25: 0xb16b8928 (stale) Mirror 28: 0xb16b8928 (stale) Mirror 30: 0xb16b8928 (stale) Mirror 31: 0xb16b8928 (stale) Mirror 32: 0xb16b8928 (stale) Mirror 33: 0x6da0e505 (stale) Mirror 34: 0x65f9d06c (stale) CRC-32 checksum value for chunk [0, 0x1deb2): Mirror 1: 0x80b3c113 Mirror 35: 0x80b3c113 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1deb2: skipped 000931 1787305442.277516 write 0x03a67f thru 0x03ffff (0x5981 bytes) 000932 1787305442.462266 read 0x03e76a thru 0x03ffff (0x1896 bytes) 000934 1787305442.464478 mapwrite 0x02b5e5 thru 0x0371c8 (0xbbe4 bytes) 000935 1787305442.591695 write 0x02be8d thru 0x03528d (0x9401 bytes) 000936 1787305442.601793 write 0x000109 thru 0x00eb01 (0xe9f9 bytes) 000937 1787305442.635536 mapwrite 0x01022b thru 0x0162c1 (0x6097 bytes) 000938 1787305442.714026 write 0x002410 thru 0x008328 (0x5f19 bytes) 000939 1787305442.715505 mapwrite 0x004f3f thru 0x008059 (0x311b bytes) 000940 1787305442.762176 write 0x01bcd8 thru 0x02a675 (0xe99e bytes) 000942 1787305442.782768 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7f5f6b60 (stale) Mirror 15: 0x7f5f6b60 (stale) Mirror 21: 0x7f5f6b60 (stale) Mirror 22: 0x7f5f6b60 (stale) Mirror 23: 0x7f5f6b60 (stale) Mirror 24: 0x7f5f6b60 (stale) Mirror 25: 0x7f5f6b60 (stale) Mirror 28: 0x7f5f6b60 (stale) Mirror 30: 0x7f5f6b60 (stale) Mirror 31: 0x7f5f6b60 (stale) Mirror 32: 0x7f5f6b60 (stale) Mirror 33: 0xaff3ea68 (stale) Mirror 34: 0x4a82682e (stale) Mirror 35: 0x73b209dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7f5f6b60 (stale) Mirror 15: 0x7f5f6b60 (stale) Mirror 21: 0x7f5f6b60 (stale) Mirror 22: 0x7f5f6b60 (stale) Mirror 23: 0x7f5f6b60 (stale) Mirror 24: 0x7f5f6b60 (stale) Mirror 25: 0x7f5f6b60 (stale) Mirror 28: 0x7f5f6b60 (stale) Mirror 30: 0x7f5f6b60 (stale) Mirror 31: 0x7f5f6b60 (stale) Mirror 32: 0x7f5f6b60 (stale) Mirror 33: 0xaff3ea68 (stale) Mirror 34: 0x4a82682e (stale) Mirror 35: 0x73b209dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000944 1787305443.889290 trunc from 0x040000 to 0x012f8b (0x2d074 bytes) 000945 1787305443.930212 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x12f8c): Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f8c): Mirror 11: 0xda8870c0 (stale) Mirror 15: 0xda8870c0 (stale) Mirror 21: 0xda8870c0 (stale) Mirror 22: 0xda8870c0 (stale) Mirror 23: 0xda8870c0 (stale) Mirror 24: 0xda8870c0 (stale) Mirror 25: 0xda8870c0 (stale) Mirror 28: 0xda8870c0 (stale) Mirror 30: 0xda8870c0 (stale) Mirror 31: 0xda8870c0 (stale) Mirror 32: 0xda8870c0 (stale) Mirror 33: 0xda8870c0 (stale) Mirror 34: 0xa10241f (stale) Mirror 35: 0xa10241f (stale) CRC-32 checksum value for chunk [0, 0x12f8c): Mirror 1: 0xaf29dbe5 Mirror 36: 0xaf29dbe5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f8c: skipped 000946 1787305445.436401 read 0x00059b thru 0x009f73 (0x99d9 bytes) 000947 1787305445.478095 mapwrite 0x02dbf1 thru 0x03c6ea (0xeafa bytes) 000948 1787305445.864681 read 0x002043 thru 0x00e97c (0xc93a bytes) 000951 1787305445.881458 trunc from 0x03c6eb to 0x035930 (0x6dba bytes) 000954 1787305445.916170 write 0x02b281 thru 0x02bf95 (0xd15 bytes) 000955 1787305445.935957 MIRROR_RESYNC 16 mirrors 000956 1787305448.706566 mapwrite 0x03551a thru 0x037654 (0x213b bytes) 000957 1787305448.986133 mapwrite 0x0366b2 thru 0x03ffff (0x994e bytes) 000958 1787305449.146852 mapwrite 0x01f9b7 thru 0x0250aa (0x56f4 bytes) 000961 1787305449.238721 trunc from 0x040000 to 0x039466 (0x6b99 bytes) 000963 1787305449.271258 read 0x037914 thru 0x039466 (0x1b53 bytes) 000964 1787305449.273056 write 0x038c42 thru 0x03e833 (0x5bf2 bytes) 000965 1787305449.281379 MIRROR_RESYNC 16 mirrors 000966 1787305451.697351 mapwrite 0x0086fd thru 0x00d1ab (0x4aaf bytes) 000967 1787305451.927090 write 0x018d9d thru 0x01c411 (0x3675 bytes) 000969 1787305451.950242 write 0x020d8e thru 0x029e06 (0x9079 bytes) 000972 1787305451.976910 read 0x026e80 thru 0x02709f (0x220 bytes) 000973 1787305451.980764 mapwrite 0x017b1d thru 0x01b3ce (0x38b2 bytes) 000974 1787305452.100995 write 0x02632b thru 0x02f966 (0x963c bytes) 000976 1787305452.118631 read 0x01e6d6 thru 0x0242f2 (0x5c1d bytes) 000978 1787305452.119825 trunc from 0x03e834 to 0x012d43 (0x2baf0 bytes) 000979 1787305452.148828 write 0x03e720 thru 0x03ffff (0x18e0 bytes) 000981 1787305452.162196 MIRROR_SPLIT mirror 5 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdb006547 (stale) Mirror 15: 0xdb006547 (stale) Mirror 21: 0xdb006547 (stale) Mirror 22: 0xdb006547 (stale) Mirror 24: 0xdb006547 (stale) Mirror 25: 0xdb006547 (stale) Mirror 28: 0xdb006547 (stale) Mirror 30: 0xdb006547 (stale) Mirror 31: 0xdb006547 (stale) Mirror 32: 0xdb006547 (stale) Mirror 33: 0xdb006547 (stale) Mirror 34: 0xdb006547 (stale) Mirror 35: 0xdb006547 (stale) Mirror 36: 0xdb006547 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdb006547 (stale) Mirror 15: 0xdb006547 (stale) Mirror 21: 0xdb006547 (stale) Mirror 22: 0xdb006547 (stale) Mirror 24: 0xdb006547 (stale) Mirror 25: 0xdb006547 (stale) Mirror 28: 0xdb006547 (stale) Mirror 30: 0xdb006547 (stale) Mirror 31: 0xdb006547 (stale) Mirror 32: 0xdb006547 (stale) Mirror 33: 0xdb006547 (stale) Mirror 34: 0xdb006547 (stale) Mirror 35: 0xdb006547 (stale) Mirror 36: 0xdb006547 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000982 1787305453.200341 write 0x01eed1 thru 0x02304c (0x417c bytes) 000983 1787305453.215054 mapwrite 0x03ce7e thru 0x03fd1b (0x2e9e bytes) 000984 1787305453.277906 mapread 0x007b21 thru 0x008ebf (0x139f bytes) 000985 1787305453.291018 write 0x0149ca thru 0x021c65 (0xd29c bytes) 000986 1787305453.301026 write 0x0138db thru 0x01b88d (0x7fb3 bytes) 000988 1787305453.314649 mapread 0x001f44 thru 0x004329 (0x23e6 bytes) 000990 1787305453.316682 write 0x015cf4 thru 0x020638 (0xa945 bytes) 000992 1787305453.318852 mapread 0x027078 thru 0x02f67d (0x8606 bytes) 000993 1787305453.360048 mapwrite 0x005646 thru 0x00f9ea (0xa3a5 bytes) 000995 1787305453.451378 mapread 0x0072bf thru 0x00ef60 (0x7ca2 bytes) 000996 1787305453.471109 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdb006547 (stale) Mirror 15: 0xdb006547 (stale) Mirror 21: 0xdb006547 (stale) Mirror 22: 0xdb006547 (stale) Mirror 24: 0xdb006547 (stale) Mirror 25: 0xdb006547 (stale) Mirror 28: 0xdb006547 (stale) Mirror 30: 0xdb006547 (stale) Mirror 31: 0xdb006547 (stale) Mirror 33: 0xdb006547 (stale) Mirror 34: 0xdb006547 (stale) Mirror 35: 0xdb006547 (stale) Mirror 36: 0xdb006547 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xdb006547 (stale) Mirror 15: 0xdb006547 (stale) Mirror 21: 0xdb006547 (stale) Mirror 22: 0xdb006547 (stale) Mirror 24: 0xdb006547 (stale) Mirror 25: 0xdb006547 (stale) Mirror 28: 0xdb006547 (stale) Mirror 30: 0xdb006547 (stale) Mirror 31: 0xdb006547 (stale) Mirror 33: 0xdb006547 (stale) Mirror 34: 0xdb006547 (stale) Mirror 35: 0xdb006547 (stale) Mirror 36: 0xdb006547 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000997 1787305454.303246 write 0x011435 thru 0x019a3b (0x8607 bytes) 000998 1787305454.319195 mapread 0x02dc92 thru 0x030a9a (0x2e09 bytes) 000999 1787305454.351926 write 0x008f99 thru 0x00c9ff (0x3a67 bytes) 001000 1787305454.363657 read 0x021ada thru 0x02a74f (0x8c76 bytes) All operations completed A-OK! PASS 70a (148s) == sanity-flr test 71: check mirror extend parent fid ==== 05:44:19 (1787305459) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 05:44:22 (1787305462) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286571 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204199 s, 5.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 05:44:32 (1787305472) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7942112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2880256 .. failed Extending file size to 7183520 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7255808 .. Extending file size to 7451872 .. Extending file size to 9143840 .. Extending file size to 8443904 .. Extending file size to 9114240 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58774 58775 58776 58778 58780 done 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) 192.168.204.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:45:54 (1787305554) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 385408 .. Extending file size to 4136480 .. done Extending file size to 3185408 .. Extending file size to 5046016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7585600 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5836832 .. Extending file size to 8010240 .. Extending file size to 8475392 .. Extending file size to 8247328 .. Extending file size to 9135712 .. Extending file size to 9217632 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8615424 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60443 60444 60445 60446 60448 done 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) 192.168.204.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:47:14 (1787305634) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207235 s, 10.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.530942 s, 3.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xda73af5 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:47:39 (1787305659) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136216 s, 15.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 05:47:47 (1787305667) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f1:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:47:56 (1787305676) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25639 s, 8.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:48:08 (1787305688) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242299 s, 8.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:48:19 (1787305699) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19488 s, 10.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:48:56 (1787305736) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385517 s, 2.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:49:59 (1787305799) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.428754 s, 4.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (68s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:51:07 (1787305867) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. Creating new pool test_204f oleg440-server: Pool lustre.test_204f created Adding targets to pool oleg440-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg440-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.412917 s, 5.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg440-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg440-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg440-server: Pool lustre.test_204f destroyed PASS 204f (91s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:52:38 (1787305958) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:52:49 (1787305969) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:52:59 (1787305979) oleg440-server: Pool lustre.test_206 created Adding targets to pool oleg440-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg440-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg440-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg440-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg440-server: Pool lustre.test_206 destroyed PASS 206 (35s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:53:34 (1787306014) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187186 s, 5.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152847 s, 6.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xeb548106 Mirror 3: 0xeb548106 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:53:50 (1787306030) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:53:54 (1787306034) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:53:58 (1787306038) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.136917 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1800000 Read this (4 bytes): BBBB 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:54:42 (1787306082) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0254733 s, 322 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:54:51 (1787306091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675846 s, 15.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x216 lmm_fid: [0x200000404:0x216:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x216 lmm_fid: [0x200000404:0x216:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834454 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 05:54:53.000000000 -0400 Modify: 2026-08-21 05:54:53.000000000 -0400 Change: 2026-08-21 05:54:55.000000000 -0400 Birth: 2026-08-21 05:54:53.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:55:03 (1787306103) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:55:07 (1787306107) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.580018 s, 70.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0798558 s, 51.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 21 05:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x218:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 21 05:55 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:55:19 (1787306119) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (13s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:55:32 (1787306132) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (17s) == sanity-flr test complete, duration 1964 sec =========== 05:55:49 (1787306149) === sanity-flr: start cleanup 05:55:51 (1787306151) === === sanity-flr: finish cleanup 05:56:00 (1787306160) ===