-----============= acceptance-small: sanity-flr ============----- Fri Aug 28 04:41:31 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg610-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:41:50 (1787906510) === oleg610-client.virtnet: executing check_config_client /mnt/lustre oleg610-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg610-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99ea42ba3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99ea42ba3800.idle_timeout=debug disable quota as required oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:42:20 (1787906540) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7651] == sanity-flr test 0a: lfs mirror create with -N option == 04:42:22 (1787906542) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:42:41 (1787906561) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:42:45 (1787906565) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:42:49 (1787906569) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:43:15 (1787906595) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:43:18 (1787906598) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:43:21 (1787906601) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99ea42ba3800.stats=clear osc.lustre-OST0001-osc-ffff99ea42ba3800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:43:29 (1787906609) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:43:40 (1787906620) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:43:51 (1787906631) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:43:56 (1787906636) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:44:04 (1787906644) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 04:44:14 (1787906654) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:44:21 (1787906661) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:44:24 (1787906664) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:44:32 (1787906672) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:44:42 (1787906682) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:44:50 (1787906690) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.64638 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129656 s, 8.1 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:45:04 (1787906704) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.520535 s, 36.3 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:45:14 (1787906714) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:45:25 (1787906725) Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg610-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg610-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (133s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:47:38 (1787906858) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.244394 s, 12.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:48:21 (1787906901) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg610-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:49:44 (1787906984) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.42733 s, 9.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 28 04:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.173s user 0m0.003s sys 0m0.214s Stopping /mnt/lustre-ost1 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ea42ba3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.368s user 0m0.001s sys 0m0.188s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg610-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg610-server oleg610-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.470s user 0m0.001s sys 0m0.178s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg610-server: oleg610-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg610-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg610-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ea42ba3800.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:51:27 (1787907087) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:51:31 (1787907091) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:51:35 (1787907095) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:51:39 (1787907099) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 04:51:47 (1787907107) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.623426 s, 20.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:52:03 (1787907123) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459954 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265461 s, 39.5 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:52:12 (1787907132) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:52:21 (1787907141) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00140311 s, 730 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00159492 s, 642 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000911441 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165445 s, 619 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 04:52:32 (1787907152) create test files with size 15761371 .. f538d7f586530ce50ddb25e7a1e4569d - 1e91df99e34cbe90ecde874003e455e8 - e351f1ab97d1bd9800f0e23c758bad02 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg610-client.virtnet /mnt/lustre (opts:) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 04:52:56 (1787907176) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:53:12 (1787907192) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:53:18 (1787907198) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161519 s, 6.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355865 s, 29.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 04:53:28 (1787907208) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184638 s, 11.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.508703 s, 8.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.26658 s, 7.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 04:53:46 (1787907226) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:53:49 (1787907229) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:53:52 (1787907232) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690546 s, 15.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 04:54:00 (1787907240) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:54:03 (1787907243) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:54:13 (1787907253) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.207295 s, 50.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg610-client.virtnet /mnt/lustre (opts:) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:54:24 (1787907264) llite.lustre-ffff99ea46213800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.302458 s, 34.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg610-client.virtnet /mnt/lustre (opts:) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff99ea45650000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:54:38 (1787907278) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:55:18 (1787907318) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:55:52 (1787907352) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:55:55 (1787907355) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.217988 s, 14.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.162579 s, 19.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc34ac666 (nosync,stale) Mirror 2: 0xc34ac666 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8b88ea17752ba3582e45aaae53517656 - mirror 1 checksum: eb93c184fb1cbc844b37a948b16d3962 - mirror 2 checksum: 8b88ea17752ba3582e45aaae53517656 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-28T04:55:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-28T04:55:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8b88ea17752ba3582e45aaae53517656 - mirror 1 checksum: eb93c184fb1cbc844b37a948b16d3962 - mirror 2 checksum: eb93c184fb1cbc844b37a948b16d3962 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-28T04:55:58-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-28T04:55:58-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:56:06 (1787907366) Failing mds1 on oleg610-server Stopping /mnt/lustre-mds1 (opts:) on oleg610-server 04:56:18 (1787907378) shut down facet: mds1 facet_host: oleg610-server facet_failover_host: oleg610-server Failover mds1 to oleg610-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg610-server: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-MDT0000 04:56:41 (1787907401) targets are mounted 04:56:41 (1787907401) facet_failover done oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647316 s, 16.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:57:13 (1787907433) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:57:17 (1787907437) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:57:21 (1787907441) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00943675 s, 1.7 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:57:30 (1787907450) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:57:33 (1787907453) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00305612 s, 1.3 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.093909 s, 43.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:57:41 (1787907461) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:58:06 (1787907486) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:58:34 (1787907514) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:58:58 (1787907538) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:59:23 (1787907563) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 04:59:30 (1787907570) mirror create pid 54339 mirror split pid 54340 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7019: 54340 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7019: 54340 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54339 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54340 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 05:00:37 (1787907637) Seed set to 7639 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004 1787907640.278933 mapwrite 0x02f0ab thru 0x03b571 (0xc4c7 bytes) 000006 1787907640.463307 write 0x0271ca thru 0x02ea0a (0x7841 bytes) 000008 1787907640.470266 mapread 0x031010 thru 0x03b571 (0xa562 bytes) 000009 1787907640.472646 mapread 0x029676 thru 0x0320ed (0x8a78 bytes) 000015 1787907640.478920 read 0x002174 thru 0x00cf2b (0xadb8 bytes) 000017 1787907640.495100 mapwrite 0x013295 thru 0x01921c (0x5f88 bytes) 000019 1787907640.614958 write 0x00e6e6 thru 0x0181e2 (0x9afd bytes) truncating to largest ever: 0x1ec3 000020 1787907640.624829 trunc from 0x03b572 to 0x001ec2 (0x396af bytes) 000022 1787907640.655514 read 0x0017f2 thru 0x001ec2 (0x6d1 bytes) truncating to largest ever: 0x6be3 000028 1787907640.673489 trunc from 0x001ec3 to 0x006be2 (0x4d20 bytes) 000031 1787907640.698761 mapread 0x000505 thru 0x003d9e (0x389a bytes) 000032 1787907640.708576 mapread 0x004f8b thru 0x0061f4 (0x126a bytes) 000033 1787907640.710313 read 0x001ecc thru 0x00540f (0x3544 bytes) truncating to largest ever: 0x26078 000035 1787907640.711983 trunc from 0x006be3 to 0x026077 (0x1f495 bytes) 000038 1787907640.754862 write 0x00bcfe thru 0x011edd (0x61e0 bytes) 000040 1787907640.785600 mapread 0x0171fe thru 0x01b20e (0x4011 bytes) 000041 1787907640.786635 mapread 0x00be39 thru 0x00fa89 (0x3c51 bytes) 000044 1787907640.799475 mapread 0x00b101 thru 0x01645a (0xb35a bytes) 000046 1787907640.806638 mapwrite 0x03ab11 thru 0x03f081 (0x4571 bytes) 000047 1787907640.961033 trunc from 0x03f082 to 0x01f684 (0x1f9fd bytes) 000048 1787907640.980999 write 0x02a52b thru 0x035652 (0xb128 bytes) 000049 1787907640.984055 mapwrite 0x02845e thru 0x02a23f (0x1de2 bytes) 000052 1787907641.092381 trunc from 0x035653 to 0x00d22b (0x28427 bytes) 000054 1787907641.134283 read 0x005666 thru 0x0092bf (0x3c5a bytes) truncating to largest ever: 0x3da85 000056 1787907641.135308 trunc from 0x00d22c to 0x03da84 (0x30859 bytes) 000058 1787907641.182533 read 0x01cfef thru 0x01e56e (0x1580 bytes) 000060 1787907641.207892 mapread 0x0261a2 thru 0x0350cc (0xef2b bytes) 000061 1787907641.211567 trunc from 0x03da85 to 0x01f59a (0x1e4ea bytes) 000064 1787907641.234980 read 0x0065d8 thru 0x00d8ae (0x72d7 bytes) 000065 1787907641.235582 mapread 0x017e07 thru 0x01f59a (0x7794 bytes) 000067 1787907641.239011 read 0x01e52d thru 0x01f59a (0x106e bytes) 000068 1787907641.239774 mapwrite 0x00cd66 thru 0x01129e (0x4539 bytes) 000069 1787907641.302607 trunc from 0x01f59b to 0x0072ac (0x182ee bytes) 000070 1787907641.318584 mapwrite 0x039aee thru 0x03ffff (0x6512 bytes) 000071 1787907641.405652 write 0x006c0f thru 0x009f57 (0x3349 bytes) 000072 1787907641.407076 mapwrite 0x01d5c3 thru 0x027dfa (0xa838 bytes) 000075 1787907641.558211 trunc from 0x040000 to 0x02c738 (0x138c7 bytes) 000076 1787907641.625243 mapread 0x003c19 thru 0x0095e0 (0x59c8 bytes) 000077 1787907641.633235 mapread 0x00eaa8 thru 0x01becc (0xd425 bytes) 000083 1787907641.648994 mapread 0x015153 thru 0x01e4c1 (0x936f bytes) truncating to largest ever: 0x3dae4 000090 1787907641.655585 trunc from 0x02c739 to 0x03dae3 (0x113ab bytes) 000094 1787907641.700124 read 0x011bbe thru 0x01aaac (0x8eef bytes) 000095 1787907641.700984 write 0x0262eb thru 0x0287c1 (0x24d7 bytes) 000096 1787907641.705001 trunc from 0x03dae4 to 0x00f01c (0x2eac7 bytes) 000097 1787907641.750116 mapwrite 0x006625 thru 0x009b15 (0x34f1 bytes) 000100 1787907641.920625 mapwrite 0x0010da thru 0x00a5af (0x94d6 bytes) 000101 1787907641.984919 mapread 0x00a981 thru 0x00e4ae (0x3b2e bytes) 000102 1787907641.985823 mapread 0x006c16 thru 0x00cf22 (0x630d bytes) 000103 1787907641.992924 MIRROR_EXTEND to 2 mirrors 000104 1787907643.138204 mapwrite 0x017c3b thru 0x026311 (0xe6d7 bytes) 000105 1787907643.430603 mapwrite 0x03be9c thru 0x03ffff (0x4164 bytes) 000106 1787907643.548155 read 0x017006 thru 0x01dcad (0x6ca8 bytes) 000107 1787907643.549524 mapread 0x01adfb thru 0x01b6ff (0x905 bytes) 000108 1787907643.551450 write 0x003167 thru 0x006db6 (0x3c50 bytes) 000109 1787907643.564401 write 0x03216b thru 0x03c2c7 (0xa15d bytes) 000110 1787907643.573371 read 0x02ff4f thru 0x03b7cb (0xb87d bytes) 000112 1787907643.593024 mapwrite 0x015972 thru 0x01a8da (0x4f69 bytes) 000113 1787907643.719895 write 0x01e06c thru 0x01fa98 (0x1a2d bytes) 000114 1787907643.722019 mapwrite 0x026c2f thru 0x02b629 (0x49fb bytes) 000115 1787907643.782190 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88ff10db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4eb702b8 Mirror 3: 0x4eb702b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000116 1787907644.979860 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88ff10db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4eb702b8 Mirror 3: 0x4eb702b8 Mirror 4: 0x4eb702b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000119 1787907646.085087 trunc from 0x040000 to 0x01809f (0x27f60 bytes) 000121 1787907646.202598 trunc from 0x0180a0 to 0x02a679 (0x125da bytes) 000124 1787907646.232564 write 0x019957 thru 0x01e057 (0x4701 bytes) 000125 1787907646.245758 mapwrite 0x02d53e thru 0x034228 (0x6ceb bytes) 000126 1787907646.354276 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34229): Mirror 2: 0xa3aefafb (stale) Mirror 4: 0x5f40fbc9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34229): Mirror 2: 0xa3aefafb (stale) Mirror 4: 0x5f40fbc9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34229: skipped 000128 1787907646.884062 write 0x03fa9d thru 0x03ffff (0x563 bytes) 000129 1787907646.894306 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88ff10db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x88ff10db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000130 1787907647.245327 mapwrite 0x01e72e thru 0x02b39f (0xcc72 bytes) 000131 1787907647.321180 write 0x010e78 thru 0x01c16d (0xb2f6 bytes) 000132 1787907647.332931 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) 000135 1787907647.503605 write 0x004068 thru 0x00d525 (0x94be bytes) 000136 1787907647.517516 write 0x01f576 thru 0x02d118 (0xdba3 bytes) 000140 1787907647.535228 write 0x02d0b7 thru 0x032742 (0x568c bytes) 000141 1787907647.541324 write 0x03c567 thru 0x03ffff (0x3a99 bytes) 000142 1787907647.548278 mapwrite 0x00168b thru 0x00d2ac (0xbc22 bytes) 000143 1787907647.669251 read 0x02e1a9 thru 0x03b9f9 (0xd851 bytes) 000147 1787907647.691494 MIRROR_EXTEND to 2 mirrors 000149 1787907648.762658 MIRROR_RESYNC 2 mirrors 000150 1787907649.116084 mapread 0x0008d5 thru 0x00fcfd (0xf429 bytes) 000152 1787907649.168282 read 0x006605 thru 0x00d944 (0x7340 bytes) 000154 1787907649.186046 mapwrite 0x034c26 thru 0x03601c (0x13f7 bytes) 000157 1787907649.477382 read 0x00dce5 thru 0x01d01f (0xf33b bytes) 000160 1787907649.517282 trunc from 0x040000 to 0x00a395 (0x35c6a bytes) 000161 1787907649.593937 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) 000162 1787907649.835267 mapread 0x009505 thru 0x00a395 (0xe91 bytes) 000163 1787907649.858769 read 0x00605c thru 0x008fb2 (0x2f57 bytes) 000164 1787907649.864309 mapread 0x003a40 thru 0x00704d (0x360e bytes) 000165 1787907649.876999 mapread 0x006b0a thru 0x00a395 (0x388c bytes) 000167 1787907649.885358 mapwrite 0x02dbc7 thru 0x02dd13 (0x14d bytes) 000170 1787907650.097688 write 0x0353fa thru 0x0362d4 (0xedb bytes) 000171 1787907650.110447 write 0x02bef4 thru 0x03a88e (0xe99b bytes) 000172 1787907650.112037 mapwrite 0x03f5ac thru 0x03ffff (0xa54 bytes) 000174 1787907650.241365 mapwrite 0x00bf39 thru 0x0199a8 (0xda70 bytes) 000179 1787907650.339342 trunc from 0x040000 to 0x0254ee (0x1ab11 bytes) 000181 1787907650.361483 read 0x004396 thru 0x00ff48 (0xbbb3 bytes) 000185 1787907650.364934 mapwrite 0x007895 thru 0x00c9da (0x5146 bytes) 000186 1787907650.435115 mapread 0x00b78a thru 0x018371 (0xcbe8 bytes) 000187 1787907650.436543 mapread 0x0052e1 thru 0x00b0d1 (0x5df1 bytes) 000188 1787907650.437759 mapwrite 0x03f372 thru 0x03ffff (0xc8e bytes) 000189 1787907650.564913 trunc from 0x040000 to 0x026084 (0x19f7b bytes) 000190 1787907650.609206 mapread 0x00c90d thru 0x012963 (0x6057 bytes) 000194 1787907650.620281 write 0x00452e thru 0x00672f (0x2202 bytes) 000195 1787907650.624619 read 0x011536 thru 0x01e457 (0xcf22 bytes) 000197 1787907650.627366 mapwrite 0x0366a1 thru 0x03e1f9 (0x7b59 bytes) 000199 1787907650.715011 mapwrite 0x03f79f thru 0x03ffff (0x861 bytes) 000200 1787907650.792600 mapread 0x02e2ac thru 0x02eb9f (0x8f4 bytes) 000201 1787907650.809600 trunc from 0x040000 to 0x02c0a3 (0x13f5c bytes) 000204 1787907650.841905 read 0x002e9f thru 0x010961 (0xdac3 bytes) 000206 1787907650.852127 read 0x01ee85 thru 0x02c0a3 (0xd21f bytes) 000208 1787907650.853184 write 0x0006f5 thru 0x00c9f0 (0xc2fc bytes) 000209 1787907650.856214 mapwrite 0x02ca96 thru 0x03b4c5 (0xea30 bytes) 000210 1787907650.996081 trunc from 0x03b4c6 to 0x03b784 (0x2bf bytes) 000211 1787907651.027034 mapread 0x014ed0 thru 0x01d322 (0x8453 bytes) 000213 1787907651.029930 trunc from 0x03b785 to 0x0308c5 (0xaebf bytes) 000216 1787907651.051136 trunc from 0x0308c6 to 0x01b239 (0x1568c bytes) 000218 1787907651.076044 trunc from 0x01b23a to 0x027ced (0xcab4 bytes) 000219 1787907651.100278 write 0x01ed57 thru 0x021ac3 (0x2d6d bytes) 000220 1787907651.114066 write 0x017ace thru 0x022649 (0xab7c bytes) 000222 1787907651.124354 mapread 0x0026ee thru 0x00851a (0x5e2d bytes) 000229 1787907651.135671 mapread 0x01c8ea thru 0x0244a4 (0x7bbb bytes) 000231 1787907651.153245 MIRROR_EXTEND to 2 mirrors 000233 1787907652.010535 write 0x01aca7 thru 0x026366 (0xb6c0 bytes) 000235 1787907652.129521 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27cee): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27cee): Mirror 2: 0x2249db4b (stale) CRC-32 checksum value for chunk [0, 0x27cee): Mirror 1: 0x7e60cdb8 Mirror 3: 0x7e60cdb8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27cee: skipped 000236 1787907653.359269 write 0x00b8ff thru 0x00d436 (0x1b38 bytes) 000237 1787907653.573910 mapwrite 0x00191a thru 0x009fe8 (0x86cf bytes) 000238 1787907653.673107 mapwrite 0x032b65 thru 0x03bc7f (0x911b bytes) 000239 1787907653.756897 mapwrite 0x02a056 thru 0x030d2a (0x6cd5 bytes) 000241 1787907653.806429 mapwrite 0x001dbf thru 0x00bbf9 (0x9e3b bytes) 000242 1787907653.901190 read 0x02d169 thru 0x034fb9 (0x7e51 bytes) 000243 1787907653.901814 mapread 0x003240 thru 0x00c75d (0x951e bytes) 000244 1787907653.907279 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bc80): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc80): Mirror 2: 0xde277eb (stale) Mirror 3: 0xf0a267db (stale) CRC-32 checksum value for chunk [0, 0x3bc80): Mirror 1: 0x6cf3e41 Mirror 4: 0x6cf3e41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc80: skipped 000246 1787907655.233917 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bc80): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc80): Mirror 2: 0xde277eb (stale) Mirror 3: 0xf0a267db (stale) CRC-32 checksum value for chunk [0, 0x3bc80): Mirror 1: 0x6cf3e41 Mirror 4: 0x6cf3e41 Mirror 5: 0x6cf3e41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc80: skipped 000247 1787907656.148529 trunc from 0x03bc80 to 0x018c2f (0x23050 bytes) 000248 1787907656.253285 trunc from 0x018c30 to 0x00cb83 (0xc0ac bytes) 000251 1787907656.275727 trunc from 0x00cb84 to 0x02731e (0x1a79b bytes) 000254 1787907656.305290 MIRROR_RESYNC 5 mirrors 000255 1787907657.673541 trunc from 0x02731f to 0x026af6 (0x828 bytes) 000257 1787907657.848321 write 0x03be49 thru 0x03e2a0 (0x2458 bytes) 000260 1787907657.857063 write 0x00ebbd thru 0x01aec0 (0xc304 bytes) 000261 1787907657.876156 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e2a1): Mirror 3: 0x93584e86 (stale) Mirror 4: 0x93584e86 (stale) Mirror 5: 0x93584e86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2a1): Mirror 3: 0x93584e86 (stale) Mirror 4: 0x93584e86 (stale) Mirror 5: 0x93584e86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2a1: skipped 000264 1787907658.457597 read 0x018e17 thru 0x01cddb (0x3fc5 bytes) 000265 1787907658.473911 trunc from 0x03e2a1 to 0x03b937 (0x2969 bytes) 000267 1787907658.529286 trunc from 0x03b938 to 0x021b39 (0x19dfe bytes) 000268 1787907658.559609 mapwrite 0x037fa6 thru 0x03ffff (0x805a bytes) 000270 1787907658.709350 read 0x00925e thru 0x016320 (0xd0c3 bytes) 000272 1787907658.722713 trunc from 0x040000 to 0x0298af (0x16750 bytes) 000273 1787907658.739512 mapwrite 0x034f62 thru 0x038bc9 (0x3c68 bytes) 000274 1787907658.885442 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38bca): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bca): Mirror 3: 0x5856bc14 (stale) Mirror 4: 0x5856bc14 (stale) Mirror 5: 0x5856bc14 (stale) CRC-32 checksum value for chunk [0, 0x38bca): Mirror 1: 0x7ac11639 Mirror 6: 0x7ac11639 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bca: skipped 000275 1787907660.561335 MIRROR_RESYNC 5 mirrors 000276 1787907662.986417 trunc from 0x038bca to 0x03b07c (0x24b3 bytes) 000279 1787907663.080357 mapread 0x01d3db thru 0x0218a0 (0x44c6 bytes) 000280 1787907663.115135 mapread 0x01d594 thru 0x020534 (0x2fa1 bytes) 000281 1787907663.115958 read 0x02c4e5 thru 0x02fbf5 (0x3711 bytes) 000282 1787907663.144480 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b07d): Mirror 3: 0x282b362e (stale) Mirror 4: 0x282b362e (stale) Mirror 5: 0x282b362e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b07d): Mirror 3: 0x282b362e (stale) Mirror 4: 0x282b362e (stale) Mirror 5: 0x282b362e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b07d: skipped 000284 1787907663.843663 trunc from 0x03b07d to 0x01ca69 (0x1e613 bytes) 000287 1787907663.875737 read 0x01679a thru 0x01702c (0x893 bytes) 000288 1787907663.882121 write 0x022d74 thru 0x023c28 (0xeb5 bytes) 000289 1787907663.884393 trunc from 0x023c29 to 0x020243 (0x39e5 bytes) 000290 1787907663.904960 mapread 0x017b6e thru 0x01e267 (0x66fa bytes) 000291 1787907663.920647 mapread 0x00de3e thru 0x01af63 (0xd126 bytes) 000292 1787907663.929035 trunc from 0x020244 to 0x0099cb (0x16878 bytes) 000293 1787907663.947023 mapwrite 0x036180 thru 0x03cf62 (0x6de3 bytes) 000295 1787907664.042408 mapwrite 0x01d4e7 thru 0x0260fb (0x8c15 bytes) 000296 1787907664.140187 mapwrite 0x026376 thru 0x03138f (0xb01a bytes) 000297 1787907664.197002 write 0x02a861 thru 0x0328ab (0x804b bytes) 000298 1787907664.199217 write 0x00b3a7 thru 0x01548c (0xa0e6 bytes) 000299 1787907664.216127 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf63): Mirror 4: 0x37ade214 (stale) Mirror 5: 0x37ade214 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf63): Mirror 4: 0x37ade214 (stale) Mirror 5: 0x37ade214 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf63: skipped 000300 1787907664.588441 mapread 0x011ecc thru 0x014b9d (0x2cd2 bytes) 000301 1787907664.619380 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf63): Mirror 4: 0x37ade214 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf63): Mirror 4: 0x37ade214 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf63: skipped 000302 1787907665.195219 write 0x038dfd thru 0x03a278 (0x147c bytes) 000306 1787907665.225166 mapread 0x029943 thru 0x0338e6 (0x9fa4 bytes) 000307 1787907665.262375 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cf63): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf63): Mirror 4: 0x37ade214 (stale) CRC-32 checksum value for chunk [0, 0x3cf63): Mirror 1: 0xe4c7026b Mirror 5: 0xe4c7026b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf63: skipped 000308 1787907667.218234 write 0x036075 thru 0x036828 (0x7b4 bytes) 000309 1787907667.293470 mapwrite 0x01bc52 thru 0x02472b (0x8ada bytes) 000311 1787907667.351508 trunc from 0x03cf63 to 0x000e01 (0x3c161 bytes) 000312 1787907667.386209 MIRROR_RESYNC 3 mirrors 000314 1787907668.799744 MIRROR_EXTEND to 4 mirrors 000315 1787907669.610079 read 0x000378 thru 0x000e01 (0xa8a bytes) 000316 1787907669.616176 trunc from 0x000e02 to 0x03aaa6 (0x39ca5 bytes) 000319 1787907669.707351 write 0x0101ec thru 0x011e2c (0x1c41 bytes) 000320 1787907669.717497 read 0x017e16 thru 0x026568 (0xe753 bytes) 000321 1787907669.735157 trunc from 0x03aaa7 to 0x03690b (0x419b bytes) 000325 1787907669.769395 read 0x001c0c thru 0x00a90b (0x8d00 bytes) 000326 1787907669.774955 MIRROR_RESYNC 4 mirrors 000327 1787907671.106832 mapread 0x0054a3 thru 0x00d542 (0x80a0 bytes) 000328 1787907671.136205 mapread 0x01c25e thru 0x026ea0 (0xac43 bytes) 000329 1787907671.157603 trunc from 0x03690c to 0x029a5e (0xcead bytes) 000330 1787907671.327658 write 0x0129bf thru 0x021a73 (0xf0b5 bytes) 000332 1787907671.356339 read 0x020cc5 thru 0x029979 (0x8cb5 bytes) 000333 1787907671.374199 mapwrite 0x00fa1d thru 0x0164f7 (0x6adb bytes) 000334 1787907671.492172 mapwrite 0x01a7b2 thru 0x0208bd (0x610c bytes) 000336 1787907671.567465 write 0x023466 thru 0x02b059 (0x7bf4 bytes) 000337 1787907671.570508 trunc from 0x02b05a to 0x0285e8 (0x2a71 bytes) 000338 1787907671.604438 trunc from 0x0285e9 to 0x0277ee (0xdfa bytes) 000339 1787907671.634273 MIRROR_RESYNC 4 mirrors 000341 1787907673.646420 read 0x005ed1 thru 0x0131e8 (0xd318 bytes) 000343 1787907673.689934 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) 000345 1787907674.186716 mapwrite 0x013754 thru 0x01b13d (0x79ea bytes) 000346 1787907674.417226 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x277ef): Mirror 6: 0xf82efe3e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x277ef): Mirror 6: 0xf82efe3e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x277ef: skipped 000348 1787907674.714362 trunc from 0x0277ef to 0x003f45 (0x238a9 bytes) 000349 1787907674.752759 mapwrite 0x0384f5 thru 0x03ffff (0x7b0b bytes) 000351 1787907674.871979 write 0x001224 thru 0x00dff4 (0xcdd1 bytes) 000356 1787907674.887058 mapwrite 0x033242 thru 0x036de7 (0x3ba6 bytes) 000357 1787907674.964953 MIRROR_RESYNC 2 mirrors 000360 1787907676.113508 write 0x01d55f thru 0x0258ab (0x834d bytes) 000362 1787907676.188311 MIRROR_RESYNC 2 mirrors 000363 1787907677.387711 mapwrite 0x00dd19 thru 0x017bad (0x9e95 bytes) 000364 1787907677.624473 write 0x00d8a5 thru 0x01c518 (0xec74 bytes) 000365 1787907677.626205 read 0x02f6a1 thru 0x0338b4 (0x4214 bytes) 000367 1787907677.642360 read 0x0278f8 thru 0x02dc6f (0x6378 bytes) 000368 1787907677.658560 mapwrite 0x03fea6 thru 0x03ffff (0x15a bytes) 000369 1787907677.720031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) 000370 1787907677.887958 read 0x029d7b thru 0x0350c6 (0xb34c bytes) 000371 1787907677.913113 MIRROR_EXTEND to 2 mirrors 000373 1787907679.175548 write 0x0200f0 thru 0x023ee2 (0x3df3 bytes) 000374 1787907679.268217 mapwrite 0x01797b thru 0x01f242 (0x78c8 bytes) 000376 1787907679.346712 mapread 0x03fa9a thru 0x03ffff (0x566 bytes) 000378 1787907679.363686 mapread 0x00c472 thru 0x015b69 (0x96f8 bytes) 000379 1787907679.383889 mapread 0x00f219 thru 0x01c62b (0xd413 bytes) 000381 1787907679.387093 trunc from 0x040000 to 0x033021 (0xcfde bytes) 000382 1787907679.412868 read 0x028a62 thru 0x033021 (0xa5c0 bytes) 000383 1787907679.422905 mapwrite 0x01ff4d thru 0x02b9ec (0xbaa0 bytes) 000385 1787907679.530467 mapwrite 0x014efb thru 0x018f60 (0x4066 bytes) 000386 1787907679.654218 trunc from 0x033022 to 0x03814f (0x512e bytes) 000389 1787907679.699284 mapread 0x019846 thru 0x01aeae (0x1669 bytes) 000390 1787907679.714286 MIRROR_RESYNC 2 mirrors 000391 1787907680.728817 trunc from 0x038150 to 0x00b408 (0x2cd47 bytes) 000393 1787907680.880966 MIRROR_RESYNC 2 mirrors 000395 1787907681.631776 mapwrite 0x02ede5 thru 0x0366bd (0x78d9 bytes) 000396 1787907681.845713 trunc from 0x0366be to 0x019a08 (0x1ccb5 bytes) 000398 1787907681.877111 mapwrite 0x014b82 thru 0x019c0d (0x508c bytes) 000402 1787907681.995908 read 0x00442d thru 0x00b36a (0x6f3e bytes) 000404 1787907682.006989 mapread 0x00ba43 thru 0x00fc44 (0x4202 bytes) 000407 1787907682.030367 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) 000409 1787907682.225205 mapwrite 0x012bf2 thru 0x012c36 (0x45 bytes) 000410 1787907682.325688 write 0x01e55f thru 0x028d7d (0xa81f bytes) 000411 1787907682.330084 mapwrite 0x035422 thru 0x03edc8 (0x99a7 bytes) 000413 1787907682.509110 write 0x01f5ba thru 0x02848c (0x8ed3 bytes) 000416 1787907682.515537 mapread 0x031a6d thru 0x031aa0 (0x34 bytes) 000417 1787907682.517955 mapwrite 0x008dde thru 0x00fec3 (0x70e6 bytes) 000418 1787907682.697005 trunc from 0x03edc9 to 0x0041f8 (0x3abd0 bytes) 000419 1787907682.755548 trunc from 0x0041f9 to 0x03166b (0x2d473 bytes) 000421 1787907682.810031 write 0x00f7b2 thru 0x013f3e (0x478d bytes) 000424 1787907682.823107 trunc from 0x03166c to 0x01ae94 (0x167d7 bytes) 000426 1787907682.865729 write 0x023c5d thru 0x03348d (0xf831 bytes) 000427 1787907682.870152 trunc from 0x03348e to 0x03b8a1 (0x8414 bytes) 000428 1787907682.902473 mapread 0x006c9e thru 0x0115d8 (0xa93b bytes) 000430 1787907682.919624 MIRROR_EXTEND to 2 mirrors 000431 1787907684.053402 mapread 0x0195f4 thru 0x01ead5 (0x54e2 bytes) 000434 1787907684.164531 trunc from 0x03b8a2 to 0x037f9a (0x3907 bytes) 000436 1787907684.284969 mapwrite 0x00b856 thru 0x01260c (0x6db7 bytes) 000439 1787907684.348225 trunc from 0x037f9b to 0x01142e (0x26b6c bytes) 000440 1787907684.367304 trunc from 0x01142f to 0x0273a9 (0x15f7b bytes) 000441 1787907684.396584 mapwrite 0x03fe7e thru 0x03ffff (0x182 bytes) 000442 1787907684.521202 write 0x015b43 thru 0x01c402 (0x68c0 bytes) 000443 1787907684.533768 write 0x01ae20 thru 0x01f6d6 (0x48b7 bytes) 000444 1787907684.536236 mapwrite 0x01efed thru 0x029bbe (0xabd2 bytes) 000445 1787907684.738284 read 0x0075c0 thru 0x00ce7e (0x58bf bytes) 000446 1787907684.738906 read 0x02d807 thru 0x03ab49 (0xd343 bytes) 000447 1787907684.790566 MIRROR_RESYNC 2 mirrors 000448 1787907686.591379 read 0x012a84 thru 0x019ed2 (0x744f bytes) 000449 1787907686.600889 mapwrite 0x01a54b thru 0x02a389 (0xfe3f bytes) 000457 1787907686.979660 write 0x02b461 thru 0x0326c6 (0x7266 bytes) 000458 1787907686.996053 read 0x027c5f thru 0x036574 (0xe916 bytes) 000461 1787907687.026404 trunc from 0x040000 to 0x0210c0 (0x1ef3f bytes) 000462 1787907687.107384 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x210c1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210c1): Mirror 2: 0x6f64dfef (stale) CRC-32 checksum value for chunk [0, 0x210c1): Mirror 1: 0x64c347d4 Mirror 3: 0x64c347d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210c1: skipped 000463 1787907688.889352 mapwrite 0x01f249 thru 0x020ff2 (0x1daa bytes) 000464 1787907689.157317 mapread 0x01c2a2 thru 0x0210c0 (0x4e1f bytes) 000466 1787907689.172068 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x210c1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210c1): Mirror 2: 0x6f64dfef (stale) Mirror 3: 0x64c347d4 (stale) CRC-32 checksum value for chunk [0, 0x210c1): Mirror 1: 0x71a9559 Mirror 4: 0x71a9559 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210c1: skipped 000467 1787907690.935760 mapread 0x0184ec thru 0x0210c0 (0x8bd5 bytes) 000468 1787907691.004495 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x210c1): Mirror 3: 0x64c347d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210c1): Mirror 3: 0x64c347d4 (stale) CRC-32 checksum value for chunk [0, 0x210c1): Mirror 1: 0x71a9559 Mirror 4: 0x71a9559 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210c1: skipped 000470 1787907692.059881 mapread 0x0160e3 thru 0x0199bb (0x38d9 bytes) 000471 1787907692.128496 mapread 0x011049 thru 0x01dd70 (0xcd28 bytes) 000473 1787907692.201294 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x210c1): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x210c1): Mirror 3: 0x64c347d4 (stale) CRC-32 checksum value for chunk [0, 0x210c1): Mirror 1: 0x71a9559 Mirror 4: 0x71a9559 Mirror 5: 0x71a9559 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x210c1: skipped 000474 1787907693.559266 mapwrite 0x03eb40 thru 0x03ffff (0x14c0 bytes) 000476 1787907693.708238 write 0x009a4a thru 0x00f0f0 (0x56a7 bytes) 000477 1787907693.722678 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa4d732ac (stale) Mirror 4: 0x3b49c3d6 (stale) Mirror 5: 0x3b49c3d6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb2b2d996 Mirror 6: 0xb2b2d996 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000479 1787907695.510776 mapread 0x0271cf thru 0x0352af (0xe0e1 bytes) 000480 1787907695.587195 read 0x0093dd thru 0x01173d (0x8361 bytes) 000481 1787907695.604682 mapread 0x034a36 thru 0x03ffff (0xb5ca bytes) 000482 1787907695.641963 write 0x01b84e thru 0x023475 (0x7c28 bytes) 000483 1787907695.728247 mapread 0x01d2ca thru 0x020f2c (0x3c63 bytes) 000487 1787907695.736420 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa4d732ac (stale) Mirror 5: 0x3b49c3d6 (stale) Mirror 6: 0xb2b2d996 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa4d732ac (stale) Mirror 5: 0x3b49c3d6 (stale) Mirror 6: 0xb2b2d996 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000488 1787907696.179928 trunc from 0x040000 to 0x005da0 (0x3a25f bytes) 000489 1787907696.201004 mapread 0x0004c1 thru 0x005da0 (0x58e0 bytes) 000490 1787907696.232699 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5da1): Mirror 3: 0x83016c65 (stale) Mirror 6: 0x83016c65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5da1): Mirror 3: 0x83016c65 (stale) Mirror 6: 0x83016c65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5da1: skipped 000491 1787907696.882654 read 0x002449 thru 0x005da0 (0x3958 bytes) 000493 1787907696.890461 write 0x021d99 thru 0x0306cc (0xe934 bytes) 000495 1787907696.899220 write 0x034b5a thru 0x03ffff (0xb4a6 bytes) 000496 1787907696.906983 write 0x022527 thru 0x0293bd (0x6e97 bytes) 000497 1787907696.909886 write 0x013482 thru 0x01b788 (0x8307 bytes) 000498 1787907696.932892 trunc from 0x040000 to 0x00d8a0 (0x3275f bytes) 000500 1787907696.955430 write 0x01b59f thru 0x024c72 (0x96d4 bytes) 000502 1787907696.965738 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24c73): Mirror 3: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c73): Mirror 3: 0x7d77652 (stale) Mirror 6: 0x65d7a101 (stale) CRC-32 checksum value for chunk [0, 0x24c73): Mirror 1: 0x4d8c7d84 Mirror 7: 0x4d8c7d84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c73: skipped 000503 1787907698.563120 mapwrite 0x036c0f thru 0x0392b5 (0x26a7 bytes) 000504 1787907699.001311 read 0x00611b thru 0x00b7ca (0x56b0 bytes) 000506 1787907699.031033 mapwrite 0x00b712 thru 0x016bdb (0xb4ca bytes) 000508 1787907699.189263 mapwrite 0x03d4fc thru 0x03e656 (0x115b bytes) 000510 1787907699.371845 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e657): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e657): Mirror 3: 0x52ac4c6f (stale) Mirror 6: 0xe74e7b19 (stale) Mirror 7: 0xbbb12da9 (stale) CRC-32 checksum value for chunk [0, 0x3e657): Mirror 1: 0xe83ce281 Mirror 8: 0xe83ce281 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e657: skipped 000511 1787907701.319166 mapwrite 0x03d666 thru 0x03ffff (0x299a bytes) 000512 1787907701.581016 mapwrite 0x01aebd thru 0x024cd3 (0x9e17 bytes) 000513 1787907701.702211 mapwrite 0x001d64 thru 0x00fdfb (0xe098 bytes) 000515 1787907701.858490 write 0x005fb9 thru 0x008242 (0x228a bytes) 000517 1787907701.861938 mapwrite 0x013f13 thru 0x016e92 (0x2f80 bytes) 000518 1787907701.989312 write 0x03038e thru 0x0391a0 (0x8e13 bytes) 000519 1787907702.005734 trunc from 0x040000 to 0x034ffa (0xb005 bytes) 000520 1787907702.029545 write 0x037104 thru 0x03ffff (0x8efc bytes) 000521 1787907702.031860 read 0x017775 thru 0x017d8f (0x61b bytes) 000525 1787907702.034100 write 0x01253f thru 0x01bdd3 (0x9895 bytes) 000526 1787907702.040065 read 0x015371 thru 0x01f0ae (0x9d3e bytes) 000527 1787907702.041246 mapwrite 0x01d203 thru 0x028c3b (0xba39 bytes) 000528 1787907702.128669 mapwrite 0x00b2f2 thru 0x012c28 (0x7937 bytes) 000532 1787907702.289937 write 0x033ce6 thru 0x03a7a2 (0x6abd bytes) 000533 1787907702.295429 read 0x00518b thru 0x007d96 (0x2c0c bytes) 000534 1787907702.296013 write 0x039aff thru 0x03ffff (0x6501 bytes) 000535 1787907702.303780 write 0x03f362 thru 0x03ffff (0xc9e bytes) 000541 1787907702.315897 trunc from 0x040000 to 0x03cc01 (0x33fe bytes) 000542 1787907702.355306 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc02): Mirror 3: 0xfc071867 (stale) Mirror 7: 0xe5becabd (stale) Mirror 8: 0xecc83365 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc02): Mirror 3: 0xfc071867 (stale) Mirror 7: 0xe5becabd (stale) Mirror 8: 0xecc83365 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc02: skipped 000543 1787907703.085877 read 0x0000d6 thru 0x00db1d (0xda48 bytes) 000545 1787907703.123744 mapread 0x01bab1 thru 0x02b9da (0xff2a bytes) 000546 1787907703.174207 trunc from 0x03cc02 to 0x006f0f (0x35cf2 bytes) 000547 1787907703.204163 mapwrite 0x0129bf thru 0x017d3a (0x537c bytes) 000548 1787907703.429517 read 0x0122d4 thru 0x017d3a (0x5a67 bytes) 000549 1787907703.433247 mapwrite 0x0023b0 thru 0x00694d (0x459e bytes) 000551 1787907703.560883 write 0x01b4e9 thru 0x01bbaa (0x6c2 bytes) 000553 1787907703.581021 MIRROR_RESYNC 4 mirrors 000556 1787907705.862819 mapwrite 0x021b1c thru 0x024569 (0x2a4e bytes) 000557 1787907706.176625 read 0x001d4b thru 0x00ded0 (0xc186 bytes) 000560 1787907706.207335 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2456a): Mirror 3: 0xe653ad99 (stale) Mirror 7: 0xe653ad99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2456a): Mirror 3: 0xe653ad99 (stale) Mirror 7: 0xe653ad99 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2456a: skipped 000562 1787907706.744269 write 0x03a3e3 thru 0x03ce80 (0x2a9e bytes) 000567 1787907706.754631 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce81): Mirror 7: 0x916e70cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce81): Mirror 7: 0x916e70cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce81: skipped 000569 1787907707.127151 MIRROR_RESYNC 2 mirrors 000570 1787907708.786716 write 0x01c606 thru 0x029e55 (0xd850 bytes) 000572 1787907708.879862 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) 000573 1787907709.052850 trunc from 0x03ce81 to 0x0303c5 (0xcabb bytes) 000579 1787907709.078605 mapread 0x009eb7 thru 0x00f30c (0x5456 bytes) 000585 1787907709.129875 MIRROR_EXTEND to 2 mirrors 000587 1787907710.897006 MIRROR_EXTEND to 3 mirrors 000588 1787907711.820498 trunc from 0x0303c6 to 0x029d42 (0x6683 bytes) 000590 1787907711.955199 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29d43): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d43): Mirror 2: 0x3b9c2002 (stale) Mirror 3: 0x3b9c2002 (stale) CRC-32 checksum value for chunk [0, 0x29d43): Mirror 1: 0x3b9c2002 Mirror 4: 0x3b9c2002 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d43: skipped 000591 1787907713.481953 mapread 0x00cf42 thru 0x0141f1 (0x72b0 bytes) 000592 1787907713.518932 trunc from 0x029d43 to 0x017211 (0x12b31 bytes) 000593 1787907713.638092 read 0x011e27 thru 0x017211 (0x53eb bytes) 000594 1787907713.646815 read 0x014351 thru 0x017211 (0x2ec1 bytes) 000596 1787907713.647982 trunc from 0x017212 to 0x02f07f (0x17e6e bytes) 000597 1787907713.666051 trunc from 0x02f080 to 0x00e9d7 (0x206a8 bytes) 000598 1787907713.687253 mapread 0x00cc8f thru 0x00e9d7 (0x1d49 bytes) 000599 1787907713.700461 mapwrite 0x0286af thru 0x02c9be (0x4310 bytes) 000602 1787907713.817866 mapread 0x00bca2 thru 0x017589 (0xb8e8 bytes) 000604 1787907713.822965 mapwrite 0x00df29 thru 0x015214 (0x72ec bytes) 000605 1787907713.893164 write 0x03c48a thru 0x03ffff (0x3b76 bytes) 000606 1787907713.895248 write 0x011ed4 thru 0x01af48 (0x9075 bytes) 000607 1787907713.897609 write 0x0362a9 thru 0x037593 (0x12eb bytes) 000608 1787907713.906330 mapread 0x029d03 thru 0x0397d1 (0xfacf bytes) 000609 1787907713.907363 read 0x022d04 thru 0x02f264 (0xc561 bytes) 000613 1787907713.910980 read 0x03ea40 thru 0x03ffff (0x15c0 bytes) 000615 1787907713.912568 write 0x039ab8 thru 0x03ffff (0x6548 bytes) 000616 1787907713.914470 read 0x022251 thru 0x0275ec (0x539c bytes) 000617 1787907713.915217 read 0x00d634 thru 0x00d7f6 (0x1c3 bytes) 000619 1787907713.932728 MIRROR_RESYNC 4 mirrors 000620 1787907717.035096 MIRROR_EXTEND to 5 mirrors 000622 1787907718.549393 read 0x0192ea thru 0x023845 (0xa55c bytes) 000623 1787907718.569111 write 0x00298c thru 0x00852c (0x5ba1 bytes) 000626 1787907718.749074 mapwrite 0x03d352 thru 0x03ffff (0x2cae bytes) 000628 1787907719.019824 mapwrite 0x00c54f thru 0x0110b5 (0x4b67 bytes) 000631 1787907719.140685 mapread 0x0043a8 thru 0x0073cd (0x3026 bytes) 000632 1787907719.144937 read 0x01eda1 thru 0x02dd01 (0xef61 bytes) 000633 1787907719.179536 mapread 0x01980a thru 0x01b3ee (0x1be5 bytes) 000634 1787907719.183659 mapread 0x012c38 thru 0x01f944 (0xcd0d bytes) 000635 1787907719.185944 trunc from 0x040000 to 0x021528 (0x1ead7 bytes) 000640 1787907719.271659 write 0x030b0b thru 0x036e06 (0x62fc bytes) 000642 1787907719.276562 trunc from 0x036e07 to 0x01917f (0x1dc87 bytes) 000645 1787907719.332318 trunc from 0x019180 to 0x039628 (0x204a9 bytes) 000646 1787907719.384466 mapread 0x03451e thru 0x039628 (0x510b bytes) 000647 1787907719.434538 read 0x006b32 thru 0x00cd9f (0x626e bytes) 000648 1787907719.435450 read 0x027ffc thru 0x02ad74 (0x2d79 bytes) 000650 1787907719.460418 mapread 0x019f42 thru 0x023de6 (0x9ea5 bytes) 000653 1787907719.472009 read 0x018411 thru 0x018dc3 (0x9b3 bytes) 000654 1787907719.478214 write 0x011f9f thru 0x01cbe5 (0xac47 bytes) 000655 1787907719.485053 write 0x03323c thru 0x033725 (0x4ea bytes) 000656 1787907719.487503 read 0x016334 thru 0x01fa64 (0x9731 bytes) 000658 1787907719.493593 write 0x02baa1 thru 0x034aea (0x904a bytes) 000659 1787907719.495665 trunc from 0x039629 to 0x010a9b (0x28b8d bytes) 000660 1787907719.536087 trunc from 0x010a9c to 0x02fad3 (0x1f038 bytes) 000661 1787907719.581955 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2fad4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fad4): Mirror 2: 0xf17ba90c (stale) Mirror 3: 0xf17ba90c (stale) Mirror 4: 0xf17ba90c (stale) Mirror 5: 0xf17ba90c (stale) CRC-32 checksum value for chunk [0, 0x2fad4): Mirror 1: 0x6bada3ba Mirror 6: 0x6bada3ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fad4: skipped 000663 1787907721.620407 MIRROR_RESYNC 6 mirrors 000665 1787907724.346111 mapread 0x01cfa3 thru 0x01f972 (0x29d0 bytes) 000666 1787907724.402414 mapread 0x019ff1 thru 0x027efd (0xdf0d bytes) 000668 1787907724.428621 trunc from 0x02fad4 to 0x01b495 (0x1463e bytes) 000669 1787907724.592244 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b496): Mirror 3: 0x9dbb03e7 (stale) Mirror 4: 0x9dbb03e7 (stale) Mirror 5: 0x9dbb03e7 (stale) Mirror 6: 0x9dbb03e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b496): Mirror 3: 0x9dbb03e7 (stale) Mirror 4: 0x9dbb03e7 (stale) Mirror 5: 0x9dbb03e7 (stale) Mirror 6: 0x9dbb03e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b496: skipped 000670 1787907725.285845 read 0x004adf thru 0x007913 (0x2e35 bytes) truncating to largest ever: 0x3e42f 000672 1787907725.311816 trunc from 0x01b496 to 0x03e42e (0x22f99 bytes) 000673 1787907725.338529 mapread 0x01e00b thru 0x027d32 (0x9d28 bytes) 000674 1787907725.371832 mapread 0x02c1f7 thru 0x03b749 (0xf553 bytes) 000675 1787907725.404825 mapwrite 0x020f8e thru 0x027abb (0x6b2e bytes) 000676 1787907725.489518 trunc from 0x03e42f to 0x03da73 (0x9bb bytes) 000677 1787907725.522167 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3da74): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da74): Mirror 3: 0xe63730e5 (stale) Mirror 4: 0xe63730e5 (stale) Mirror 5: 0xe63730e5 (stale) Mirror 6: 0xe63730e5 (stale) CRC-32 checksum value for chunk [0, 0x3da74): Mirror 1: 0x18a3085 Mirror 7: 0x18a3085 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da74: skipped 000679 1787907727.630369 trunc from 0x03da74 to 0x025dd4 (0x17c9f bytes) 000683 1787907727.797712 trunc from 0x025dd5 to 0x00c715 (0x196bf bytes) 000684 1787907727.851702 read 0x00a386 thru 0x00c715 (0x2390 bytes) 000685 1787907727.870441 write 0x0101f6 thru 0x01137a (0x1185 bytes) 000686 1787907727.873263 write 0x020bc2 thru 0x025aa5 (0x4ee4 bytes) 000687 1787907727.878502 write 0x005c05 thru 0x00a46c (0x4868 bytes) 000688 1787907727.887608 read 0x003801 thru 0x006989 (0x3189 bytes) 000689 1787907727.901803 mapread 0x010e6e thru 0x01763e (0x67d1 bytes) 000690 1787907727.905371 trunc from 0x025aa6 to 0x00ddec (0x17cb9 bytes) 000692 1787907727.938550 write 0x021a1a thru 0x029ea8 (0x848f bytes) 000694 1787907727.947934 mapwrite 0x00e888 thru 0x01ddf0 (0xf569 bytes) 000696 1787907728.400955 mapread 0x01b700 thru 0x029ea8 (0xe7a9 bytes) 000697 1787907728.470663 write 0x016d6a thru 0x026734 (0xf9cb bytes) 000698 1787907728.477685 trunc from 0x029ea9 to 0x03070c (0x6864 bytes) 000699 1787907728.547704 read 0x00dd37 thru 0x0165e7 (0x88b1 bytes) 000700 1787907728.558876 read 0x0160a6 thru 0x01dc1b (0x7b76 bytes) 000701 1787907728.567115 read 0x00928e thru 0x00dcc0 (0x4a33 bytes) 000702 1787907728.574356 read 0x0152f7 thru 0x01eead (0x9bb7 bytes) 000703 1787907728.588105 MIRROR_RESYNC 6 mirrors 000704 1787907732.554891 mapread 0x003649 thru 0x010e22 (0xd7da bytes) 000706 1787907732.586617 mapwrite 0x021a41 thru 0x02cd37 (0xb2f7 bytes) 000707 1787907732.842762 trunc from 0x03070d to 0x03a55d (0x9e51 bytes) 000708 1787907732.922944 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a55e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a55e): Mirror 3: 0x695924d6 (stale) Mirror 4: 0x695924d6 (stale) Mirror 5: 0x695924d6 (stale) Mirror 6: 0x695924d6 (stale) Mirror 7: 0x695924d6 (stale) CRC-32 checksum value for chunk [0, 0x3a55e): Mirror 1: 0xc40a67b2 Mirror 8: 0xc40a67b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a55e: skipped 000709 1787907735.093898 mapread 0x02f9a1 thru 0x03856c (0x8bcc bytes) 000710 1787907735.164415 mapread 0x02b3d7 thru 0x03983e (0xe468 bytes) 000711 1787907735.178541 mapwrite 0x03cdf8 thru 0x03ffff (0x3208 bytes) 000712 1787907735.591906 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d48e5a9 (stale) Mirror 4: 0x8d48e5a9 (stale) Mirror 6: 0x8d48e5a9 (stale) Mirror 7: 0x8d48e5a9 (stale) Mirror 8: 0x2739873f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d48e5a9 (stale) Mirror 4: 0x8d48e5a9 (stale) Mirror 6: 0x8d48e5a9 (stale) Mirror 7: 0x8d48e5a9 (stale) Mirror 8: 0x2739873f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000715 1787907736.343368 write 0x02c9d6 thru 0x03911e (0xc749 bytes) 000716 1787907736.366233 mapread 0x03e968 thru 0x03ffff (0x1698 bytes) 000717 1787907736.405572 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8d48e5a9 (stale) Mirror 6: 0x8d48e5a9 (stale) Mirror 7: 0x8d48e5a9 (stale) Mirror 8: 0x2739873f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8d48e5a9 (stale) Mirror 6: 0x8d48e5a9 (stale) Mirror 7: 0x8d48e5a9 (stale) Mirror 8: 0x2739873f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000718 1787907737.266640 read 0x0358ea thru 0x03ffff (0xa716 bytes) 000719 1787907737.280938 MIRROR_RESYNC 5 mirrors 000721 1787907740.437121 read 0x021422 thru 0x02480f (0x33ee bytes) 000722 1787907740.447396 read 0x02d39b thru 0x02f2df (0x1f45 bytes) 000723 1787907740.461722 mapread 0x01e5c3 thru 0x026b9f (0x85dd bytes) 000724 1787907740.462679 read 0x0262fe thru 0x02bbbc (0x58bf bytes) 000725 1787907740.467848 MIRROR_RESYNC 5 mirrors 000726 1787907741.331740 read 0x0076a6 thru 0x010eb1 (0x980c bytes) 000727 1787907741.336951 MIRROR_RESYNC 5 mirrors 000730 1787907742.006860 mapwrite 0x030c13 thru 0x03ff49 (0xf337 bytes) 000731 1787907742.301225 mapread 0x008e52 thru 0x01788d (0xea3c bytes) 000732 1787907742.326585 read 0x02949f thru 0x02e623 (0x5185 bytes) 000733 1787907742.332959 write 0x020e81 thru 0x026c33 (0x5db3 bytes) 000734 1787907742.348082 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x52a041d9 (stale) Mirror 7: 0x52a041d9 (stale) Mirror 8: 0x52a041d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x52a041d9 (stale) Mirror 7: 0x52a041d9 (stale) Mirror 8: 0x52a041d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000736 1787907742.811430 write 0x02e614 thru 0x0349ba (0x63a7 bytes) 000737 1787907742.825905 trunc from 0x040000 to 0x02e26a (0x11d95 bytes) 000738 1787907742.855022 mapwrite 0x02e0fb thru 0x039903 (0xb809 bytes) 000739 1787907743.028319 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39904): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39904): Mirror 6: 0x67f35905 (stale) Mirror 7: 0x67f35905 (stale) Mirror 8: 0x67f35905 (stale) CRC-32 checksum value for chunk [0, 0x39904): Mirror 1: 0x54216b60 Mirror 9: 0x54216b60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39904: skipped 000740 1787907744.912424 mapread 0x01bfb6 thru 0x028112 (0xc15d bytes) 000741 1787907745.029689 read 0x03257d thru 0x037d3e (0x57c2 bytes) 000743 1787907745.079977 write 0x026703 thru 0x032b7a (0xc478 bytes) 000744 1787907745.246487 write 0x02bb6e thru 0x033a9d (0x7f30 bytes) 000746 1787907745.254304 mapread 0x00f37d thru 0x014da6 (0x5a2a bytes) 000748 1787907745.286119 read 0x02e6d2 thru 0x039903 (0xb232 bytes) 000751 1787907745.316054 write 0x00cece thru 0x00ec4e (0x1d81 bytes) 000752 1787907745.339594 MIRROR_RESYNC 5 mirrors 000753 1787907747.365745 trunc from 0x039904 to 0x008a7b (0x30e88 bytes) 000754 1787907747.460131 read 0x0031be thru 0x008a7b (0x58be bytes) 000755 1787907747.468328 write 0x004d05 thru 0x00a818 (0x5b14 bytes) 000757 1787907747.472559 read 0x0021a1 thru 0x0078eb (0x574b bytes) 000763 1787907747.481336 write 0x03c4dc thru 0x03db2c (0x1651 bytes) 000764 1787907747.482433 write 0x00689f thru 0x0135c5 (0xcd27 bytes) 000766 1787907747.492463 mapread 0x03ca9b thru 0x03db2c (0x1092 bytes) 000767 1787907747.493725 mapread 0x0215f1 thru 0x030f7f (0xf98f bytes) 000768 1787907747.507506 read 0x00a9f2 thru 0x00f84e (0x4e5d bytes) 000769 1787907747.508537 mapwrite 0x02cb74 thru 0x03476b (0x7bf8 bytes) 000770 1787907747.578041 MIRROR_RESYNC 5 mirrors 000771 1787907750.225476 trunc from 0x03db2d to 0x0180d4 (0x25a58 bytes) 000772 1787907750.361484 mapwrite 0x03d9d5 thru 0x03ffff (0x262b bytes) 000774 1787907750.446581 mapwrite 0x013814 thru 0x023317 (0xfb04 bytes) 000777 1787907750.529122 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf4486f91 (stale) Mirror 7: 0xf4486f91 (stale) Mirror 9: 0xf4486f91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf4486f91 (stale) Mirror 7: 0xf4486f91 (stale) Mirror 9: 0xf4486f91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000778 1787907751.109295 write 0x02358e thru 0x0296a0 (0x6113 bytes) 000780 1787907751.127769 trunc from 0x040000 to 0x038a39 (0x75c6 bytes) 000781 1787907751.162346 write 0x01f3a7 thru 0x0280bd (0x8d17 bytes) 000782 1787907751.177855 read 0x037895 thru 0x038a39 (0x11a5 bytes) 000785 1787907751.188175 mapwrite 0x02adc5 thru 0x03192d (0x6b69 bytes) 000786 1787907751.296208 mapread 0x0356fb thru 0x038a39 (0x333f bytes) 000789 1787907751.319742 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38a3a): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a3a): Mirror 6: 0x817c3f96 (stale) Mirror 7: 0x817c3f96 (stale) Mirror 9: 0x817c3f96 (stale) CRC-32 checksum value for chunk [0, 0x38a3a): Mirror 1: 0xa99d0a94 Mirror 10: 0xa99d0a94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a3a: skipped 000791 1787907753.046774 read 0x03045e thru 0x038a39 (0x85dc bytes) 000792 1787907753.083377 mapread 0x034e6e thru 0x035838 (0x9cb bytes) 000793 1787907753.102139 mapread 0x012665 thru 0x014666 (0x2002 bytes) 000794 1787907753.139744 read 0x022753 thru 0x030b95 (0xe443 bytes) 000795 1787907753.169296 read 0x010444 thru 0x01abdc (0xa799 bytes) 000800 1787907753.217406 trunc from 0x038a3a to 0x03a94b (0x1f12 bytes) 000801 1787907753.313409 trunc from 0x03a94c to 0x012de8 (0x27b63 bytes) 000805 1787907753.339411 mapread 0x00dc8c thru 0x010522 (0x2897 bytes) 000807 1787907753.352694 read 0x003f7f thru 0x007695 (0x3717 bytes) 000808 1787907753.353419 trunc from 0x012de9 to 0x033beb (0x20e03 bytes) 000810 1787907753.372290 trunc from 0x033bec to 0x02533f (0xe8ac bytes) 000811 1787907753.399821 trunc from 0x025340 to 0x02b2d8 (0x5f99 bytes) 000812 1787907753.429296 write 0x005bf7 thru 0x00d1b3 (0x75bd bytes) 000814 1787907753.431940 read 0x016043 thru 0x02073d (0xa6fb bytes) 000816 1787907753.443193 read 0x01618c thru 0x01f8c2 (0x9737 bytes) 000817 1787907753.449903 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b2d9): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b2d9): Mirror 6: 0xd99439b2 (stale) Mirror 7: 0xd99439b2 (stale) Mirror 9: 0xd99439b2 (stale) Mirror 10: 0xe3daa70a (stale) CRC-32 checksum value for chunk [0, 0x2b2d9): Mirror 1: 0x66e3f03e Mirror 11: 0x66e3f03e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b2d9: skipped 000818 1787907755.153174 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b2d9): Mirror 6: 0xd99439b2 (stale) Mirror 7: 0xd99439b2 (stale) Mirror 9: 0xd99439b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b2d9): Mirror 6: 0xd99439b2 (stale) Mirror 7: 0xd99439b2 (stale) Mirror 9: 0xd99439b2 (stale) CRC-32 checksum value for chunk [0, 0x2b2d9): Mirror 1: 0x66e3f03e Mirror 11: 0x66e3f03e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b2d9: skipped 000819 1787907756.064166 read 0x0083e1 thru 0x00c6fe (0x431e bytes) 000820 1787907756.091224 trunc from 0x02b2d9 to 0x026276 (0x5062 bytes) 000821 1787907756.195733 trunc from 0x026277 to 0x02bfc8 (0x5d52 bytes) 000822 1787907756.242270 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfc9): Mirror 7: 0x46c01e6e (stale) Mirror 9: 0x46c01e6e (stale) Mirror 11: 0x5536842a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfc9): Mirror 7: 0x46c01e6e (stale) Mirror 9: 0x46c01e6e (stale) Mirror 11: 0x5536842a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfc9: skipped 000823 1787907756.735115 read 0x01ef3f thru 0x027010 (0x80d2 bytes) 000824 1787907756.747952 mapread 0x02126e thru 0x02bfc8 (0xad5b bytes) 000826 1787907756.802275 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bfc9): Mirror 9: 0x46c01e6e (stale) Mirror 11: 0x5536842a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bfc9): Mirror 9: 0x46c01e6e (stale) Mirror 11: 0x5536842a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bfc9: skipped 000827 1787907757.247219 mapread 0x019285 thru 0x028d9b (0xfb17 bytes) 000828 1787907757.337679 mapwrite 0x024b47 thru 0x02bcdd (0x7197 bytes) 000829 1787907757.404330 mapread 0x00c85a thru 0x01bbfd (0xf3a4 bytes) 000830 1787907757.417298 mapwrite 0x024c42 thru 0x02f133 (0xa4f2 bytes) 000831 1787907757.548267 read 0x02e8a5 thru 0x02f133 (0x88f bytes) 000834 1787907757.553548 trunc from 0x02f134 to 0x02cc9a (0x2499 bytes) 000835 1787907757.580582 MIRROR_RESYNC 3 mirrors 000837 1787907758.948541 trunc from 0x02cc9b to 0x0011b9 (0x2bae1 bytes) 000839 1787907759.020608 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11ba): Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ba): Mirror 9: 0xc6bba6cc (stale) Mirror 11: 0xc6bba6cc (stale) CRC-32 checksum value for chunk [0, 0x11ba): Mirror 1: 0xc6bba6cc Mirror 12: 0xc6bba6cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ba: skipped 000840 1787907760.096908 write 0x01ba38 thru 0x020c0a (0x51d3 bytes) 000841 1787907760.184743 read 0x00e0d2 thru 0x013f82 (0x5eb1 bytes) 000842 1787907760.193391 mapread 0x003e4b thru 0x0098f5 (0x5aab bytes) 000843 1787907760.215622 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20c0b): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c0b): Mirror 9: 0xc9353b6c (stale) Mirror 11: 0xc9353b6c (stale) Mirror 12: 0x8502c050 (stale) CRC-32 checksum value for chunk [0, 0x20c0b): Mirror 1: 0xe47d0e35 Mirror 13: 0xe47d0e35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c0b: skipped 000844 1787907761.757373 mapread 0x002928 thru 0x00e054 (0xb72d bytes) 000845 1787907761.848953 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20c0b): Mirror 11: 0xc9353b6c (stale) Mirror 12: 0x8502c050 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c0b): Mirror 11: 0xc9353b6c (stale) Mirror 12: 0x8502c050 (stale) CRC-32 checksum value for chunk [0, 0x20c0b): Mirror 1: 0xe47d0e35 Mirror 13: 0xe47d0e35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c0b: skipped 000846 1787907762.497566 trunc from 0x020c0b to 0x023f3a (0x3330 bytes) 000847 1787907762.739463 read 0x011cc5 thru 0x020253 (0xe58f bytes) 000848 1787907762.757467 trunc from 0x023f3b to 0x018fcc (0xaf6e bytes) 000849 1787907762.790369 write 0x02d6ea thru 0x033645 (0x5f5c bytes) 000850 1787907762.792559 write 0x030762 thru 0x036ab5 (0x6354 bytes) 000851 1787907762.807163 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36ab6): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36ab6): Mirror 11: 0xcc79a54f (stale) Mirror 12: 0xdc84f773 (stale) Mirror 13: 0x36ea887c (stale) CRC-32 checksum value for chunk [0, 0x36ab6): Mirror 1: 0x3f27ba7a Mirror 14: 0x3f27ba7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36ab6: skipped 000854 1787907764.337460 read 0x00e0fc thru 0x00f328 (0x122d bytes) 000855 1787907764.374502 mapwrite 0x00200e thru 0x0044b1 (0x24a4 bytes) 000856 1787907764.641386 mapwrite 0x00e1ef thru 0x0182cd (0xa0df bytes) 000858 1787907764.777007 write 0x004308 thru 0x00be94 (0x7b8d bytes) 000859 1787907764.781745 read 0x02e7f4 thru 0x02e819 (0x26 bytes) 000860 1787907764.799111 trunc from 0x036ab6 to 0x01373b (0x2337a bytes) 000863 1787907764.831175 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1373c): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1373c): Mirror 11: 0xff546982 (stale) Mirror 12: 0x4efb545e (stale) Mirror 13: 0x4efb545e (stale) Mirror 14: 0x4efb545e (stale) CRC-32 checksum value for chunk [0, 0x1373c): Mirror 1: 0xe5262702 Mirror 15: 0xe5262702 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1373c: skipped 000864 1787907766.381643 read 0x00ccf7 thru 0x01373b (0x6a45 bytes) 000865 1787907766.421402 trunc from 0x01373c to 0x00ec3e (0x4afd bytes) 000866 1787907766.621438 trunc from 0x00ec3f to 0x014cae (0x6070 bytes) 000867 1787907766.657418 write 0x021f33 thru 0x02d976 (0xba44 bytes) 000868 1787907766.662489 mapread 0x0058a1 thru 0x00dbca (0x832a bytes) 000875 1787907766.706483 write 0x02619b thru 0x026bc0 (0xa26 bytes) 000876 1787907766.707614 read 0x0008fa thru 0x00db36 (0xd23d bytes) 000877 1787907766.723761 mapwrite 0x009c51 thru 0x010282 (0x6632 bytes) 000878 1787907766.834348 trunc from 0x02d977 to 0x01f54a (0xe42c bytes) 000879 1787907766.871131 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f54b): Mirror 12: 0xc13ccb42 (stale) Mirror 13: 0x85cc568c (stale) Mirror 14: 0xc13ccb42 (stale) Mirror 15: 0x963014b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f54b): Mirror 12: 0xc13ccb42 (stale) Mirror 13: 0x85cc568c (stale) Mirror 14: 0xc13ccb42 (stale) Mirror 15: 0x963014b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f54b: skipped 000880 1787907767.315189 mapread 0x00edeb thru 0x01af3f (0xc155 bytes) 000881 1787907767.343559 mapread 0x00e29f thru 0x013ec1 (0x5c23 bytes) 000882 1787907767.345211 mapread 0x00cc3b thru 0x00de2a (0x11f0 bytes) 000885 1787907767.347649 trunc from 0x01f54b to 0x028000 (0x8ab6 bytes) 000887 1787907767.362529 write 0x008278 thru 0x0150f6 (0xce7f bytes) 000888 1787907767.369317 write 0x004392 thru 0x00c2c1 (0x7f30 bytes) 000889 1787907767.379916 MIRROR_RESYNC 5 mirrors 000890 1787907769.411310 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) truncating to largest ever: 0x3f87e 000892 1787907769.735216 trunc from 0x028001 to 0x03f87d (0x1787d bytes) 000893 1787907769.833953 mapwrite 0x0004ce thru 0x00391d (0x3450 bytes) 000895 1787907769.907492 trunc from 0x03f87e to 0x012e35 (0x2ca48 bytes) 000896 1787907769.949947 mapwrite 0x02ad7c thru 0x02bea3 (0x1128 bytes) 000897 1787907770.023661 trunc from 0x02bea4 to 0x02fe08 (0x3f65 bytes) 000899 1787907770.043570 mapwrite 0x031a4d thru 0x03ffff (0xe5b3 bytes) 000902 1787907770.154367 write 0x00ebfd thru 0x016aa4 (0x7ea8 bytes) 000903 1787907770.161572 mapread 0x030562 thru 0x03efb4 (0xea53 bytes) 000904 1787907770.167370 write 0x006837 thru 0x013845 (0xd00f bytes) 000905 1787907770.170263 trunc from 0x040000 to 0x0098c5 (0x3673a bytes) 000907 1787907770.195413 mapwrite 0x027eb4 thru 0x02fda4 (0x7ef1 bytes) 000908 1787907770.286129 read 0x026777 thru 0x02abf9 (0x4483 bytes) 000909 1787907770.286873 read 0x01110a thru 0x01944d (0x8344 bytes) 000910 1787907770.300405 mapwrite 0x02fb31 thru 0x03d148 (0xd618 bytes) 000911 1787907770.412107 write 0x0395fd thru 0x03ffff (0x6a03 bytes) 000914 1787907770.415822 trunc from 0x040000 to 0x02fd3b (0x102c4 bytes) 000916 1787907770.446517 read 0x011646 thru 0x018592 (0x6f4d bytes) 000917 1787907770.447342 mapread 0x001ebc thru 0x002412 (0x557 bytes) 000918 1787907770.449336 mapwrite 0x03c8c4 thru 0x03ffff (0x373c bytes) 000919 1787907770.523290 write 0x012df9 thru 0x0224cb (0xf6d3 bytes) 000921 1787907770.529735 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x97a72ba5 (stale) Mirror 14: 0x97a72ba5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x97a72ba5 (stale) Mirror 14: 0x97a72ba5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000924 1787907771.046914 read 0x0082e1 thru 0x015c9e (0xd9be bytes) 000925 1787907771.071247 mapread 0x0126f5 thru 0x0179da (0x52e6 bytes) 000927 1787907771.127236 mapread 0x01d82a thru 0x01e9fe (0x11d5 bytes) 000928 1787907771.153924 mapread 0x026572 thru 0x03412d (0xdbbc bytes) 000930 1787907771.216727 mapwrite 0x027bf4 thru 0x02c2fd (0x470a bytes) 000931 1787907771.428640 MIRROR_RESYNC 3 mirrors 000935 1787907773.719478 trunc from 0x040000 to 0x000868 (0x3f797 bytes) 000936 1787907773.870433 read 0x0006b8 thru 0x000868 (0x1b1 bytes) 000937 1787907773.899755 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x869): Mirror 14: 0x2c4324e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x869): Mirror 14: 0x2c4324e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x869: skipped 000940 1787907774.260497 mapread 0x00007a thru 0x000868 (0x7ef bytes) 000942 1787907774.271660 trunc from 0x000869 to 0x01cc67 (0x1c3ff bytes) 000947 1787907774.294877 mapread 0x003502 thru 0x0045a2 (0x10a1 bytes) 000949 1787907774.307456 mapread 0x01cb2d thru 0x01cc67 (0x13b bytes) 000950 1787907774.317003 trunc from 0x01cc68 to 0x010dfc (0xbe6b bytes) 000951 1787907774.335043 mapwrite 0x013201 thru 0x01a1e5 (0x6fe5 bytes) 000952 1787907774.479659 trunc from 0x01a1e6 to 0x038823 (0x1e63e bytes) 000953 1787907774.518451 mapwrite 0x02e6f7 thru 0x02e7cb (0xd5 bytes) 000955 1787907774.607771 write 0x0206d3 thru 0x02be93 (0xb7c1 bytes) 000956 1787907774.615856 mapread 0x02f4b9 thru 0x037ed1 (0x8a19 bytes) 000957 1787907774.624327 read 0x01516b thru 0x022892 (0xd728 bytes) 000958 1787907774.646518 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x190:0x0]: Invalid argument (22) 000959 1787907774.914850 mapread 0x0081ac thru 0x00a02b (0x1e80 bytes) 000963 1787907774.940305 mapread 0x001884 thru 0x0023df (0xb5c bytes) 000964 1787907774.946346 mapwrite 0x030bb7 thru 0x030d76 (0x1c0 bytes) 000966 1787907775.003466 mapwrite 0x01074c thru 0x01ada4 (0xa659 bytes) 000967 1787907775.101316 mapread 0x02e056 thru 0x037d14 (0x9cbf bytes) 000968 1787907775.103036 read 0x02f69c thru 0x038823 (0x9188 bytes) 000969 1787907775.104317 mapwrite 0x01df3d thru 0x02d402 (0xf4c6 bytes) 000970 1787907775.202884 write 0x03d53d thru 0x03ffff (0x2ac3 bytes) 000971 1787907775.204661 trunc from 0x040000 to 0x01650a (0x29af5 bytes) 000972 1787907775.224325 mapread 0x00d4c8 thru 0x00e3f2 (0xf2b bytes) 000973 1787907775.225987 mapread 0x000a4b thru 0x00d2eb (0xc8a1 bytes) 000978 1787907775.237427 read 0x0058a9 thru 0x014452 (0xebaa bytes) 000980 1787907775.240314 mapwrite 0x024384 thru 0x02bcb8 (0x7935 bytes) 000982 1787907775.361829 mapread 0x008498 thru 0x00ce28 (0x4991 bytes) 000984 1787907775.369017 mapwrite 0x039e86 thru 0x03ffff (0x617a bytes) 000988 1787907775.499944 write 0x0246e1 thru 0x02709c (0x29bc bytes) 000990 1787907775.503760 read 0x03475f thru 0x0347cb (0x6d bytes) 000992 1787907775.506473 mapwrite 0x030f12 thru 0x03ff82 (0xf071 bytes) 001000 1787907775.660081 mapwrite 0x0327d1 thru 0x03ffff (0xd82f bytes) All operations completed A-OK! PASS 70a (143s) == sanity-flr test 71: check mirror extend parent fid ==== 05:03:00 (1787907780) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 05:03:05 (1787907785) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.401918 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200362 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 05:03:16 (1787907796) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5488992 .. Extending file size to 5855456 .. done Extending file size to 5977408 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6008480 .. Extending file size to 6060576 .. Extending file size to 7987424 .. Extending file size to 8717376 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8015840 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8050464 .. Extending file size to 8124448 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8914144 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8318496 .. Extending file size to 8610272 .. Waiting 57388 57389 57390 57392 57393 done 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) 192.168.206.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:04:45 (1787907885) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3750624 .. Extending file size to 2323840 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4478304 .. Extending file size to 5193376 .. Extending file size to 8226432 .. Extending file size to 7328384 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8626560 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7756256 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8676640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7871168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8223232 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8959840 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8480256 .. Extending file size to 9021664 .. Waiting 58810 58811 58813 58815 58817 done 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) 192.168.206.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:06:04 (1787907964) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126441 s, 16.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.503777 s, 4.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe549cd86 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:06:23 (1787907983) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0930591 s, 22.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 05:06:30 (1787907990) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:06:37 (1787907997) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221161 s, 9.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:06:46 (1787908006) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153219 s, 13.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:06:54 (1787908014) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147367 s, 14.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:07:26 (1787908046) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg610-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138496 s, 7.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:08:25 (1787908105) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg610-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151225 s, 13.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:09:19 (1787908159) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg610-server mds-ost sync done. Creating new pool test_204f oleg610-server: Pool lustre.test_204f created Adding targets to pool oleg610-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg610-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164243 s, 12.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg610-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg610-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg610-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:10:38 (1787908238) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:10:46 (1787908246) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:10:55 (1787908255) oleg610-server: Pool lustre.test_206 created Adding targets to pool oleg610-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg610-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg610-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg610-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg610-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:11:19 (1787908279) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875506 s, 12.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447299 s, 23.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd1d9081d Mirror 3: 0xd1d9081d PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:11:31 (1787908291) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:11:35 (1787908295) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:11:39 (1787908299) Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0961272 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1650000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xaa0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xaa0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xaa0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1e30000 Read this (4 bytes): BBBB 192.168.206.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg610-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:12:16 (1787908336) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0422344 s, 194 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:12:23 (1787908343) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323715 s, 32.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x208 lmm_fid: [0x200000404:0x208:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x208 lmm_fid: [0x200000404:0x208:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834440 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-28 05:12:25.000000000 -0400 Modify: 2026-08-28 05:12:25.000000000 -0400 Change: 2026-08-28 05:12:26.000000000 -0400 Birth: 2026-08-28 05:12:25.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:12:31 (1787908351) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:12:33 (1787908353) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.554479 s, 73.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0402913 s, 102 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 28 05:12 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 28 05:12 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:12:42 (1787908362) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:12:49 (1787908369) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1890 sec =========== 05:13:02 (1787908382) === sanity-flr: start cleanup 05:13:04 (1787908384) === === sanity-flr: finish cleanup 05:13:13 (1787908393) ===