-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 21:09:49 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg139-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:10:11 (1781226611) === oleg139-client.virtnet: executing check_config_client /mnt/lustre oleg139-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg139-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fe4c9080800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe4c9080800.idle_timeout=debug disable quota as required oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:10:46 (1781226646) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7580] == sanity-flr test 0a: lfs mirror create with -N option == 21:10:49 (1781226649) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:11:07 (1781226667) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:11:09 (1781226669) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:11:13 (1781226673) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:11:41 (1781226701) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:11:45 (1781226705) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 21:11:49 (1781226709) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9fe4c9080800.stats=clear osc.lustre-OST0001-osc-ffff9fe4c9080800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:11:58 (1781226718) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:12:10 (1781226730) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:12:21 (1781226741) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:12:26 (1781226746) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:12:37 (1781226757) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 21:12:48 (1781226768) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:12:55 (1781226775) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:12:59 (1781226779) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:13:08 (1781226788) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:13:20 (1781226800) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:13:27 (1781226807) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 2.05315 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791199 s, 13.3 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:13:41 (1781226821) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.79156 s, 26.5 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:13:51 (1781226831) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:14:01 (1781226841) Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg139-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg139-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:16:20 (1781226980) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.754204 s, 15.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:17:04 (1781227024) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg139-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:18:27 (1781227107) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.56613 s, 14.7 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 21:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.900s user 0m0.001s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe4c9080800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.137s user 0m0.000s sys 0m0.178s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg139-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg139-server oleg139-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.364s user 0m0.002s sys 0m0.144s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg139-server: oleg139-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg139-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg139-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe4c9080800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:20:00 (1781227200) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:20:04 (1781227204) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:20:07 (1781227207) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 21:20:11 (1781227211) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 21:20:19 (1781227219) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.604082 s, 20.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:20:34 (1781227234) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315443 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030449 s, 34.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:20:41 (1781227241) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:20:49 (1781227249) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000810886 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105252 s, 973 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0023341 s, 439 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114035 s, 898 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 21:20:59 (1781227259) create test files with size 15732311 .. af57db340c2d7806f1cb5172cceba36f - 647cdf536f2acb9e11a694f8e55c9245 - a2dd6048f02792c5983b6e04b3c4f236 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg139-client.virtnet /mnt/lustre (opts:) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 21:21:23 (1781227283) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:21:37 (1781227297) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:21:44 (1781227304) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178376 s, 5.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237822 s, 44.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 21:21:55 (1781227315) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113034 s, 18.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.275995 s, 15.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191614 s, 10.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 21:22:11 (1781227331) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 21:22:14 (1781227334) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:22:20 (1781227340) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347216 s, 30.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 21:22:29 (1781227349) llite.lustre-ffff9fe4c8aef000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 44a: lfs mirror split check =========== 21:22:38 (1781227358) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:22:42 (1781227362) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:22:55 (1781227375) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.421167 s, 24.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1a:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg139-client.virtnet /mnt/lustre (opts:) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:23:11 (1781227391) llite.lustre-ffff9fe4c9a2e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.331929 s, 31.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 36878 flags: 1 Stopping client oleg139-client.virtnet /mnt/lustre (opts:) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 36878 flags: 1 llite.lustre-ffff9fe4c9b64800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:23:29 (1781227409) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:24:10 (1781227450) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:24:52 (1781227492) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 21:24:56 (1781227496) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.185127 s, 17.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0623999 s, 50.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x59937401 (nosync,stale) Mirror 2: 0x59937401 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 39fa099594ddfe51771907b6ffb916c3 - mirror 1 checksum: 56eed137cf9cf5bcdbf5ec753a79d374 - mirror 2 checksum: 39fa099594ddfe51771907b6ffb916c3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:24:59 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 21:24:59 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 39fa099594ddfe51771907b6ffb916c3 - mirror 1 checksum: 56eed137cf9cf5bcdbf5ec753a79d374 - mirror 2 checksum: 56eed137cf9cf5bcdbf5ec753a79d374 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:24:59 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 21:24:59 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:25:07 (1781227507) Failing mds1 on oleg139-server Stopping /mnt/lustre-mds1 (opts:) on oleg139-server 21:25:14 (1781227514) shut down facet: mds1 facet_host: oleg139-server facet_failover_host: oleg139-server Failover mds1 to oleg139-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 21:25:35 (1781227535) targets are mounted 21:25:35 (1781227535) facet_failover done oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179717 s, 58.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:25:52 (1781227552) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:25:57 (1781227557) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:26:00 (1781227560) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00562646 s, 2.9 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:26:08 (1781227568) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:26:12 (1781227572) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:26:15 (1781227575) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:26:39 (1781227599) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:27:08 (1781227628) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:27:34 (1781227654) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:27:59 (1781227679) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 21:28:08 (1781227688) mirror create pid 54141 mirror split pid 54142 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54141 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54142 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 21:29:15 (1781227755) Seed set to 7758 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1781227758.480841 mapwrite 0x001e42 thru 0x0114ef (0xf6ae bytes) 000010 1781227758.568854 write 0x00a49d thru 0x013c51 (0x97b5 bytes) 000013 1781227758.571067 read 0x00c99a thru 0x013c51 (0x72b8 bytes) 000014 1781227758.571879 read 0x00b1e3 thru 0x013c51 (0x8a6f bytes) 000015 1781227758.572520 mapread 0x01261a thru 0x013c51 (0x1638 bytes) truncating to largest ever: 0x28c21 000017 1781227758.574581 trunc from 0x013c52 to 0x028c20 (0x14fcf bytes) 000018 1781227758.586915 trunc from 0x028c21 to 0x0005ec (0x28634 bytes) 000021 1781227758.598290 trunc from 0x0005ed to 0x0286b5 (0x280c9 bytes) 000022 1781227758.607932 trunc from 0x0286b6 to 0x0104fe (0x181b7 bytes) truncating to largest ever: 0x366b2 000024 1781227758.637492 trunc from 0x0104ff to 0x0366b1 (0x261b3 bytes) 000025 1781227758.663366 read 0x01a130 thru 0x020e49 (0x6d1a bytes) 000026 1781227758.705288 read 0x023612 thru 0x0271e7 (0x3bd6 bytes) 000027 1781227758.706719 mapread 0x02a66e thru 0x0366b1 (0xc044 bytes) 000029 1781227758.712581 read 0x00eb93 thru 0x0117fe (0x2c6c bytes) 000032 1781227758.714672 mapwrite 0x0287dc thru 0x030b5b (0x8380 bytes) 000034 1781227758.796142 read 0x013e0b thru 0x018bec (0x4de2 bytes) 000038 1781227758.803114 trunc from 0x0366b2 to 0x0102bf (0x263f2 bytes) 000039 1781227758.826832 read 0x00d9c5 thru 0x0102bf (0x28fb bytes) 000040 1781227758.827507 trunc from 0x0102c0 to 0x01a423 (0xa164 bytes) 000042 1781227758.851096 write 0x0249f5 thru 0x02c420 (0x7a2c bytes) 000043 1781227758.853304 mapwrite 0x00abc7 thru 0x00ed9f (0x41d9 bytes) 000044 1781227758.920275 trunc from 0x02c421 to 0x02a92e (0x1af2 bytes) 000050 1781227758.946368 mapread 0x0061fd thru 0x014971 (0xe775 bytes) 000051 1781227758.958463 mapread 0x01fc60 thru 0x02614d (0x64ee bytes) 000052 1781227758.959339 write 0x0005af thru 0x00c807 (0xc259 bytes) 000056 1781227758.963254 read 0x022780 thru 0x02a92e (0x81af bytes) 000058 1781227758.965025 trunc from 0x02a92f to 0x02e688 (0x3d5a bytes) 000061 1781227758.986628 trunc from 0x02e689 to 0x0251e0 (0x94a8 bytes) 000063 1781227759.016631 read 0x0103ff thru 0x0155e3 (0x51e5 bytes) 000064 1781227759.017681 write 0x00683f thru 0x016652 (0xfe14 bytes) 000065 1781227759.020769 trunc from 0x0251e1 to 0x035a07 (0x10827 bytes) 000066 1781227759.046142 read 0x0321dc thru 0x035a07 (0x382c bytes) 000067 1781227759.056888 mapwrite 0x01c2ab thru 0x02aed2 (0xec28 bytes) 000069 1781227759.095737 mapwrite 0x01ac71 thru 0x023c77 (0x9007 bytes) 000072 1781227759.135494 write 0x0014ba thru 0x00c5d4 (0xb11b bytes) 000073 1781227759.136568 mapwrite 0x0109e1 thru 0x01fc6c (0xf28c bytes) 000074 1781227759.180476 mapwrite 0x0353b5 thru 0x036ca1 (0x18ed bytes) 000077 1781227759.257658 mapwrite 0x003f2d thru 0x00a22a (0x62fe bytes) 000079 1781227759.317330 mapwrite 0x01789d thru 0x024bdf (0xd343 bytes) 000080 1781227759.360527 write 0x00a997 thru 0x017be3 (0xd24d bytes) 000081 1781227759.362256 read 0x00be9c thru 0x018c48 (0xcdad bytes) 000083 1781227759.363459 write 0x00335b thru 0x00beca (0x8b70 bytes) 000084 1781227759.364576 read 0x036268 thru 0x036ca1 (0xa3a bytes) 000085 1781227759.365068 write 0x007320 thru 0x00acd9 (0x39ba bytes) 000086 1781227759.365898 mapwrite 0x0011b2 thru 0x007d44 (0x6b93 bytes) 000087 1781227759.440279 mapread 0x002ed0 thru 0x0089cd (0x5afe bytes) 000088 1781227759.442411 mapread 0x013e34 thru 0x01ffca (0xc197 bytes) 000090 1781227759.449823 mapwrite 0x020b16 thru 0x0272a0 (0x678b bytes) 000095 1781227759.495527 write 0x030529 thru 0x038a67 (0x853f bytes) 000096 1781227759.497723 mapread 0x013b0c thru 0x01568e (0x1b83 bytes) 000097 1781227759.499881 write 0x01ec8a thru 0x02c734 (0xdaab bytes) 000100 1781227759.504565 trunc from 0x038a68 to 0x01a901 (0x1e166 bytes) 000103 1781227759.527101 read 0x0021e8 thru 0x007739 (0x5552 bytes) 000105 1781227759.528734 read 0x009b44 thru 0x014de4 (0xb2a1 bytes) 000110 1781227759.533565 mapwrite 0x03c03d thru 0x03ffff (0x3fc3 bytes) 000114 1781227759.645927 read 0x0063da thru 0x00e040 (0x7c67 bytes) 000115 1781227759.647279 mapread 0x0226a8 thru 0x02ca40 (0xa399 bytes) 000117 1781227759.649385 trunc from 0x040000 to 0x03276e (0xd891 bytes) 000118 1781227759.669576 MIRROR_EXTEND to 2 mirrors 000121 1781227760.223690 trunc from 0x03276f to 0x02e989 (0x3de5 bytes) 000124 1781227760.402451 mapwrite 0x026b58 thru 0x031c9b (0xb144 bytes) 000125 1781227760.642517 mapread 0x007f83 thru 0x017a6e (0xfaec bytes) 000126 1781227760.677978 mapwrite 0x02ecd9 thru 0x031e2c (0x3154 bytes) 000127 1781227760.850619 read 0x00d7fc thru 0x00e047 (0x84c bytes) 000130 1781227760.868456 mapwrite 0x037224 thru 0x03ffff (0x8ddc bytes) 000132 1781227761.092805 read 0x03b5de thru 0x03d1c0 (0x1be3 bytes) 000136 1781227761.102900 mapwrite 0x03a43e thru 0x03ffff (0x5bc2 bytes) 000137 1781227761.211375 read 0x0068e1 thru 0x010c59 (0xa379 bytes) 000139 1781227761.212600 trunc from 0x040000 to 0x008316 (0x37ce9 bytes) 000141 1781227761.284517 read 0x000b61 thru 0x008316 (0x77b6 bytes) 000142 1781227761.307056 MIRROR_RESYNC 2 mirrors 000143 1781227761.837129 mapread 0x006667 thru 0x008316 (0x1cb0 bytes) 000144 1781227761.852083 write 0x015143 thru 0x01c226 (0x70e4 bytes) 000145 1781227761.917987 mapwrite 0x01e6cc thru 0x0220a0 (0x39d5 bytes) 000147 1781227762.126564 read 0x0026bd thru 0x007c37 (0x557b bytes) 000148 1781227762.138621 read 0x01b41c thru 0x0220a0 (0x6c85 bytes) 000151 1781227762.140439 mapwrite 0x03cbe5 thru 0x03ffff (0x341b bytes) 000152 1781227762.222772 trunc from 0x040000 to 0x018094 (0x27f6b bytes) 000153 1781227762.247745 trunc from 0x018095 to 0x00dca6 (0xa3ee bytes) 000154 1781227762.297963 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000155 1781227762.555545 trunc from 0x00dca7 to 0x015568 (0x78c2 bytes) 000156 1781227762.585526 mapread 0x00806c thru 0x015568 (0xd4fd bytes) 000157 1781227762.614753 trunc from 0x015569 to 0x01c6a2 (0x713a bytes) 000158 1781227762.641896 write 0x0119a5 thru 0x0131af (0x180b bytes) 000159 1781227762.643050 mapwrite 0x03ad2f thru 0x03ffff (0x52d1 bytes) 000160 1781227762.715860 read 0x0368f6 thru 0x03b20d (0x4918 bytes) 000162 1781227762.721770 MIRROR_EXTEND to 2 mirrors 000164 1781227763.276631 read 0x005e51 thru 0x014aa9 (0xec59 bytes) 000166 1781227763.285381 mapread 0x0235e0 thru 0x032a01 (0xf422 bytes) 000167 1781227763.314066 trunc from 0x040000 to 0x0123ac (0x2dc53 bytes) 000169 1781227763.403677 write 0x0237ce thru 0x02bf4c (0x877f bytes) 000171 1781227763.409894 read 0x02b974 thru 0x02bf4c (0x5d9 bytes) 000175 1781227763.413670 write 0x00d5a4 thru 0x01c4a7 (0xef04 bytes) 000177 1781227763.426936 read 0x00d215 thru 0x01c9bf (0xf7ab bytes) 000179 1781227763.430622 mapread 0x022095 thru 0x024c3d (0x2ba9 bytes) 000181 1781227763.437008 mapread 0x01936b thru 0x01a9bf (0x1655 bytes) 000183 1781227763.450154 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bf4d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf4d): Mirror 2: 0xf8e8db1f (stale) CRC-32 checksum value for chunk [0, 0x2bf4d): Mirror 1: 0xf7b2163f Mirror 3: 0xf7b2163f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf4d: skipped 000184 1781227764.395364 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2bf4d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf4d): Mirror 2: 0xf8e8db1f (stale) CRC-32 checksum value for chunk [0, 0x2bf4d): Mirror 1: 0xf7b2163f Mirror 3: 0xf7b2163f Mirror 4: 0xf7b2163f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf4d: skipped truncating to largest ever: 0x37082 000185 1781227765.704567 trunc from 0x02bf4d to 0x037081 (0xb135 bytes) 000186 1781227765.837692 MIRROR_RESYNC 4 mirrors 000187 1781227766.778478 write 0x004be4 thru 0x006e4d (0x226a bytes) 000188 1781227766.913570 mapread 0x01fa10 thru 0x02a3cc (0xa9bd bytes) 000189 1781227766.966584 read 0x004cf2 thru 0x011633 (0xc942 bytes) 000192 1781227766.977698 mapwrite 0x035907 thru 0x03fdca (0xa4c4 bytes) 000193 1781227767.079352 read 0x033493 thru 0x0352f1 (0x1e5f bytes) 000194 1781227767.080107 read 0x00f37b thru 0x017840 (0x84c6 bytes) 000195 1781227767.091949 mapread 0x012625 thru 0x01b675 (0x9051 bytes) 000196 1781227767.099161 MIRROR_RESYNC 4 mirrors 000197 1781227768.117671 trunc from 0x03fdcb to 0x014efa (0x2aed0 bytes) 000198 1781227768.180243 mapread 0x00d57f thru 0x014efa (0x797c bytes) 000199 1781227768.203645 read 0x00a329 thru 0x00d98b (0x3663 bytes) 000201 1781227768.205023 read 0x01148c thru 0x014efa (0x3a6f bytes) 000202 1781227768.205546 mapread 0x0134eb thru 0x014efa (0x1a10 bytes) 000204 1781227768.209754 MIRROR_RESYNC 4 mirrors 000205 1781227769.243651 mapread 0x00f35f thru 0x014efa (0x5b9c bytes) 000206 1781227769.268115 mapwrite 0x03c152 thru 0x03ffff (0x3eae bytes) 000208 1781227769.453159 mapwrite 0x003611 thru 0x01198e (0xe37e bytes) 000209 1781227769.514353 trunc from 0x040000 to 0x00d5d2 (0x32a2d bytes) 000211 1781227769.547244 write 0x0308ef thru 0x03bc1f (0xb331 bytes) 000213 1781227769.550546 trunc from 0x03bc20 to 0x012ef4 (0x28d2b bytes) 000216 1781227769.573097 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x12ef5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12ef5): Mirror 2: 0x42eac1d1 (stale) Mirror 3: 0x42eac1d1 (stale) Mirror 4: 0x42eac1d1 (stale) CRC-32 checksum value for chunk [0, 0x12ef5): Mirror 1: 0xfa1710a9 Mirror 5: 0xfa1710a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12ef5: skipped 000217 1781227770.449722 mapwrite 0x016c5f thru 0x017470 (0x812 bytes) 000219 1781227770.599667 trunc from 0x017471 to 0x0299b7 (0x12547 bytes) 000221 1781227770.626835 mapread 0x009920 thru 0x01487e (0xaf5f bytes) 000222 1781227770.664448 read 0x004f4e thru 0x010aca (0xbb7d bytes) 000224 1781227770.666890 write 0x03df6f thru 0x03ffff (0x2091 bytes) 000225 1781227770.668652 mapwrite 0x029949 thru 0x02ffc1 (0x6679 bytes) 000226 1781227770.810107 read 0x007c7f thru 0x00ae02 (0x3184 bytes) 000227 1781227770.814308 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4912a08a (stale) Mirror 3: 0x4912a08a (stale) Mirror 5: 0x1805e641 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4912a08a (stale) Mirror 3: 0x4912a08a (stale) Mirror 5: 0x1805e641 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000228 1781227771.415071 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4912a08a (stale) Mirror 5: 0x1805e641 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4912a08a (stale) Mirror 5: 0x1805e641 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000230 1781227771.928150 MIRROR_RESYNC 3 mirrors 000232 1781227772.848033 read 0x009f13 thru 0x011512 (0x7600 bytes) 000233 1781227772.864314 mapwrite 0x028465 thru 0x02b4b8 (0x3054 bytes) 000235 1781227773.227687 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc78f369f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc78f369f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1781227773.625143 trunc from 0x040000 to 0x000feb (0x3f014 bytes) 000238 1781227773.650423 trunc from 0x000fec to 0x003a6d (0x2a82 bytes) 000240 1781227773.675384 trunc from 0x003a6e to 0x030d82 (0x2d315 bytes) 000242 1781227773.698564 write 0x015cba thru 0x019a4a (0x3d91 bytes) 000243 1781227773.714172 mapwrite 0x03d633 thru 0x03ffff (0x29cd bytes) 000244 1781227773.789336 mapwrite 0x0073e9 thru 0x00ece9 (0x7901 bytes) 000245 1781227773.843685 mapwrite 0x02b824 thru 0x03b3a1 (0xfb7e bytes) 000246 1781227773.978364 write 0x00a9f1 thru 0x01822d (0xd83d bytes) 000248 1781227773.990532 read 0x002712 thru 0x01175f (0xf04e bytes) 000249 1781227773.991440 read 0x00990d thru 0x0181c6 (0xe8ba bytes) 000251 1781227774.006944 MIRROR_RESYNC 2 mirrors 000252 1781227774.848300 mapwrite 0x004c94 thru 0x00a30c (0x5679 bytes) 000255 1781227775.077563 trunc from 0x040000 to 0x01d5b3 (0x22a4c bytes) 000257 1781227775.095579 mapwrite 0x01f38d thru 0x029070 (0x9ce4 bytes) 000258 1781227775.187459 write 0x01baf4 thru 0x02591d (0x9e2a bytes) 000260 1781227775.196305 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29071): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29071): Mirror 2: 0x36ddaf0d (stale) CRC-32 checksum value for chunk [0, 0x29071): Mirror 1: 0xac4a70b6 Mirror 3: 0xac4a70b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29071: skipped 000261 1781227775.965210 mapwrite 0x038922 thru 0x03f16d (0x684c bytes) 000265 1781227776.341483 read 0x008918 thru 0x015574 (0xcc5d bytes) 000267 1781227776.367754 mapwrite 0x009ea6 thru 0x00aed2 (0x102d bytes) 000271 1781227776.485547 mapwrite 0x0148c8 thru 0x018640 (0x3d79 bytes) 000272 1781227776.614544 read 0x027525 thru 0x033784 (0xc260 bytes) 000273 1781227776.642707 mapwrite 0x010889 thru 0x012ed2 (0x264a bytes) 000274 1781227776.795607 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f16e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f16e): Mirror 2: 0x3f79adad (stale) Mirror 3: 0x7d432901 (stale) CRC-32 checksum value for chunk [0, 0x3f16e): Mirror 1: 0x7e1cc228 Mirror 4: 0x7e1cc228 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f16e: skipped 000277 1781227777.881273 mapwrite 0x008a70 thru 0x00e8aa (0x5e3b bytes) 000278 1781227778.092626 trunc from 0x03f16e to 0x01e5da (0x20b93 bytes) 000279 1781227778.144133 mapwrite 0x02c8c3 thru 0x03c237 (0xf975 bytes) 000280 1781227778.323043 write 0x025472 thru 0x02727e (0x1e0d bytes) 000281 1781227778.324279 mapread 0x03a2db thru 0x03c237 (0x1f5d bytes) 000283 1781227778.340012 write 0x03dcd1 thru 0x03e7c8 (0xaf8 bytes) 000284 1781227778.352636 MIRROR_RESYNC 4 mirrors 000288 1781227779.647059 MIRROR_EXTEND to 5 mirrors 000289 1781227780.761453 trunc from 0x03e7c9 to 0x007cc5 (0x36b03 bytes) 000290 1781227780.956857 trunc from 0x007cc6 to 0x022ed0 (0x1b20b bytes) 000293 1781227781.002198 mapread 0x00f850 thru 0x013517 (0x3cc8 bytes) 000295 1781227781.088397 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22ed1): Mirror 2: 0x4d68f11a (stale) Mirror 4: 0x4d68f11a (stale) Mirror 5: 0x4d68f11a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22ed1): Mirror 2: 0x4d68f11a (stale) Mirror 4: 0x4d68f11a (stale) Mirror 5: 0x4d68f11a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22ed1: skipped 000297 1781227781.657494 mapread 0x022e79 thru 0x022ed0 (0x58 bytes) 000298 1781227781.665617 mapwrite 0x02bf7e thru 0x02d534 (0x15b7 bytes) 000299 1781227781.743691 mapwrite 0x024030 thru 0x025faa (0x1f7b bytes) 000300 1781227781.817189 read 0x023a43 thru 0x029eae (0x646c bytes) 000304 1781227781.820325 read 0x01570f thru 0x024c47 (0xf539 bytes) 000305 1781227781.833742 mapwrite 0x02d802 thru 0x038be8 (0xb3e7 bytes) 000306 1781227781.942003 read 0x02c370 thru 0x02ff45 (0x3bd6 bytes) 000307 1781227781.943027 mapwrite 0x028e3a thru 0x03429d (0xb464 bytes) 000308 1781227782.007348 mapwrite 0x008b3f thru 0x00f272 (0x6734 bytes) 000310 1781227782.064672 trunc from 0x038be9 to 0x020c08 (0x17fe0 bytes) 000311 1781227782.091677 read 0x00bf1f thru 0x012d58 (0x6e3a bytes) 000312 1781227782.093733 mapread 0x00c6b2 thru 0x01342c (0x6d7b bytes) 000315 1781227782.099126 trunc from 0x020c09 to 0x01e3fd (0x280b bytes) 000316 1781227782.118754 write 0x01e391 thru 0x026cc3 (0x8933 bytes) 000317 1781227782.123299 mapwrite 0x0395c9 thru 0x03ffff (0x6a37 bytes) 000318 1781227782.247921 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xcfbc7e18 (stale) Mirror 5: 0xcfbc7e18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xcfbc7e18 (stale) Mirror 5: 0xcfbc7e18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000319 1781227782.563614 write 0x0221a0 thru 0x02c88b (0xa6ec bytes) 000320 1781227782.575531 read 0x00abb2 thru 0x00e82a (0x3c79 bytes) 000323 1781227782.584690 write 0x017008 thru 0x01bfb5 (0x4fae bytes) 000324 1781227782.604222 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xcfbc7e18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xcfbc7e18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000326 1781227782.924422 write 0x016095 thru 0x024338 (0xe2a4 bytes) 000328 1781227782.940654 mapwrite 0x00707e thru 0x00f34b (0x82ce bytes) 000330 1781227783.006386 mapwrite 0x032d98 thru 0x03d68f (0xa8f8 bytes) 000332 1781227783.103442 read 0x02110c thru 0x029917 (0x880c bytes) 000333 1781227783.115234 trunc from 0x040000 to 0x016eed (0x29112 bytes) 000334 1781227783.140227 mapread 0x0136d3 thru 0x016eed (0x381b bytes) 000335 1781227783.155221 mapread 0x00284e thru 0x004b00 (0x22b3 bytes) truncating to largest ever: 0x3a292 000336 1781227783.157913 trunc from 0x016eee to 0x03a291 (0x233a4 bytes) 000339 1781227783.180557 write 0x02482f thru 0x025912 (0x10e4 bytes) 000341 1781227783.195127 write 0x01a18e thru 0x01e05d (0x3ed0 bytes) 000344 1781227783.208703 read 0x00f489 thru 0x012847 (0x33bf bytes) 000345 1781227783.210078 read 0x0388e1 thru 0x03a291 (0x19b1 bytes) 000346 1781227783.220657 trunc from 0x03a292 to 0x026f60 (0x13331 bytes) 000348 1781227783.243677 mapread 0x010656 thru 0x01a4b2 (0x9e5d bytes) 000349 1781227783.246593 mapread 0x018da1 thru 0x024879 (0xbad9 bytes) 000350 1781227783.253348 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000351 1781227783.427517 mapwrite 0x0242b3 thru 0x027b01 (0x384f bytes) 000352 1781227783.556371 read 0x027a0e thru 0x027b01 (0xf4 bytes) 000353 1781227783.557075 read 0x00ba82 thru 0x011a2f (0x5fae bytes) 000355 1781227783.589329 trunc from 0x027b02 to 0x01db31 (0x9fd0 bytes) 000358 1781227783.620680 write 0x033194 thru 0x034205 (0x1072 bytes) 000360 1781227783.625641 mapread 0x021435 thru 0x02ac61 (0x982d bytes) 000363 1781227783.687701 mapwrite 0x0012af thru 0x009153 (0x7ea5 bytes) 000364 1781227783.785361 read 0x005ef8 thru 0x013231 (0xd33a bytes) 000368 1781227783.790956 write 0x01c290 thru 0x0217bd (0x552e bytes) 000374 1781227783.802907 write 0x011479 thru 0x01ccdb (0xb863 bytes) 000378 1781227783.809305 write 0x03adc2 thru 0x03ffff (0x523e bytes) 000380 1781227783.813708 mapwrite 0x017917 thru 0x023a14 (0xc0fe bytes) 000383 1781227783.918845 MIRROR_EXTEND to 2 mirrors 000386 1781227784.912205 mapread 0x0206d0 thru 0x025364 (0x4c95 bytes) 000387 1781227784.936146 read 0x03b230 thru 0x03ffff (0x4dd0 bytes) 000388 1781227784.947278 mapwrite 0x0371c3 thru 0x03ffff (0x8e3d bytes) 000389 1781227785.154462 write 0x03dd33 thru 0x03ffff (0x22cd bytes) 000390 1781227785.156284 mapwrite 0x0336d5 thru 0x036522 (0x2e4e bytes) 000391 1781227785.214522 read 0x019b02 thru 0x025a4b (0xbf4a bytes) 000393 1781227785.229425 read 0x006347 thru 0x015ca2 (0xf95c bytes) 000394 1781227785.230344 mapwrite 0x00fa4e thru 0x01d59e (0xdb51 bytes) 000395 1781227785.333559 write 0x009e57 thru 0x00c57f (0x2729 bytes) 000397 1781227785.339427 read 0x03b578 thru 0x03ffff (0x4a88 bytes) 000398 1781227785.344377 mapread 0x01becb thru 0x01d157 (0x128d bytes) 000399 1781227785.345713 mapread 0x0206cc thru 0x022f13 (0x2848 bytes) 000400 1781227785.346753 write 0x0172e6 thru 0x01e87e (0x7599 bytes) 000401 1781227785.348651 trunc from 0x040000 to 0x00232d (0x3dcd2 bytes) 000403 1781227785.379434 mapwrite 0x014756 thru 0x019073 (0x491e bytes) 000406 1781227785.518130 write 0x01bee3 thru 0x01e6ce (0x27ec bytes) 000407 1781227785.521366 mapread 0x01c7bf thru 0x01e6ce (0x1f10 bytes) 000408 1781227785.523583 mapwrite 0x02ce2b thru 0x02f4ac (0x2682 bytes) 000409 1781227785.615522 trunc from 0x02f4ad to 0x010d9c (0x1e710 bytes) 000411 1781227785.635946 trunc from 0x010d9d to 0x012e12 (0x2076 bytes) 000413 1781227785.658584 write 0x01f790 thru 0x02def1 (0xe762 bytes) 000416 1781227785.662110 read 0x00fac2 thru 0x01b127 (0xb666 bytes) 000417 1781227785.669809 read 0x0083f0 thru 0x0177a5 (0xf3b6 bytes) 000418 1781227785.670728 read 0x02679a thru 0x02b5f6 (0x4e5d bytes) 000419 1781227785.671991 write 0x029209 thru 0x030d9f (0x7b97 bytes) 000420 1781227785.674322 trunc from 0x030da0 to 0x021304 (0xfa9b bytes) 000424 1781227785.703639 MIRROR_RESYNC 2 mirrors 000426 1781227786.347280 mapwrite 0x03b64f thru 0x03edf8 (0x37aa bytes) 000427 1781227786.511989 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000428 1781227786.668614 trunc from 0x03edf9 to 0x033a4e (0xb3aa bytes) 000432 1781227786.694577 mapread 0x030f60 thru 0x033a4e (0x2aef bytes) truncating to largest ever: 0x3a516 000434 1781227786.715163 trunc from 0x033a4f to 0x03a515 (0x6ac7 bytes) 000435 1781227786.735726 mapread 0x00ed6c thru 0x019866 (0xaafb bytes) 000436 1781227786.753717 mapread 0x013022 thru 0x01cbbf (0x9b9e bytes) 000439 1781227786.756736 read 0x014643 thru 0x0177a7 (0x3165 bytes) 000440 1781227786.758356 read 0x01a269 thru 0x01e794 (0x452c bytes) 000441 1781227786.760291 mapwrite 0x007164 thru 0x00c357 (0x51f4 bytes) 000443 1781227786.814581 write 0x007319 thru 0x00be6f (0x4b57 bytes) 000445 1781227786.818174 write 0x00fd57 thru 0x011e82 (0x212c bytes) 000446 1781227786.819561 mapread 0x0159c8 thru 0x023a31 (0xe06a bytes) 000447 1781227786.831290 read 0x036a92 thru 0x03a514 (0x3a83 bytes) 000448 1781227786.832137 read 0x027250 thru 0x02f615 (0x83c6 bytes) 000450 1781227786.834153 mapread 0x01bb28 thru 0x02684a (0xad23 bytes) 000451 1781227786.836158 mapread 0x02b034 thru 0x0373ab (0xc378 bytes) 000452 1781227786.838388 read 0x02eaec thru 0x039188 (0xa69d bytes) 000454 1781227786.840112 write 0x03d46c thru 0x03d891 (0x426 bytes) 000457 1781227786.843654 read 0x0384f3 thru 0x03d891 (0x539f bytes) 000458 1781227786.849989 read 0x01a59c thru 0x026ac7 (0xc52c bytes) 000462 1781227786.853537 trunc from 0x03d892 to 0x01455b (0x29336 bytes) 000464 1781227786.879397 write 0x02f91a thru 0x0375d4 (0x7cbb bytes) 000465 1781227786.883014 read 0x02be60 thru 0x0375d4 (0xb775 bytes) 000466 1781227786.891478 mapread 0x001499 thru 0x0051f8 (0x3d60 bytes) 000468 1781227786.893104 trunc from 0x0375d5 to 0x0027c0 (0x34e14 bytes) 000470 1781227786.909195 trunc from 0x0027c1 to 0x00da79 (0xb2b9 bytes) 000471 1781227786.926729 read 0x00c49b thru 0x00da79 (0x15df bytes) 000472 1781227786.934013 mapread 0x00d70e thru 0x00da79 (0x36c bytes) 000475 1781227786.936400 write 0x032903 thru 0x03eb68 (0xc266 bytes) 000476 1781227786.938239 read 0x0319fa thru 0x031af1 (0xf8 bytes) 000477 1781227786.945241 mapwrite 0x033aea thru 0x03ffff (0xc516 bytes) 000478 1781227787.042613 mapread 0x032c53 thru 0x03ffff (0xd3ad bytes) 000479 1781227787.044687 read 0x0175d2 thru 0x01882c (0x125b bytes) 000480 1781227787.045689 mapwrite 0x011476 thru 0x01a72d (0x92b8 bytes) truncating to largest ever: 0x3cc60 000481 1781227787.078071 trunc from 0x040000 to 0x03cc5f (0x33a0 bytes) 000485 1781227787.097531 read 0x01b1f4 thru 0x02362f (0x843c bytes) 000486 1781227787.099310 trunc from 0x03cc60 to 0x00110c (0x3bb53 bytes) 000487 1781227787.112992 write 0x03fa10 thru 0x03ffff (0x5f0 bytes) 000488 1781227787.118157 MIRROR_EXTEND to 2 mirrors 000489 1781227787.832542 mapread 0x012ef4 thru 0x0181b2 (0x52bf bytes) 000491 1781227787.849171 write 0x0060be thru 0x0074a8 (0x13eb bytes) 000492 1781227787.944931 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x34801e8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ef9d8d4 Mirror 3: 0x3ef9d8d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000493 1781227788.740532 mapread 0x028dc7 thru 0x02df14 (0x514e bytes) 000495 1781227788.784362 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x34801e8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ef9d8d4 Mirror 3: 0x3ef9d8d4 Mirror 4: 0x3ef9d8d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000497 1781227789.722276 mapread 0x00605f thru 0x007f31 (0x1ed3 bytes) 000499 1781227789.735282 trunc from 0x040000 to 0x031fc4 (0xe03b bytes) 000500 1781227789.828933 write 0x01d628 thru 0x0233fc (0x5dd5 bytes) 000501 1781227789.840235 trunc from 0x031fc5 to 0x015906 (0x1c6be bytes) 000503 1781227789.869327 write 0x02ea75 thru 0x031eec (0x3478 bytes) 000504 1781227789.872282 mapwrite 0x00be91 thru 0x019305 (0xd475 bytes) 000505 1781227789.973282 read 0x00c80a thru 0x01af40 (0xe737 bytes) 000506 1781227789.974110 read 0x026440 thru 0x02e3b2 (0x7f73 bytes) 000508 1781227790.006184 trunc from 0x031eed to 0x02859f (0x994d bytes) 000509 1781227790.037228 read 0x02737d thru 0x027ff9 (0xc7d bytes) 000510 1781227790.037937 mapread 0x00c6ba thru 0x011ef1 (0x5838 bytes) 000512 1781227790.044405 trunc from 0x0285a0 to 0x00a97e (0x1dc21 bytes) 000513 1781227790.108662 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xa97f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa97f): Mirror 2: 0xc936b720 (stale) Mirror 3: 0x84dc961a (stale) Mirror 4: 0x84dc961a (stale) CRC-32 checksum value for chunk [0, 0xa97f): Mirror 1: 0x84dc961a Mirror 5: 0x84dc961a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa97f: skipped 000514 1781227791.158456 mapwrite 0x03c1a4 thru 0x03ffff (0x3e5c bytes) 000516 1781227791.460024 read 0x006a60 thru 0x00b89a (0x4e3b bytes) 000517 1781227791.475842 write 0x03ea7a thru 0x03f61c (0xba3 bytes) 000518 1781227791.480187 mapwrite 0x00e095 thru 0x01ae7f (0xcdeb bytes) 000521 1781227791.620085 read 0x021fbe thru 0x02599f (0x39e2 bytes) 000522 1781227791.636830 mapread 0x02b64f thru 0x033d03 (0x86b5 bytes) 000523 1781227791.646839 mapread 0x03d331 thru 0x03dcbd (0x98d bytes) 000524 1781227791.651367 write 0x013b00 thru 0x01802a (0x452b bytes) 000525 1781227791.660043 mapwrite 0x0355e9 thru 0x03aaf0 (0x5508 bytes) 000528 1781227791.789360 mapwrite 0x01a0ce thru 0x025759 (0xb68c bytes) 000530 1781227791.882476 write 0x02c561 thru 0x0357dd (0x927d bytes) 000532 1781227791.890141 mapwrite 0x035dbd thru 0x03fae7 (0x9d2b bytes) 000535 1781227791.958278 mapwrite 0x035cab thru 0x03ffff (0xa355 bytes) 000536 1781227792.016885 read 0x016561 thru 0x017d5b (0x17fb bytes) 000542 1781227792.029687 mapread 0x02bf05 thru 0x037d99 (0xbe95 bytes) 000545 1781227792.036371 write 0x012287 thru 0x01c073 (0x9ded bytes) 000547 1781227792.039544 write 0x026d4f thru 0x0297b7 (0x2a69 bytes) 000548 1781227792.041265 mapwrite 0x02bb95 thru 0x0389e4 (0xce50 bytes) 000549 1781227792.187071 write 0x011cec thru 0x01bb1a (0x9e2f bytes) 000557 1781227792.203913 trunc from 0x040000 to 0x019a80 (0x2657f bytes) 000559 1781227792.256310 mapwrite 0x034e6e thru 0x039799 (0x492c bytes) 000561 1781227792.398310 mapwrite 0x036f6c thru 0x03fff2 (0x9087 bytes) 000562 1781227792.531243 mapwrite 0x00193b thru 0x00bdfb (0xa4c1 bytes) 000563 1781227792.601077 read 0x0296fb thru 0x036505 (0xce0b bytes) 000565 1781227792.602334 mapwrite 0x001ac1 thru 0x00e459 (0xc999 bytes) 000571 1781227792.646709 write 0x0337cd thru 0x03a0b0 (0x68e4 bytes) 000573 1781227792.649067 read 0x005e92 thru 0x0089db (0x2b4a bytes) 000574 1781227792.649981 read 0x0226d6 thru 0x02eaad (0xc3d8 bytes) 000576 1781227792.651586 mapwrite 0x03eda7 thru 0x03ffff (0x1259 bytes) 000578 1781227792.704297 mapwrite 0x00ef82 thru 0x0112c6 (0x2345 bytes) 000579 1781227792.736724 read 0x01457a thru 0x0176bb (0x3142 bytes) 000580 1781227792.737525 mapread 0x006572 thru 0x00fa7c (0x950b bytes) 000581 1781227792.739401 trunc from 0x040000 to 0x015502 (0x2aafd bytes) 000583 1781227792.770485 write 0x0356de thru 0x03bff4 (0x6917 bytes) 000585 1781227792.777213 mapwrite 0x00d809 thru 0x013d7d (0x6575 bytes) 000586 1781227792.848651 mapread 0x015fc6 thru 0x0212bb (0xb2f6 bytes) 000587 1781227792.869315 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bff5): Mirror 2: 0xcf68f5a (stale) Mirror 3: 0xbb802bf4 (stale) Mirror 4: 0xbb802bf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bff5): Mirror 2: 0xcf68f5a (stale) Mirror 3: 0xbb802bf4 (stale) Mirror 4: 0xbb802bf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bff5: skipped 000588 1781227793.291166 write 0x026dd4 thru 0x028345 (0x1572 bytes) 000589 1781227793.303279 write 0x00ccea thru 0x016db6 (0xa0cd bytes) 000590 1781227793.327996 mapwrite 0x007517 thru 0x0076d0 (0x1ba bytes) 000592 1781227793.371981 mapread 0x03012f thru 0x03bff4 (0xbec6 bytes) 000593 1781227793.409130 mapwrite 0x013729 thru 0x018393 (0x4c6b bytes) 000594 1781227793.530239 MIRROR_RESYNC 4 mirrors 000597 1781227794.905170 read 0x00d467 thru 0x01254a (0x50e4 bytes) 000598 1781227794.914349 read 0x039731 thru 0x03bff4 (0x28c4 bytes) 000599 1781227794.924819 read 0x006aea thru 0x014532 (0xda49 bytes) 000600 1781227794.925305 mapread 0x039642 thru 0x03bff4 (0x29b3 bytes) 000602 1781227794.926377 mapread 0x0335e4 thru 0x03a3f2 (0x6e0f bytes) 000604 1781227794.929239 write 0x01c785 thru 0x0265d2 (0x9e4e bytes) 000607 1781227795.017604 mapread 0x03a658 thru 0x03b8ed (0x1296 bytes) 000608 1781227795.019392 read 0x0015b2 thru 0x001e63 (0x8b2 bytes) 000609 1781227795.020244 mapwrite 0x02578f thru 0x028a0f (0x3281 bytes) 000614 1781227795.068380 mapread 0x0152b6 thru 0x016fdf (0x1d2a bytes) 000615 1781227795.070268 write 0x015596 thru 0x01d0d2 (0x7b3d bytes) 000617 1781227795.072848 mapread 0x00089b thru 0x00e67f (0xdde5 bytes) 000618 1781227795.075457 read 0x02f737 thru 0x03510a (0x59d4 bytes) 000620 1781227795.077383 mapread 0x004507 thru 0x006786 (0x2280 bytes) 000621 1781227795.079436 mapread 0x0319de thru 0x037d77 (0x639a bytes) 000623 1781227795.082212 mapwrite 0x03601a thru 0x03ffff (0x9fe6 bytes) 000628 1781227795.157153 write 0x02c98e thru 0x03a207 (0xd87a bytes) 000631 1781227795.159771 mapwrite 0x03d49e thru 0x03fbd9 (0x273c bytes) 000632 1781227795.196192 write 0x003553 thru 0x008eff (0x59ad bytes) 000634 1781227795.203532 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52b346c7 (stale) Mirror 3: 0x52b346c7 (stale) Mirror 4: 0x52b346c7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef0dc840 Mirror 5: 0xef0dc840 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1781227795.850283 trunc from 0x040000 to 0x00cacb (0x33534 bytes) 000636 1781227795.991599 mapwrite 0x026519 thru 0x02b5e6 (0x50ce bytes) 000637 1781227796.087733 mapwrite 0x021242 thru 0x024401 (0x31c0 bytes) 000638 1781227796.138768 mapread 0x020e02 thru 0x02b5e6 (0xa7e5 bytes) 000640 1781227796.163324 trunc from 0x02b5e7 to 0x002151 (0x29495 bytes) 000643 1781227796.187003 read 0x000f12 thru 0x002151 (0x1240 bytes) 000644 1781227796.193468 read 0x000c35 thru 0x002151 (0x151d bytes) 000645 1781227796.194551 mapwrite 0x013798 thru 0x020f22 (0xd78b bytes) 000646 1781227796.267326 write 0x038e68 thru 0x03ffff (0x7198 bytes) 000648 1781227796.276632 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52b346c7 (stale) Mirror 3: 0x52b346c7 (stale) Mirror 4: 0x52b346c7 (stale) Mirror 5: 0xef0dc840 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x555d0854 Mirror 6: 0x555d0854 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000649 1781227796.996548 read 0x017af9 thru 0x01f7c7 (0x7ccf bytes) 000650 1781227797.008139 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x52b346c7 (stale) Mirror 4: 0x52b346c7 (stale) Mirror 5: 0xef0dc840 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x52b346c7 (stale) Mirror 4: 0x52b346c7 (stale) Mirror 5: 0xef0dc840 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x555d0854 Mirror 6: 0x555d0854 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000651 1781227797.557162 mapread 0x02e14f thru 0x031a04 (0x38b6 bytes) 000653 1781227797.577813 trunc from 0x040000 to 0x03c74f (0x38b0 bytes) 000654 1781227797.707049 read 0x0271b1 thru 0x02ebdb (0x7a2b bytes) 000655 1781227797.720391 write 0x02b81d thru 0x03a6b2 (0xee96 bytes) 000656 1781227797.732026 MIRROR_RESYNC 5 mirrors 000658 1781227798.643748 mapwrite 0x031311 thru 0x0384ff (0x71ef bytes) 000660 1781227798.831221 read 0x00248f thru 0x006156 (0x3cc8 bytes) 000661 1781227798.841884 mapwrite 0x00c759 thru 0x01a8d5 (0xe17d bytes) 000662 1781227798.918975 read 0x0072de thru 0x0116fc (0xa41f bytes) 000668 1781227798.927772 mapwrite 0x01c215 thru 0x024d44 (0x8b30 bytes) 000671 1781227799.001860 trunc from 0x03c750 to 0x00911a (0x33635 bytes) 000673 1781227799.026939 write 0x025833 thru 0x0337b6 (0xdf84 bytes) 000677 1781227799.033402 write 0x03bf2d thru 0x03ffff (0x40d3 bytes) 000678 1781227799.042014 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x23007cee (stale) Mirror 5: 0x23007cee (stale) Mirror 6: 0x23007cee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x23007cee (stale) Mirror 5: 0x23007cee (stale) Mirror 6: 0x23007cee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000679 1781227799.636234 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x23007cee (stale) Mirror 6: 0x23007cee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x23007cee (stale) Mirror 6: 0x23007cee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000680 1781227800.148585 read 0x03d289 thru 0x03ffff (0x2d77 bytes) 000681 1781227800.156248 mapread 0x02836b thru 0x0299dd (0x1673 bytes) 000682 1781227800.170313 read 0x01fc7f thru 0x02be8e (0xc210 bytes) 000683 1781227800.181409 mapread 0x03d41b thru 0x03ffff (0x2be5 bytes) 000686 1781227800.183676 write 0x03f7d7 thru 0x03ffff (0x829 bytes) 000689 1781227800.192418 write 0x012002 thru 0x015aef (0x3aee bytes) truncating to largest ever: 0x3f154 000690 1781227800.196382 trunc from 0x040000 to 0x03f153 (0xeac bytes) 000692 1781227800.216054 mapwrite 0x02548b thru 0x02d4a0 (0x8016 bytes) 000694 1781227800.268722 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f154): Mirror 6: 0x3daa566e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f154): Mirror 6: 0x3daa566e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f154: skipped 000696 1781227800.553547 mapread 0x010e7d thru 0x0148ff (0x3a83 bytes) 000698 1781227800.580455 mapread 0x0047ed thru 0x00f545 (0xad59 bytes) 000700 1781227800.590149 mapread 0x027f1b thru 0x035a60 (0xdb46 bytes) 000701 1781227800.603946 write 0x013e86 thru 0x0178b5 (0x3a30 bytes) 000704 1781227800.612757 trunc from 0x03f154 to 0x033d08 (0xb44b bytes) 000706 1781227800.633485 mapwrite 0x01fd5d thru 0x028826 (0x8aca bytes) 000708 1781227800.680406 read 0x025db2 thru 0x02bf9e (0x61ed bytes) 000710 1781227800.682498 mapwrite 0x015079 thru 0x018b0d (0x3a95 bytes) 000712 1781227800.716523 read 0x0128d3 thru 0x017651 (0x4d7f bytes) 000713 1781227800.717225 read 0x007aad thru 0x00d064 (0x55b8 bytes) 000714 1781227800.722095 MIRROR_RESYNC 2 mirrors 000715 1781227801.280732 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000716 1781227801.524529 MIRROR_EXTEND to 2 mirrors 000718 1781227802.294179 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000720 1781227802.432992 trunc from 0x033d09 to 0x037ee5 (0x41dd bytes) 000721 1781227802.467848 mapread 0x02c332 thru 0x034e0c (0x8adb bytes) 000723 1781227802.504443 trunc from 0x037ee6 to 0x02434a (0x13b9b bytes) 000728 1781227802.541703 mapread 0x000ab1 thru 0x008157 (0x76a7 bytes) 000729 1781227802.556349 mapwrite 0x03d62b thru 0x03ffff (0x29d5 bytes) 000731 1781227802.659371 trunc from 0x040000 to 0x0293ca (0x16c35 bytes) 000737 1781227802.693744 trunc from 0x0293cb to 0x02a8d7 (0x150d bytes) 000739 1781227802.717979 mapwrite 0x031319 thru 0x03b34c (0xa034 bytes) 000740 1781227802.811715 read 0x0293e9 thru 0x02ad0a (0x1922 bytes) 000741 1781227802.816904 read 0x03454f thru 0x03b34c (0x6dfe bytes) 000742 1781227802.817550 mapread 0x014970 thru 0x0151ad (0x83e bytes) 000745 1781227802.845055 mapread 0x034441 thru 0x03b34c (0x6f0c bytes) 000747 1781227802.849089 trunc from 0x03b34d to 0x032b9c (0x87b0 bytes) 000748 1781227802.865951 mapwrite 0x037b62 thru 0x03f47b (0x791a bytes) 000749 1781227802.974120 mapread 0x0325b1 thru 0x035e30 (0x3880 bytes) 000758 1781227803.006094 mapwrite 0x023d3b thru 0x024b5e (0xe24 bytes) 000759 1781227803.066345 mapwrite 0x03ee4c thru 0x03ffff (0x11b4 bytes) 000766 1781227803.156835 trunc from 0x040000 to 0x00cfae (0x33051 bytes) 000767 1781227803.178864 mapread 0x000858 thru 0x007848 (0x6ff1 bytes) 000768 1781227803.185338 write 0x0067c4 thru 0x00ebea (0x8427 bytes) 000770 1781227803.194354 mapread 0x000536 thru 0x00ebea (0xe6b5 bytes) 000771 1781227803.199175 read 0x004469 thru 0x00ebea (0xa782 bytes) 000772 1781227803.199964 mapread 0x000531 thru 0x00294c (0x241c bytes) 000773 1781227803.204259 mapread 0x008332 thru 0x00bbaf (0x387e bytes) 000775 1781227803.209777 read 0x00c77b thru 0x00ebea (0x2470 bytes) 000780 1781227803.215351 mapwrite 0x00211b thru 0x011bf0 (0xfad6 bytes) 000782 1781227803.336937 read 0x00bde7 thru 0x011bf0 (0x5e0a bytes) 000783 1781227803.340078 read 0x0098f7 thru 0x0102eb (0x69f5 bytes) 000784 1781227803.345716 mapread 0x0086d5 thru 0x008a08 (0x334 bytes) 000785 1781227803.349224 read 0x008136 thru 0x011071 (0x8f3c bytes) 000787 1781227803.354834 mapread 0x001f8e thru 0x007aea (0x5b5d bytes) 000790 1781227803.360026 mapwrite 0x0041ba thru 0x0104d7 (0xc31e bytes) 000796 1781227803.462208 read 0x00d8fa thru 0x011bf0 (0x42f7 bytes) 000797 1781227803.463629 mapwrite 0x03e712 thru 0x03ffff (0x18ee bytes) 000799 1781227803.561029 trunc from 0x040000 to 0x0394d6 (0x6b29 bytes) 000800 1781227803.577313 mapwrite 0x01e5f4 thru 0x0289c1 (0xa3ce bytes) 000802 1781227803.629707 mapwrite 0x03bb0b thru 0x03ffff (0x44f5 bytes) 000803 1781227803.724137 mapwrite 0x00d586 thru 0x00db0b (0x586 bytes) 000805 1781227803.813159 read 0x032694 thru 0x0356ff (0x306c bytes) 000807 1781227803.814516 mapread 0x03d460 thru 0x03ffff (0x2ba0 bytes) 000808 1781227803.818050 mapwrite 0x014a78 thru 0x019a3e (0x4fc7 bytes) 000809 1781227803.906072 mapwrite 0x01fb65 thru 0x02edf3 (0xf28f bytes) 000812 1781227804.006252 MIRROR_EXTEND to 2 mirrors 000813 1781227804.815397 write 0x019887 thru 0x01f988 (0x6102 bytes) 000815 1781227804.918593 MIRROR_RESYNC 2 mirrors 000816 1781227805.655934 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000822 1781227805.826300 mapwrite 0x03da26 thru 0x03ffff (0x25da bytes) 000823 1781227805.960708 trunc from 0x040000 to 0x00e982 (0x3167d bytes) 000824 1781227806.023243 MIRROR_EXTEND to 2 mirrors 000826 1781227806.857595 mapread 0x00497b thru 0x00e982 (0xa008 bytes) 000828 1781227806.895944 mapwrite 0x030e39 thru 0x039fcf (0x9197 bytes) 000831 1781227807.117759 write 0x03d37c thru 0x03f32c (0x1fb1 bytes) 000832 1781227807.120936 mapwrite 0x00a340 thru 0x00f721 (0x53e2 bytes) 000835 1781227807.193357 trunc from 0x03f32d to 0x0186f0 (0x26c3c bytes) 000836 1781227807.224413 trunc from 0x0186f1 to 0x01de10 (0x5720 bytes) 000837 1781227807.245756 trunc from 0x01de11 to 0x01a59e (0x3872 bytes) 000838 1781227807.265971 mapread 0x007b2a thru 0x014159 (0xc630 bytes) 000839 1781227807.283623 write 0x01c21b thru 0x0230f2 (0x6ed8 bytes) 000840 1781227807.286895 mapwrite 0x00d353 thru 0x0141ec (0x6e9a bytes) 000843 1781227807.363079 read 0x007a57 thru 0x00a37c (0x2926 bytes) 000844 1781227807.366268 mapwrite 0x0105b9 thru 0x0134f8 (0x2f40 bytes) 000845 1781227807.413973 write 0x03a992 thru 0x03f79b (0x4e0a bytes) 000846 1781227807.415310 mapwrite 0x033cc3 thru 0x03ffff (0xc33d bytes) 000847 1781227807.525713 mapread 0x02edc1 thru 0x03b7c8 (0xca08 bytes) 000849 1781227807.541276 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000851 1781227807.759659 mapwrite 0x01839c thru 0x023b0b (0xb770 bytes) 000853 1781227807.919746 mapwrite 0x031ce2 thru 0x034427 (0x2746 bytes) 000854 1781227808.020275 write 0x02485a thru 0x0331c3 (0xe96a bytes) 000857 1781227808.022811 trunc from 0x040000 to 0x035f57 (0xa0a8 bytes) 000858 1781227808.038639 mapread 0x01b74c thru 0x0273aa (0xbc5f bytes) 000859 1781227808.045440 MIRROR_EXTEND to 2 mirrors 000861 1781227808.787625 MIRROR_EXTEND to 3 mirrors 000864 1781227809.906710 trunc from 0x035f58 to 0x016631 (0x1f926 bytes) 000866 1781227810.107082 mapread 0x00713d thru 0x01123a (0xa0fe bytes) 000867 1781227810.147746 read 0x00b0b2 thru 0x00bac3 (0xa12 bytes) 000868 1781227810.149687 mapread 0x004c8b thru 0x00e3ad (0x9723 bytes) 000869 1781227810.158642 mapwrite 0x036174 thru 0x03730f (0x119c bytes) 000870 1781227810.250321 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: 0x5bd711d5 (stale) Mirror 3: 0x5bd711d5 (stale) CRC-32 checksum value for chunk [0, 0x37310): Mirror 1: 0x9d622b03 Mirror 4: 0x9d622b03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37310: skipped 000871 1781227811.310888 read 0x018301 thru 0x020ce7 (0x89e7 bytes) 000872 1781227811.332007 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: 0x5bd711d5 (stale) Mirror 3: 0x5bd711d5 (stale) CRC-32 checksum value for chunk [0, 0x37310): Mirror 1: 0x9d622b03 Mirror 4: 0x9d622b03 Mirror 5: 0x9d622b03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37310: skipped 000874 1781227812.664536 write 0x013f97 thru 0x01ceea (0x8f54 bytes) 000877 1781227812.855149 mapwrite 0x02c317 thru 0x035a02 (0x96ec bytes) 000879 1781227813.004916 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: 0x5bd711d5 (stale) Mirror 3: 0x5bd711d5 (stale) Mirror 4: 0x9d622b03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: 0x5bd711d5 (stale) Mirror 3: 0x5bd711d5 (stale) Mirror 4: 0x9d622b03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37310: skipped 000880 1781227813.391100 mapwrite 0x02d02d thru 0x02e27d (0x1251 bytes) 000881 1781227813.502256 mapread 0x01ff27 thru 0x027d11 (0x7deb bytes) 000882 1781227813.524399 mapread 0x01ad4e thru 0x01defc (0x31af bytes) 000884 1781227813.531452 mapwrite 0x0262b1 thru 0x02d833 (0x7583 bytes) 000887 1781227813.692993 write 0x00bb48 thru 0x014e7d (0x9336 bytes) 000888 1781227813.717491 write 0x01cc15 thru 0x020afc (0x3ee8 bytes) 000890 1781227813.733048 MIRROR_RESYNC 4 mirrors 000891 1781227814.940637 write 0x0179bc thru 0x02417d (0xc7c2 bytes) 000892 1781227815.026887 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37310): Mirror 2: 0x8d69c1d3 (stale) Mirror 3: 0x8d69c1d3 (stale) Mirror 4: 0x8d69c1d3 (stale) CRC-32 checksum value for chunk [0, 0x37310): Mirror 1: 0xa48914dd Mirror 5: 0xa48914dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37310: skipped 000893 1781227816.118582 trunc from 0x037310 to 0x015b3f (0x217d0 bytes) 000894 1781227816.234043 write 0x027e9d thru 0x033ae2 (0xbc46 bytes) 000898 1781227816.241107 trunc from 0x033ae3 to 0x03a1a3 (0x66c1 bytes) 000899 1781227816.271556 read 0x033bc6 thru 0x03a1a3 (0x65de bytes) 000900 1781227816.283079 mapwrite 0x023139 thru 0x032c95 (0xfb5d bytes) 000903 1781227816.401983 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1a4): Mirror 2: 0x7821a963 (stale) Mirror 3: 0x7821a963 (stale) Mirror 4: 0x7821a963 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1a4): Mirror 2: 0x7821a963 (stale) Mirror 3: 0x7821a963 (stale) Mirror 4: 0x7821a963 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1a4: skipped 000905 1781227817.033632 mapread 0x034b2a thru 0x03a1a3 (0x567a bytes) 000906 1781227817.063400 read 0x02b7da thru 0x035a83 (0xa2aa bytes) 000907 1781227817.075266 write 0x029778 thru 0x032186 (0x8a0f bytes) 000908 1781227817.092115 mapwrite 0x017649 thru 0x01d550 (0x5f08 bytes) 000909 1781227817.176817 write 0x016417 thru 0x01ddb6 (0x79a0 bytes) 000910 1781227817.178083 trunc from 0x03a1a4 to 0x02e844 (0xb95f bytes) 000911 1781227817.202557 read 0x02bb34 thru 0x02e844 (0x2d11 bytes) 000913 1781227817.208570 write 0x03e26d thru 0x03ead0 (0x864 bytes) 000915 1781227817.215949 mapread 0x017b02 thru 0x01b1ab (0x36aa bytes) 000916 1781227817.218088 read 0x01fc8f thru 0x02297f (0x2cf1 bytes) 000918 1781227817.234256 write 0x00c8ce thru 0x00d3a3 (0xad6 bytes) 000919 1781227817.241185 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 2: 0xe144a0ac (stale) Mirror 3: 0xe144a0ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 2: 0xe144a0ac (stale) Mirror 3: 0xe144a0ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ead1: skipped 000920 1781227817.719822 mapread 0x03a75e thru 0x03ead0 (0x4373 bytes) 000921 1781227817.752681 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 3: 0xe144a0ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 3: 0xe144a0ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ead1: skipped 000923 1781227818.245445 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 3: 0xe144a0ac (stale) CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 1: 0x1f09f1ce Mirror 4: 0x1f09f1ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ead1: skipped 000924 1781227819.204610 read 0x01f15c thru 0x029611 (0xa4b6 bytes) 000926 1781227819.210681 write 0x00a866 thru 0x012e99 (0x8634 bytes) 000928 1781227819.293364 mapread 0x0110a7 thru 0x016d9e (0x5cf8 bytes) 000930 1781227819.305454 write 0x0056e0 thru 0x00ccde (0x75ff bytes) 000931 1781227819.319752 mapread 0x003f61 thru 0x00ea39 (0xaad9 bytes) 000934 1781227819.324326 mapread 0x01d371 thru 0x022df7 (0x5a87 bytes) 000936 1781227819.336415 write 0x031ba3 thru 0x0334f8 (0x1956 bytes) 000937 1781227819.349411 mapwrite 0x00fb66 thru 0x018630 (0x8acb bytes) 000938 1781227819.427097 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 3: 0xe144a0ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ead1): Mirror 3: 0xe144a0ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ead1: skipped 000939 1781227819.711674 write 0x009022 thru 0x017f1d (0xeefc bytes) 000942 1781227819.728670 mapwrite 0x0302dd thru 0x03c703 (0xc427 bytes) 000944 1781227819.832892 read 0x03cbbf thru 0x03d951 (0xd93 bytes) 000947 1781227819.852676 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000951 1781227820.087813 mapwrite 0x02d299 thru 0x03a0e5 (0xce4d bytes) 000953 1781227820.297526 mapwrite 0x00114c thru 0x003f10 (0x2dc5 bytes) 000956 1781227820.400871 write 0x01e614 thru 0x0229b8 (0x43a5 bytes) 000959 1781227820.420901 trunc from 0x03ead1 to 0x006000 (0x38ad0 bytes) 000961 1781227820.465531 write 0x01e8dd thru 0x028c47 (0xa36b bytes) truncating to largest ever: 0x3fc3a 000962 1781227820.470191 trunc from 0x028c48 to 0x03fc39 (0x16ff2 bytes) 000964 1781227820.504915 mapwrite 0x02e927 thru 0x03167b (0x2d55 bytes) 000967 1781227820.579526 trunc from 0x03fc3a to 0x02b275 (0x149c4 bytes) 000969 1781227820.600870 MIRROR_EXTEND to 2 mirrors 000971 1781227821.422432 mapread 0x00274d thru 0x00caf6 (0xa3aa bytes) 000972 1781227821.452674 read 0x01c098 thru 0x020ef4 (0x4e5d bytes) 000973 1781227821.468555 mapread 0x0092b8 thru 0x00b292 (0x1fdb bytes) 000977 1781227821.482585 write 0x0259e8 thru 0x030203 (0xa81c bytes) 000980 1781227821.643670 mapread 0x00fd5d thru 0x015e69 (0x610d bytes) 000981 1781227821.666126 read 0x0275f3 thru 0x030203 (0x8c11 bytes) 000982 1781227821.667098 read 0x02b367 thru 0x030203 (0x4e9d bytes) 000985 1781227821.676275 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000986 1781227821.847884 MIRROR_EXTEND to 2 mirrors 000987 1781227822.674556 MIRROR_RESYNC 2 mirrors 000990 1781227823.126951 read 0x007be9 thru 0x0081cf (0x5e7 bytes) 000993 1781227823.136993 write 0x02fce1 thru 0x036451 (0x6771 bytes) 000994 1781227823.255738 MIRROR_RESYNC 2 mirrors 000995 1781227823.848137 read 0x007d4f thru 0x0087eb (0xa9d bytes) 000996 1781227823.864391 mapread 0x0064f5 thru 0x010602 (0xa10e bytes) 000997 1781227823.906239 read 0x016e59 thru 0x022db1 (0xbf59 bytes) 000998 1781227823.920433 read 0x004b06 thru 0x00f86b (0xad66 bytes) 000999 1781227823.922026 read 0x0132a6 thru 0x01f003 (0xbd5e bytes) 001000 1781227823.922609 trunc from 0x036452 to 0x028e12 (0xd63f bytes) All operations completed A-OK! PASS 70a (75s) == sanity-flr test 71: check mirror extend parent fid ==== 21:30:30 (1781227830) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 21:30:35 (1781227835) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225435 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212227 s, 4.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 21:30:46 (1781227846) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8687968 .. Extending file size to 3230848 .. Extending file size to 5308928 .. Extending file size to 6842048 .. Extending file size to 8633280 .. Extending file size to 9155712 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57206 57207 57208 57210 57211 failed 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) 192.168.201.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:32:12 (1781227932) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 650880 .. Extending file size to 7455456 .. done Extending file size to 1185600 .. Extending file size to 8167264 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9235648 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4076512 .. Extending file size to 8427424 .. Extending file size to 9356480 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9161600 .. Extending file size to 9293472 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58971 58972 58973 58976 58979 failed 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) 192.168.201.139@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:33:33 (1781228013) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202826 s, 10.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.414975 s, 5.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xe2a8525b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:33:54 (1781228034) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142957 s, 14.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 21:34:02 (1781228042) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ce:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:34:09 (1781228049) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14654 s, 14.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:34:17 (1781228057) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193815 s, 10.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:34:25 (1781228065) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.226966 s, 9.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:35:00 (1781228100) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg139-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240461 s, 4.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:36:00 (1781228160) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg139-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.352481 s, 5.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:37:04 (1781228224) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg139-server mds-ost sync done. Creating new pool test_204f oleg139-server: Pool lustre.test_204f created Adding targets to pool oleg139-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg139-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219313 s, 9.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg139-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg139-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg139-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:38:24 (1781228304) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:38:34 (1781228314) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:38:44 (1781228324) oleg139-server: Pool lustre.test_206 created Adding targets to pool oleg139-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg139-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg139-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg139-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg139-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:39:09 (1781228349) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753818 s, 13.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245337 s, 42.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc493cc8b Mirror 3: 0xc493cc8b PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:39:20 (1781228360) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:39:23 (1781228363) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:39:26 (1781228366) Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.100465 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1aa0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1aa0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1aa0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1e20000 Read this (4 bytes): BBBB 192.168.201.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg139-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:40:04 (1781228404) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0317377 s, 258 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:40:13 (1781228413) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564462 s, 18.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f3 lmm_fid: [0x200000404:0x1f3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834419 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 21:40:15.000000000 -0400 Modify: 2026-06-11 21:40:15.000000000 -0400 Change: 2026-06-11 21:40:17.000000000 -0400 Birth: 2026-06-11 21:40:15.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:40:22 (1781228422) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:40:26 (1781228426) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.641932 s, 63.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0472075 s, 86.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 21:40 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 21:40 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:40:35 (1781228435) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:40:44 (1781228444) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1863 sec =========== 21:40:54 (1781228454) === sanity-flr: start cleanup 21:40:55 (1781228455) === === sanity-flr: finish cleanup 21:41:02 (1781228462) ===