-----============= acceptance-small: sanity-flr ============----- Tue Aug 18 20:08:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg412-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:09:02 (1787098142) === oleg412-client.virtnet: executing check_config_client /mnt/lustre oleg412-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg412-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93850a34c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93850a34c000.idle_timeout=debug disable quota as required oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:09:32 (1787098172) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7658] == sanity-flr test 0a: lfs mirror create with -N option == 20:09:34 (1787098174) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:09:48 (1787098188) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:09:51 (1787098191) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:09:53 (1787098193) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:10:14 (1787098214) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:10:17 (1787098217) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:10:20 (1787098220) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93850a34c000.stats=clear osc.lustre-OST0001-osc-ffff93850a34c000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:10:27 (1787098227) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:10:37 (1787098237) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:10:45 (1787098245) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:10:48 (1787098248) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:10:55 (1787098255) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 20:11:04 (1787098264) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:11:10 (1787098270) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:11:12 (1787098272) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:11:18 (1787098278) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:11:27 (1787098287) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:11:34 (1787098294) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.99544 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864934 s, 12.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:11:44 (1787098304) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.229358 s, 32.0 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:11:51 (1787098311) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:12:00 (1787098320) Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg412-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (123s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:14:03 (1787098443) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.208863 s, 20.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (35s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:14:39 (1787098479) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:15:55 (1787098555) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.56146 s, 11.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 18 20:16 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.954s user 0m0.000s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93850a34c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.991s user 0m0.001s sys 0m0.164s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.146s user 0m0.000s sys 0m0.196s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg412-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93850a34c000.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:17:25 (1787098645) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:17:28 (1787098648) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:17:31 (1787098651) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 20:17:34 (1787098654) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 20:17:41 (1787098661) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.611874 s, 20.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:17:54 (1787098674) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258556 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290573 s, 36.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:18:03 (1787098683) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:18:11 (1787098691) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000771162 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00319432 s, 321 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000967598 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110098 s, 930 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 20:18:24 (1787098704) create test files with size 15756038 .. 7cef2be915e5d27a172de301c4df8da2 - f00446fff95899582f652c90f8d68f62 - 2f78ec23e7ef8ae81387eed5b8cf005a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 20:18:49 (1787098729) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:19:05 (1787098745) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:19:12 (1787098752) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114809 s, 9.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498809 s, 21.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 20:19:21 (1787098761) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122939 s, 17.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.467825 s, 9.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.307958 s, 6.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 20:19:39 (1787098779) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 20:19:41 (1787098781) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:19:44 (1787098784) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645532 s, 16.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 20:19:53 (1787098793) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:19:56 (1787098796) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:20:07 (1787098807) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.337962 s, 31.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:20:18 (1787098818) llite.lustre-ffff93850959b000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.278547 s, 37.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 12302 flags: 1 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 12302 flags: 1 llite.lustre-ffff9385034cc000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:20:31 (1787098831) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:21:09 (1787098869) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:21:43 (1787098903) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 20:21:46 (1787098906) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.181343 s, 17.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140876 s, 22.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xaaeeab50 (nosync,stale) Mirror 2: 0xaaeeab50 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 93b2e3c15a896cadae15d1cf59894b89 - mirror 1 checksum: 8896b19bd6e4687777cae46adcefbe48 - mirror 2 checksum: 93b2e3c15a896cadae15d1cf59894b89 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-18T20:21:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-18T20:21:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 93b2e3c15a896cadae15d1cf59894b89 - mirror 1 checksum: 8896b19bd6e4687777cae46adcefbe48 - mirror 2 checksum: 8896b19bd6e4687777cae46adcefbe48 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-18T20:21:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-18T20:21:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:21:57 (1787098917) Failing mds1 on oleg412-server Stopping /mnt/lustre-mds1 (opts:) on oleg412-server 20:22:08 (1787098928) shut down facet: mds1 facet_host: oleg412-server facet_failover_host: oleg412-server Failover mds1 to oleg412-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 20:22:40 (1787098960) targets are mounted 20:22:40 (1787098960) facet_failover done oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353702 s, 29.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:22:57 (1787098977) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:23:00 (1787098980) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:23:03 (1787098983) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00862818 s, 1.9 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:23:10 (1787098990) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:23:13 (1787098993) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0163766 s, 250 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0816232 s, 50.2 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:23:20 (1787099000) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:23:41 (1787099021) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:24:09 (1787099049) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:24:32 (1787099072) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:24:55 (1787099095) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 20:25:02 (1787099102) mirror create pid 54350 mirror split pid 54352 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54350 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54352 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 20:26:08 (1787099168) Seed set to 9170 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787099170.991634 MIRROR_EXTEND to 2 mirrors skipping zero size read truncating to largest ever: 0x2aef5 000005 1787099171.420974 trunc from 00000000 to 0x02aef4 (0x2aef5 bytes) 000006 1787099171.558283 mapread 0x018f3a thru 0x024263 (0xb32a bytes) 000009 1787099171.633193 mapread 0x00646b thru 0x0105e7 (0xa17d bytes) 000010 1787099171.657403 mapread 0x0046e7 thru 0x00c6b0 (0x7fca bytes) 000011 1787099171.666311 read 0x01617c thru 0x01b57b (0x5400 bytes) 000012 1787099171.667117 write 0x010545 thru 0x01cda1 (0xc85d bytes) 000015 1787099171.693724 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2aef5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aef5): Mirror 2: 0x3d3fb8d5 (stale) CRC-32 checksum value for chunk [0, 0x2aef5): Mirror 1: 0x19619dd1 Mirror 3: 0x19619dd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aef5: skipped 000017 1787099172.728715 write 0x033ac4 thru 0x039c3a (0x6177 bytes) 000018 1787099172.798922 write 0x0365d5 thru 0x0399d5 (0x3401 bytes) 000021 1787099172.802757 mapread 0x02ef9a thru 0x03178f (0x27f6 bytes) 000022 1787099172.822215 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39c3b): Mirror 2: 0xad6f7771 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c3b): Mirror 2: 0xad6f7771 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c3b: skipped 000023 1787099173.108903 mapread 0x01675a thru 0x01aaa5 (0x434c bytes) 000024 1787099173.160873 mapwrite 0x03ff3a thru 0x03ffff (0xc6 bytes) 000025 1787099173.239090 read 0x0204b9 thru 0x024f36 (0x4a7e bytes) 000026 1787099173.246765 write 0x03bdfc thru 0x03ffff (0x4204 bytes) 000027 1787099173.247917 write 0x01ccbe thru 0x028e5f (0xc1a2 bytes) 000028 1787099173.263204 write 0x011a7d thru 0x012995 (0xf19 bytes) 000031 1787099173.267342 mapwrite 0x034c0f thru 0x03c76b (0x7b5d bytes) 000033 1787099173.354254 mapwrite 0x036f3f thru 0x03ffff (0x90c1 bytes) 000034 1787099173.419869 mapwrite 0x003e2f thru 0x013d8b (0xff5d bytes) 000035 1787099173.516675 write 0x00d1e1 thru 0x018a69 (0xb889 bytes) 000037 1787099173.527767 mapwrite 0x031d49 thru 0x03404f (0x2307 bytes) 000038 1787099173.578680 mapread 0x008c31 thru 0x011ad7 (0x8ea7 bytes) 000039 1787099173.580170 read 0x003b9d thru 0x00d786 (0x9bea bytes) 000041 1787099173.583587 read 0x008ff4 thru 0x01123b (0x8248 bytes) 000045 1787099173.587839 mapwrite 0x00fb77 thru 0x01760f (0x7a99 bytes) 000047 1787099173.652860 mapread 0x01b508 thru 0x01b580 (0x79 bytes) 000048 1787099173.660413 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000049 1787099173.813182 mapread 0x03cf9e thru 0x03ffff (0x3062 bytes) 000053 1787099173.847929 read 0x019035 thru 0x0203bb (0x7387 bytes) 000054 1787099173.856183 read 0x0353bf thru 0x0381de (0x2e20 bytes) 000056 1787099173.867656 mapread 0x00b101 thru 0x014ac9 (0x99c9 bytes) 000058 1787099173.870832 write 0x038298 thru 0x03ffbc (0x7d25 bytes) 000063 1787099173.880354 MIRROR_EXTEND to 2 mirrors 000065 1787099175.124751 MIRROR_EXTEND to 3 mirrors 000066 1787099175.964966 mapwrite 0x01d218 thru 0x02981f (0xc608 bytes) 000067 1787099176.156103 trunc from 0x040000 to 0x00e291 (0x31d6e bytes) 000069 1787099176.181218 mapwrite 0x02ed6d thru 0x03cd08 (0xdf9c bytes) 000071 1787099176.276942 write 0x02085a thru 0x021272 (0xa19 bytes) 000072 1787099176.290697 mapread 0x02f046 thru 0x0330ba (0x4075 bytes) 000073 1787099176.292249 write 0x03bcf7 thru 0x03ffff (0x4309 bytes) 000074 1787099176.294050 read 0x019a9e thru 0x019aa8 (0xb bytes) 000075 1787099176.300811 write 0x01a5c7 thru 0x01fb90 (0x55ca bytes) 000077 1787099176.312626 write 0x031fd7 thru 0x03c95c (0xa986 bytes) 000078 1787099176.314382 trunc from 0x040000 to 0x00f6a4 (0x3095b bytes) truncating to largest ever: 0x3b64a 000080 1787099176.331518 trunc from 0x00f6a5 to 0x03b649 (0x2bfa5 bytes) 000081 1787099176.348168 mapwrite 0x00ee0a thru 0x00f9e3 (0xbda bytes) 000085 1787099176.393736 read 0x000360 thru 0x005a50 (0x56f1 bytes) 000087 1787099176.417440 mapwrite 0x0011b7 thru 0x009307 (0x8151 bytes) 000088 1787099176.478715 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b64a): Mirror 3: 0x341e50c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b64a): Mirror 3: 0x341e50c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b64a: skipped 000090 1787099176.909240 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000092 1787099177.175886 write 0x015c20 thru 0x01b703 (0x5ae4 bytes) 000095 1787099177.196668 mapread 0x030519 thru 0x033f8c (0x3a74 bytes) 000096 1787099177.241276 mapwrite 0x007010 thru 0x00b93a (0x492b bytes) 000099 1787099177.348697 write 0x027379 thru 0x029b8c (0x2814 bytes) 000100 1787099177.368308 mapread 0x03243b thru 0x036570 (0x4136 bytes) 000101 1787099177.369487 read 0x018715 thru 0x021b4c (0x9438 bytes) 000103 1787099177.388123 mapwrite 0x03b4bf thru 0x03dcb8 (0x27fa bytes) 000106 1787099177.506949 mapread 0x00d271 thru 0x00e5e0 (0x1370 bytes) 000107 1787099177.510001 write 0x018ac1 thru 0x01d8f2 (0x4e32 bytes) 000109 1787099177.514331 mapwrite 0x036eb2 thru 0x03d19a (0x62e9 bytes) 000110 1787099177.590115 mapread 0x00acdd thru 0x015c4d (0xaf71 bytes) 000111 1787099177.608381 write 0x015f4e thru 0x024ce5 (0xed98 bytes) 000114 1787099177.618868 mapwrite 0x01a72a thru 0x020046 (0x591d bytes) 000115 1787099177.702002 mapwrite 0x01768f thru 0x0239cb (0xc33d bytes) 000116 1787099177.783150 mapwrite 0x00dff9 thru 0x00fe26 (0x1e2e bytes) 000117 1787099177.834552 write 0x01b2c1 thru 0x02a993 (0xf6d3 bytes) 000121 1787099177.843593 trunc from 0x03dcb9 to 0x015857 (0x28461 bytes) 000122 1787099177.860903 mapwrite 0x01544a thru 0x019ec6 (0x4a7d bytes) 000124 1787099177.962820 read 0x00b840 thru 0x014f91 (0x9752 bytes) 000126 1787099177.966616 mapwrite 0x016404 thru 0x021f01 (0xbafe bytes) 000127 1787099178.063577 trunc from 0x021f02 to 0x034791 (0x12890 bytes) 000128 1787099178.077790 trunc from 0x034792 to 0x013f9d (0x207f4 bytes) 000130 1787099178.098863 write 0x003b5c thru 0x00f570 (0xba15 bytes) 000131 1787099178.102006 mapwrite 0x00e37a thru 0x01a55e (0xc1e5 bytes) 000132 1787099178.216323 write 0x00f957 thru 0x019ad4 (0xa17e bytes) 000135 1787099178.221915 read 0x003439 thru 0x00dfce (0xab96 bytes) 000137 1787099178.224334 mapwrite 0x007949 thru 0x00ebff (0x72b7 bytes) 000138 1787099178.269005 read 0x0044d9 thru 0x00c6cb (0x81f3 bytes) 000140 1787099178.270450 trunc from 0x01a55f to 0x038fa8 (0x1ea4a bytes) 000141 1787099178.291234 trunc from 0x038fa9 to 0x02fd68 (0x9240 bytes) 000142 1787099178.304959 write 0x00f649 thru 0x016f66 (0x791e bytes) 000143 1787099178.306233 mapread 0x005a62 thru 0x00dc3c (0x81db bytes) 000145 1787099178.309782 read 0x003fdf thru 0x01039e (0xc3c0 bytes) 000146 1787099178.310610 read 0x014292 thru 0x023898 (0xf607 bytes) 000150 1787099178.322704 write 0x024ba8 thru 0x02c761 (0x7bba bytes) 000151 1787099178.324140 trunc from 0x02fd69 to 0x01eef7 (0x10e71 bytes) 000152 1787099178.343336 mapread 0x01bc29 thru 0x01d278 (0x1650 bytes) 000155 1787099178.347211 read 0x01276d thru 0x01d216 (0xaaaa bytes) 000156 1787099178.347958 write 0x0187e5 thru 0x02770a (0xef26 bytes) 000157 1787099178.350238 read 0x00a38c thru 0x018318 (0xdf8d bytes) 000158 1787099178.351306 read 0x01ec10 thru 0x025d0a (0x70fb bytes) 000160 1787099178.353849 write 0x026c44 thru 0x02a665 (0x3a22 bytes) 000161 1787099178.355864 write 0x00822f thru 0x00c277 (0x4049 bytes) 000163 1787099178.358103 write 0x003ae5 thru 0x00ad45 (0x7261 bytes) 000164 1787099178.360682 write 0x00de04 thru 0x00e038 (0x235 bytes) 000168 1787099178.368564 MIRROR_EXTEND to 2 mirrors 000169 1787099179.156834 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000172 1787099179.325772 trunc from 0x02a666 to 0x02934c (0x1319 bytes) 000173 1787099179.382700 write 0x018506 thru 0x026196 (0xdc91 bytes) 000174 1787099179.392659 mapread 0x002580 thru 0x00bc42 (0x96c3 bytes) 000177 1787099179.425862 write 0x01408b thru 0x018310 (0x4286 bytes) 000180 1787099179.433051 read 0x00cc0d thru 0x016d69 (0xa15d bytes) 000183 1787099179.442508 mapwrite 0x01b64d thru 0x02031f (0x4cd3 bytes) 000184 1787099179.500631 read 0x021af4 thru 0x027718 (0x5c25 bytes) 000185 1787099179.513840 write 0x00bbf9 thru 0x018952 (0xcd5a bytes) 000186 1787099179.515396 mapwrite 0x03a140 thru 0x03ffff (0x5ec0 bytes) 000190 1787099179.619898 read 0x030ee1 thru 0x03895c (0x7a7c bytes) 000191 1787099179.634214 read 0x01c669 thru 0x0250ab (0x8a43 bytes) 000192 1787099179.635352 mapread 0x01c625 thru 0x028c65 (0xc641 bytes) 000193 1787099179.637823 read 0x03619a thru 0x03e39b (0x8202 bytes) 000196 1787099179.640148 mapread 0x0025b1 thru 0x006d0e (0x475e bytes) 000197 1787099179.641674 read 0x0016e1 thru 0x006100 (0x4a20 bytes) 000202 1787099179.649809 trunc from 0x040000 to 0x00314b (0x3ceb4 bytes) 000203 1787099179.668330 mapread 0x0021e4 thru 0x00314b (0xf68 bytes) 000204 1787099179.670021 mapread 0x002e31 thru 0x00314b (0x31b bytes) 000205 1787099179.672818 trunc from 0x00314c to 0x031a26 (0x2e8db bytes) 000206 1787099179.691906 write 0x00d8cd thru 0x014142 (0x6876 bytes) truncating to largest ever: 0x3daf0 000209 1787099179.739972 trunc from 0x031a27 to 0x03daef (0xc0c9 bytes) 000210 1787099179.760734 read 0x0146c2 thru 0x015153 (0xa92 bytes) 000211 1787099179.761427 trunc from 0x03daf0 to 0x010cd0 (0x2ce1f bytes) 000212 1787099179.789543 write 0x037015 thru 0x03ffff (0x8feb bytes) 000214 1787099179.796929 write 0x025914 thru 0x0288bb (0x2fa8 bytes) 000215 1787099179.821276 mapread 0x00f25c thru 0x018b34 (0x98d9 bytes) 000216 1787099179.825326 mapread 0x017474 thru 0x01eac3 (0x7650 bytes) 000218 1787099179.830271 mapwrite 0x0121bb thru 0x0175cc (0x5412 bytes) 000220 1787099179.979046 write 0x00239f thru 0x002d92 (0x9f4 bytes) 000222 1787099179.982609 read 0x02645c thru 0x02980e (0x33b3 bytes) 000223 1787099179.984849 trunc from 0x040000 to 0x030738 (0xf8c7 bytes) 000224 1787099180.006033 mapread 0x011817 thru 0x015f25 (0x470f bytes) 000227 1787099180.009769 trunc from 0x030739 to 0x0191e7 (0x17551 bytes) 000231 1787099180.035930 mapwrite 0x00fa16 thru 0x01eca3 (0xf28e bytes) 000232 1787099180.186797 read 0x001da7 thru 0x00289e (0xaf8 bytes) 000233 1787099180.187728 mapread 0x01e461 thru 0x01eca3 (0x843 bytes) 000235 1787099180.191936 read 0x00c9be thru 0x010a3b (0x407e bytes) 000236 1787099180.192870 read 0x0122fb thru 0x01442c (0x2132 bytes) 000237 1787099180.194649 read 0x008641 thru 0x00a738 (0x20f8 bytes) 000241 1787099180.203157 trunc from 0x01eca4 to 0x03d7d0 (0x1eb2d bytes) 000242 1787099180.222610 write 0x038e39 thru 0x03ffff (0x71c7 bytes) 000243 1787099180.239138 write 0x003d97 thru 0x012a18 (0xec82 bytes) 000244 1787099180.246752 read 0x00e391 thru 0x012343 (0x3fb3 bytes) 000246 1787099180.250725 write 0x03f4c0 thru 0x03ffff (0xb40 bytes) 000253 1787099180.257437 mapread 0x01e84d thru 0x0229eb (0x419f bytes) 000254 1787099180.259756 read 0x001720 thru 0x009bf5 (0x84d6 bytes) 000256 1787099180.261745 read 0x00f045 thru 0x019b53 (0xab0f bytes) 000258 1787099180.263897 trunc from 0x040000 to 0x004a4e (0x3b5b1 bytes) 000259 1787099180.302095 mapread 0x00234f thru 0x004a4e (0x2700 bytes) 000261 1787099180.306893 mapwrite 0x019cca thru 0x01b537 (0x186e bytes) 000264 1787099180.409462 write 0x03bb82 thru 0x03e567 (0x29e6 bytes) 000266 1787099180.413929 mapread 0x005942 thru 0x010fe6 (0xb6a5 bytes) 000270 1787099180.421772 read 0x02cbde thru 0x032e86 (0x62a9 bytes) 000272 1787099180.436849 mapwrite 0x01f899 thru 0x02f0c8 (0xf830 bytes) 000274 1787099180.524975 trunc from 0x03e568 to 0x01cda4 (0x217c3 bytes) 000278 1787099180.555977 mapwrite 0x02fe9a thru 0x03f9e7 (0xfb4e bytes) 000279 1787099180.803556 mapwrite 0x032b3b thru 0x03bf21 (0x93e7 bytes) 000280 1787099180.881784 trunc from 0x03f9e8 to 0x0151ee (0x2a7f9 bytes) 000281 1787099180.906265 mapread 0x00ec8f thru 0x0151ee (0x6560 bytes) 000282 1787099180.909809 read 0x013305 thru 0x014e6c (0x1b68 bytes) 000284 1787099180.912993 write 0x02db22 thru 0x0399b3 (0xbe92 bytes) 000287 1787099180.919041 mapwrite 0x00bb76 thru 0x0139c0 (0x7e4b bytes) 000289 1787099181.025102 mapread 0x037404 thru 0x0399b3 (0x25b0 bytes) 000290 1787099181.026120 mapread 0x0133f4 thru 0x014df1 (0x19fe bytes) 000296 1787099181.034181 mapread 0x012cc2 thru 0x01652a (0x3869 bytes) 000298 1787099181.050192 mapwrite 0x009e71 thru 0x01837d (0xe50d bytes) 000299 1787099181.108876 mapwrite 0x020ecc thru 0x022b7b (0x1cb0 bytes) 000302 1787099181.204875 mapread 0x0003ac thru 0x006fd1 (0x6c26 bytes) 000304 1787099181.209783 write 0x022541 thru 0x02ef5d (0xca1d bytes) 000305 1787099181.214250 trunc from 0x0399b4 to 0x03d239 (0x3886 bytes) 000306 1787099181.251719 read 0x01f10f thru 0x022e3b (0x3d2d bytes) 000307 1787099181.254529 trunc from 0x03d23a to 0x02b63d (0x11bfc bytes) 000309 1787099181.294526 write 0x02738a thru 0x03572b (0xe3a2 bytes) 000311 1787099181.305902 write 0x034e30 thru 0x03ffff (0xb1d0 bytes) 000312 1787099181.311624 mapwrite 0x00d962 thru 0x016bc0 (0x925f bytes) 000313 1787099181.458984 mapread 0x031b45 thru 0x03f627 (0xdae3 bytes) 000314 1787099181.460412 mapwrite 0x00851c thru 0x0160d3 (0xdbb8 bytes) 000320 1787099181.555621 write 0x010b4f thru 0x0208d5 (0xfd87 bytes) 000321 1787099181.558934 mapwrite 0x039e19 thru 0x03ffff (0x61e7 bytes) 000322 1787099181.647048 MIRROR_EXTEND to 2 mirrors 000324 1787099182.436013 MIRROR_EXTEND to 3 mirrors 000326 1787099183.413005 mapread 0x00f9a2 thru 0x00fbb5 (0x214 bytes) 000328 1787099183.430049 mapread 0x009469 thru 0x00d7f6 (0x438e bytes) 000330 1787099183.465701 MIRROR_RESYNC 3 mirrors 000333 1787099183.967548 write 0x022372 thru 0x02db61 (0xb7f0 bytes) 000336 1787099184.099469 trunc from 0x040000 to 0x01c0dc (0x23f23 bytes) 000337 1787099184.144576 mapwrite 0x00a25e thru 0x00f9a4 (0x5747 bytes) 000338 1787099184.231558 mapread 0x00ce57 thru 0x01c0dc (0xf286 bytes) 000339 1787099184.242504 write 0x011690 thru 0x01c783 (0xb0f4 bytes) 000340 1787099184.244444 trunc from 0x01c784 to 0x018ddb (0x39a8 bytes) 000341 1787099184.259811 write 0x037060 thru 0x03ffff (0x8fa0 bytes) 000342 1787099184.263811 trunc from 0x040000 to 0x020727 (0x1f8d8 bytes) 000345 1787099184.303701 write 0x006906 thru 0x00f9ac (0x90a7 bytes) 000347 1787099184.306775 mapwrite 0x00480d thru 0x00d3d2 (0x8bc6 bytes) 000349 1787099184.440971 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20728): Mirror 3: 0x6630b4a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20728): Mirror 3: 0x6630b4a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20728: skipped 000350 1787099184.911745 write 0x00a78b thru 0x013d77 (0x95ed bytes) 000352 1787099184.935764 write 0x03bec0 thru 0x03ffff (0x4140 bytes) 000354 1787099184.947552 trunc from 0x040000 to 0x00b38d (0x34c72 bytes) 000355 1787099184.969111 mapread 0x00953b thru 0x00b38d (0x1e53 bytes) 000357 1787099184.980957 mapread 0x006a7a thru 0x00b38d (0x4914 bytes) 000360 1787099185.005315 write 0x028a85 thru 0x03190c (0x8e88 bytes) 000361 1787099185.007916 write 0x03213f thru 0x03e403 (0xc2c5 bytes) 000363 1787099185.015892 write 0x0026b6 thru 0x003329 (0xc74 bytes) 000364 1787099185.017161 mapread 0x009e4c thru 0x00dc39 (0x3dee bytes) 000367 1787099185.044163 write 0x026c42 thru 0x02862b (0x19ea bytes) 000369 1787099185.053798 read 0x02e03a thru 0x034bc4 (0x6b8b bytes) 000371 1787099185.061834 MIRROR_RESYNC 2 mirrors 000374 1787099186.062961 mapread 0x017986 thru 0x027461 (0xfadc bytes) 000375 1787099186.106508 MIRROR_RESYNC 2 mirrors 000376 1787099186.506888 trunc from 0x03e404 to 0x0359af (0x8a54 bytes) 000377 1787099186.631883 mapread 0x0194d6 thru 0x0252a0 (0xbdcb bytes) 000384 1787099186.681991 mapread 0x02186e thru 0x02738f (0x5b22 bytes) 000386 1787099186.689469 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000389 1787099186.928252 trunc from 0x0359b0 to 0x026ec7 (0xeae8 bytes) 000391 1787099186.993622 mapread 0x000eb7 thru 0x00f4e2 (0xe62c bytes) 000392 1787099187.045744 trunc from 0x026ec8 to 0x0180c6 (0xee01 bytes) 000396 1787099187.067450 mapwrite 0x010839 thru 0x01b3bf (0xab87 bytes) 000399 1787099187.190027 trunc from 0x01b3c0 to 0x00ea2c (0xc993 bytes) 000400 1787099187.212583 read 0x005890 thru 0x00ea2c (0x919d bytes) 000405 1787099187.218530 mapread 0x00a447 thru 0x00d6f9 (0x32b3 bytes) 000406 1787099187.220113 write 0x008811 thru 0x018748 (0xff38 bytes) 000407 1787099187.226555 read 0x00e868 thru 0x017425 (0x8bbe bytes) 000408 1787099187.227678 mapwrite 0x009f1c thru 0x00e7f9 (0x48de bytes) 000409 1787099187.278546 trunc from 0x018749 to 0x001d6c (0x169dc bytes) 000410 1787099187.296112 mapwrite 0x03cf1e thru 0x03ffff (0x30e2 bytes) 000416 1787099187.393546 mapread 0x033d13 thru 0x03f96e (0xbc5c bytes) 000420 1787099187.399894 mapread 0x00ecc0 thru 0x012486 (0x37c7 bytes) 000421 1787099187.401493 mapread 0x02e3ca thru 0x02f57c (0x11b3 bytes) 000423 1787099187.404026 read 0x018e2b thru 0x025daf (0xcf85 bytes) 000424 1787099187.410108 MIRROR_EXTEND to 2 mirrors 000426 1787099188.341798 read 0x02153b thru 0x023c75 (0x273b bytes) 000427 1787099188.353202 write 0x03f5cc thru 0x03ffff (0xa34 bytes) 000429 1787099188.412969 mapread 0x024880 thru 0x024a67 (0x1e8 bytes) 000432 1787099188.422652 MIRROR_RESYNC 2 mirrors 000434 1787099189.050461 mapread 0x00c88c thru 0x018cb2 (0xc427 bytes) 000439 1787099189.079702 mapwrite 0x034467 thru 0x03c24b (0x7de5 bytes) 000440 1787099189.251844 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8beb863c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5a31a95e Mirror 3: 0x5a31a95e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000444 1787099190.311171 read 0x01f5a3 thru 0x02c74e (0xd1ac bytes) 000445 1787099190.324344 mapread 0x02edba thru 0x030582 (0x17c9 bytes) 000446 1787099190.367766 mapwrite 0x0077e9 thru 0x008bba (0x13d2 bytes) 000449 1787099190.541483 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8beb863c (stale) Mirror 3: 0x5a31a95e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x722599cb Mirror 4: 0x722599cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000450 1787099192.027453 trunc from 0x040000 to 0x02aa07 (0x155f8 bytes) 000451 1787099192.247977 write 0x01d6de thru 0x02005f (0x2982 bytes) 000452 1787099192.278888 mapwrite 0x013a6b thru 0x01ed25 (0xb2bb bytes) 000454 1787099192.379263 MIRROR_RESYNC 4 mirrors 000455 1787099193.723047 write 0x02be77 thru 0x032b68 (0x6cf2 bytes) 000456 1787099193.804956 mapread 0x0270f7 thru 0x0279ab (0x8b5 bytes) 000458 1787099193.816979 MIRROR_RESYNC 4 mirrors 000460 1787099194.988656 trunc from 0x032b69 to 0x02205f (0x10b09 bytes) 000461 1787099195.067440 write 0x00da9f thru 0x014c0d (0x716f bytes) 000463 1787099195.079559 trunc from 0x022060 to 0x000ad3 (0x2158c bytes) 000464 1787099195.100882 trunc from 0x000ad4 to 0x01b9d1 (0x1aefe bytes) 000466 1787099195.123302 mapread 0x0184fd thru 0x01b9d1 (0x34d5 bytes) 000468 1787099195.149685 mapread 0x018882 thru 0x01b9d1 (0x3150 bytes) 000469 1787099195.150770 mapwrite 0x03268a thru 0x03d0bf (0xaa36 bytes) 000470 1787099195.450262 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 3: 0xc70c4d6b (stale) Mirror 4: 0xc70c4d6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0c0): Mirror 3: 0xc70c4d6b (stale) Mirror 4: 0xc70c4d6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0c0: skipped 000471 1787099195.869402 mapread 0x02556b thru 0x034a08 (0xf49e bytes) 000472 1787099195.905977 mapwrite 0x023a33 thru 0x032d4c (0xf31a bytes) 000474 1787099195.957973 read 0x021f68 thru 0x02cfd1 (0xb06a bytes) 000475 1787099195.958928 trunc from 0x03d0c0 to 0x00aa74 (0x3264b bytes) 000476 1787099195.983269 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xaa75): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa75): Mirror 3: 0xef86b2ea (stale) Mirror 4: 0xef86b2ea (stale) CRC-32 checksum value for chunk [0, 0xaa75): Mirror 1: 0x879f2940 Mirror 5: 0x879f2940 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa75: skipped 000479 1787099197.135974 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xaa75): Mirror 3: 0xef86b2ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa75): Mirror 3: 0xef86b2ea (stale) CRC-32 checksum value for chunk [0, 0xaa75): Mirror 1: 0x879f2940 Mirror 5: 0x879f2940 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa75: skipped 000481 1787099197.685877 mapwrite 0x027c93 thru 0x02c246 (0x45b4 bytes) 000482 1787099197.983239 write 0x02ac13 thru 0x035bd0 (0xafbe bytes) 000483 1787099197.984774 write 0x012830 thru 0x01938c (0x6b5d bytes) 000484 1787099198.013867 mapwrite 0x01d29f thru 0x027dd8 (0xab3a bytes) 000485 1787099198.147193 read 0x00917d thru 0x014107 (0xaf8b bytes) 000487 1787099198.160555 trunc from 0x035bd1 to 0x038fa1 (0x33d1 bytes) 000489 1787099198.188747 mapread 0x023991 thru 0x02ffb9 (0xc629 bytes) 000491 1787099198.193317 read 0x011691 thru 0x013cb5 (0x2625 bytes) 000492 1787099198.200344 MIRROR_RESYNC 3 mirrors 000493 1787099199.233906 mapread 0x0196c6 thru 0x01fbe3 (0x651e bytes) 000494 1787099199.290284 mapread 0x00625e thru 0x0115bc (0xb35f bytes) 000497 1787099199.310668 write 0x00d392 thru 0x012fdd (0x5c4c bytes) 000498 1787099199.395859 read 0x02f067 thru 0x038c1e (0x9bb8 bytes) 000499 1787099199.410354 MIRROR_RESYNC 3 mirrors 000502 1787099200.440477 mapread 0x0136f7 thru 0x021c58 (0xe562 bytes) 000505 1787099200.502901 write 0x030702 thru 0x031b52 (0x1451 bytes) 000508 1787099200.626345 mapread 0x02b1ab thru 0x0331f9 (0x804f bytes) 000509 1787099200.646303 trunc from 0x038fa2 to 0x025019 (0x13f88 bytes) 000510 1787099200.721499 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2501a): Mirror 5: 0x65b77a75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2501a): Mirror 5: 0x65b77a75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2501a: skipped 000512 1787099201.055603 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000513 1787099201.338837 read 0x021c82 thru 0x025019 (0x3398 bytes) 000514 1787099201.352005 write 0x0346ab thru 0x03ffff (0xb955 bytes) 000516 1787099201.360555 read 0x01407c thru 0x01911a (0x509f bytes) 000517 1787099201.372410 write 0x000908 thru 0x007857 (0x6f50 bytes) 000518 1787099201.373925 mapwrite 0x010a75 thru 0x02034a (0xf8d6 bytes) 000519 1787099201.429324 write 0x02b99b thru 0x037cff (0xc365 bytes) 000520 1787099201.430667 mapwrite 0x03f2b1 thru 0x03ffff (0xd4f bytes) 000522 1787099201.475560 MIRROR_EXTEND to 2 mirrors 000523 1787099202.055172 write 0x03f16b thru 0x03ffff (0xe95 bytes) 000525 1787099202.127048 mapread 0x02b2a0 thru 0x03af4e (0xfcaf bytes) 000527 1787099202.159885 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000528 1787099202.308441 read 0x0374ba thru 0x03ffff (0x8b46 bytes) 000529 1787099202.318693 mapwrite 0x000465 thru 0x00b06c (0xac08 bytes) 000530 1787099202.409519 read 0x004e04 thru 0x008798 (0x3995 bytes) 000532 1787099202.412047 write 0x014f9d thru 0x022a9e (0xdb02 bytes) 000533 1787099202.425646 mapwrite 0x02f88c thru 0x031f98 (0x270d bytes) 000534 1787099202.465877 mapwrite 0x0333b2 thru 0x038d7e (0x59cd bytes) 000535 1787099202.515380 write 0x02c7b3 thru 0x037c28 (0xb476 bytes) 000537 1787099202.519467 mapread 0x03c8b8 thru 0x03f18f (0x28d8 bytes) 000538 1787099202.520715 mapwrite 0x0328ec thru 0x03ffff (0xd714 bytes) 000539 1787099202.562300 mapread 0x012c77 thru 0x0218f1 (0xec7b bytes) 000542 1787099202.565831 trunc from 0x040000 to 0x01dd36 (0x222c9 bytes) 000543 1787099202.626366 read 0x01c3a8 thru 0x01ca23 (0x67c bytes) 000544 1787099202.627192 mapread 0x016271 thru 0x016c88 (0xa18 bytes) 000545 1787099202.631565 mapread 0x005b3c thru 0x00eb3f (0x9004 bytes) 000546 1787099202.636585 trunc from 0x01dd37 to 0x01851d (0x5819 bytes) 000547 1787099202.666846 trunc from 0x01851e to 0x030cc3 (0x187a6 bytes) 000548 1787099202.705401 mapwrite 0x03a61f thru 0x03ffff (0x59e1 bytes) 000551 1787099202.809170 mapread 0x00a9d8 thru 0x0150db (0xa704 bytes) 000554 1787099202.823533 mapread 0x030774 thru 0x032c71 (0x24fe bytes) 000557 1787099202.833126 trunc from 0x040000 to 0x03a6df (0x5920 bytes) 000559 1787099202.862388 write 0x015d46 thru 0x01a9e3 (0x4c9e bytes) 000562 1787099202.869541 write 0x022750 thru 0x02a38b (0x7c3c bytes) 000565 1787099202.873510 mapwrite 0x024d0f thru 0x02d40d (0x86ff bytes) 000569 1787099202.928321 read 0x016c01 thru 0x02474b (0xdb4b bytes) 000573 1787099202.931791 trunc from 0x03a6e0 to 0x01f346 (0x1b399 bytes) 000574 1787099202.946635 mapwrite 0x008748 thru 0x0158ee (0xd1a7 bytes) 000577 1787099202.997355 read 0x01979d thru 0x01c558 (0x2dbc bytes) 000580 1787099203.000331 read 0x00f021 thru 0x010c46 (0x1c26 bytes) 000583 1787099203.003328 trunc from 0x01f347 to 0x02154d (0x2207 bytes) 000586 1787099203.021106 mapread 0x0124d3 thru 0x015f2d (0x3a5b bytes) 000587 1787099203.022814 write 0x01898b thru 0x022e59 (0xa4cf bytes) 000588 1787099203.025185 write 0x01427c thru 0x01a673 (0x63f8 bytes) 000591 1787099203.028452 write 0x001a82 thru 0x011a61 (0xffe0 bytes) 000592 1787099203.034851 MIRROR_EXTEND to 2 mirrors 000593 1787099203.839737 trunc from 0x022e5a to 0x00d1e5 (0x15c74 bytes) 000594 1787099204.026903 mapwrite 0x0253e9 thru 0x030f2b (0xbb43 bytes) 000596 1787099204.144706 trunc from 0x030f2c to 0x0152ee (0x1bc3d bytes) 000597 1787099204.161931 MIRROR_RESYNC 2 mirrors 000598 1787099204.797383 write 0x03796b thru 0x03bff4 (0x468a bytes) 000600 1787099204.884102 trunc from 0x03bff5 to 0x01da92 (0x1e562 bytes) 000602 1787099204.930530 write 0x035bbc thru 0x0368a6 (0xceb bytes) 000605 1787099204.939029 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x368a7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x368a7): Mirror 2: 0x69b9faa6 (stale) CRC-32 checksum value for chunk [0, 0x368a7): Mirror 1: 0xfd958bbd Mirror 3: 0xfd958bbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x368a7: skipped 000606 1787099206.136855 mapwrite 0x03b62b thru 0x03ffff (0x49d5 bytes) 000607 1787099206.303803 trunc from 0x040000 to 0x01b66b (0x24994 bytes) 000608 1787099206.323152 trunc from 0x01b66c to 0x018263 (0x3408 bytes) 000609 1787099206.337815 read 0x00bc15 thru 0x018263 (0xc64f bytes) 000610 1787099206.351813 MIRROR_RESYNC 3 mirrors 000612 1787099207.197706 MIRROR_EXTEND to 4 mirrors 000613 1787099208.024535 read 0x017acc thru 0x018263 (0x798 bytes) 000615 1787099208.057596 trunc from 0x018264 to 0x00bec5 (0xc39e bytes) 000617 1787099208.180753 read 0x009094 thru 0x00bec5 (0x2e32 bytes) 000620 1787099208.192526 trunc from 0x00bec6 to 0x00eaef (0x2c2a bytes) 000622 1787099208.227384 MIRROR_RESYNC 4 mirrors 000623 1787099209.124755 mapread 0x006d2d thru 0x00eaef (0x7dc3 bytes) 000624 1787099209.145456 write 0x0079c1 thru 0x0113df (0x9a1f bytes) 000625 1787099209.224544 mapread 0x00e905 thru 0x0113df (0x2adb bytes) 000626 1787099209.225734 trunc from 0x0113e0 to 0x015d76 (0x4997 bytes) 000627 1787099209.270027 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x15d77): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15d77): Mirror 2: 0x486a6b19 (stale) Mirror 3: 0x486a6b19 (stale) Mirror 4: 0x486a6b19 (stale) CRC-32 checksum value for chunk [0, 0x15d77): Mirror 1: 0x417a655c Mirror 5: 0x417a655c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15d77: skipped 000628 1787099210.357381 mapread 0x0156b9 thru 0x015d76 (0x6be bytes) 000629 1787099210.367859 mapwrite 0x033efb thru 0x03db25 (0x9c2b bytes) 000631 1787099210.571529 write 0x00db12 thru 0x0158aa (0x7d99 bytes) 000632 1787099210.587375 read 0x0317ae thru 0x03db25 (0xc378 bytes) 000633 1787099210.592633 mapwrite 0x021652 thru 0x025d54 (0x4703 bytes) 000634 1787099210.660195 mapwrite 0x00622f thru 0x00685c (0x62e bytes) 000635 1787099210.732902 MIRROR_RESYNC 5 mirrors 000636 1787099211.949595 mapwrite 0x02afe9 thru 0x02d207 (0x221f bytes) 000637 1787099212.115728 read 0x00c100 thru 0x00f45b (0x335c bytes) 000638 1787099212.132662 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db26): Mirror 2: 0x6ef99517 (stale) Mirror 3: 0x6ef99517 (stale) Mirror 5: 0x6ef99517 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db26): Mirror 2: 0x6ef99517 (stale) Mirror 3: 0x6ef99517 (stale) Mirror 5: 0x6ef99517 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db26: skipped 000639 1787099212.513688 mapread 0x01f6f7 thru 0x027833 (0x813d bytes) 000645 1787099212.544390 trunc from 0x03db26 to 0x01c224 (0x21901 bytes) 000646 1787099212.564094 mapread 0x000bd0 thru 0x0095aa (0x89db bytes) 000648 1787099212.573664 mapwrite 0x001776 thru 0x00ce03 (0xb68e bytes) 000656 1787099212.630313 write 0x02b502 thru 0x0356f0 (0xa1ef bytes) 000657 1787099212.637893 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x356f1): Mirror 2: 0xda33aa96 (stale) Mirror 5: 0xda33aa96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x356f1): Mirror 2: 0xda33aa96 (stale) Mirror 5: 0xda33aa96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x356f1: skipped 000660 1787099212.923995 write 0x018fcc thru 0x023862 (0xa897 bytes) 000662 1787099212.938108 read 0x0220e7 thru 0x029cf8 (0x7c12 bytes) 000663 1787099212.951995 MIRROR_RESYNC 3 mirrors 000664 1787099213.863118 mapwrite 0x0298da thru 0x02b270 (0x1997 bytes) 000665 1787099214.000358 mapread 0x015d90 thru 0x021d18 (0xbf89 bytes) 000669 1787099214.038870 MIRROR_RESYNC 3 mirrors 000671 1787099215.098943 trunc from 0x0356f1 to 0x00f7eb (0x25f05 bytes) 000673 1787099215.190496 write 0x025b18 thru 0x034f82 (0xf46b bytes) 000675 1787099215.200220 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34f83): Mirror 2: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f83): Mirror 2: 0x9fcfb939 (stale) Mirror 5: 0x9fcfb939 (stale) CRC-32 checksum value for chunk [0, 0x34f83): Mirror 1: 0x9dbb63f1 Mirror 6: 0x9dbb63f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f83: skipped 000676 1787099216.218351 read 0x027558 thru 0x02c220 (0x4cc9 bytes) 000678 1787099216.242032 read 0x0012ad thru 0x0058fc (0x4650 bytes) 000679 1787099216.260994 mapwrite 0x0212d7 thru 0x02d8c7 (0xc5f1 bytes) 000680 1787099216.556953 mapread 0x00d6c1 thru 0x00eb02 (0x1442 bytes) 000681 1787099216.575016 mapread 0x01909d thru 0x01ccbb (0x3c1f bytes) 000683 1787099216.606428 trunc from 0x034f83 to 0x01e287 (0x16cfb bytes) 000684 1787099216.634808 read 0x006efe thru 0x00ac5d (0x3d60 bytes) 000685 1787099216.655486 mapread 0x001f93 thru 0x0040fc (0x216a bytes) 000686 1787099216.657604 mapread 0x000dbf thru 0x004c28 (0x3e6a bytes) 000687 1787099216.659345 write 0x03b743 thru 0x03ffff (0x48bd bytes) 000688 1787099216.661432 mapread 0x027100 thru 0x02730b (0x20c bytes) 000689 1787099216.670545 write 0x021e65 thru 0x025d4c (0x3ee8 bytes) 000690 1787099216.676779 MIRROR_RESYNC 4 mirrors 000691 1787099218.262213 MIRROR_RESYNC 4 mirrors 000692 1787099218.890697 read 0x03faf9 thru 0x03ffff (0x507 bytes) 000693 1787099218.903735 mapwrite 0x001a0e thru 0x00d386 (0xb979 bytes) 000694 1787099219.183762 read 0x0289f1 thru 0x030d27 (0x8337 bytes) 000697 1787099219.215905 mapwrite 0x00d958 thru 0x018b74 (0xb21d bytes) 000698 1787099219.277926 read 0x03bc97 thru 0x03ffff (0x4369 bytes) 000702 1787099219.286956 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfcfc921c (stale) Mirror 5: 0xfcfc921c (stale) Mirror 6: 0xfcfc921c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdb741d32 Mirror 7: 0xdb741d32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000703 1787099220.141281 mapread 0x02091d thru 0x02c8ff (0xbfe3 bytes) 000705 1787099220.191321 trunc from 0x040000 to 0x03a688 (0x5977 bytes) 000706 1787099220.316901 read 0x00e718 thru 0x0136e2 (0x4fcb bytes) 000708 1787099220.328290 read 0x0165a8 thru 0x01a0ca (0x3b23 bytes) 000710 1787099220.342035 read 0x026ada thru 0x030855 (0x9d7c bytes) 000712 1787099220.343529 mapwrite 0x03809b thru 0x03f19c (0x7102 bytes) 000713 1787099220.434924 mapwrite 0x02f1ae thru 0x03b19b (0xbfee bytes) 000714 1787099220.504214 mapwrite 0x0204c3 thru 0x028b46 (0x8684 bytes) 000715 1787099220.600748 mapread 0x033a95 thru 0x03f19c (0xb708 bytes) 000716 1787099220.610331 mapwrite 0x01d4db thru 0x020713 (0x3239 bytes) 000717 1787099220.720391 mapwrite 0x0244b0 thru 0x03417e (0xfccf bytes) 000720 1787099220.801024 read 0x003f26 thru 0x00635b (0x2436 bytes) 000721 1787099220.802151 trunc from 0x03f19d to 0x010ae6 (0x2e6b6 bytes) 000723 1787099220.822639 write 0x000e9b thru 0x010ca6 (0xfe0c bytes) 000724 1787099220.824885 write 0x026a67 thru 0x02ecd4 (0x826e bytes) 000725 1787099220.827886 mapread 0x00142b thru 0x00b992 (0xa568 bytes) 000728 1787099220.831091 read 0x0002de thru 0x004b28 (0x484b bytes) 000729 1787099220.831904 mapwrite 0x0184cc thru 0x026ed9 (0xea0e bytes) 000730 1787099220.915483 read 0x02e9e1 thru 0x02ecd4 (0x2f4 bytes) 000732 1787099220.917460 mapwrite 0x01b7a2 thru 0x01f2de (0x3b3d bytes) 000733 1787099220.986225 read 0x00f16a thru 0x011033 (0x1eca bytes) 000734 1787099220.987507 write 0x036c1a thru 0x039c6d (0x3054 bytes) 000735 1787099220.997929 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39c6e): Mirror 2: 0x451fb3c8 (stale) Mirror 6: 0x451fb3c8 (stale) Mirror 7: 0x3986bac6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c6e): Mirror 2: 0x451fb3c8 (stale) Mirror 6: 0x451fb3c8 (stale) Mirror 7: 0x3986bac6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c6e: skipped 000736 1787099221.497892 mapread 0x02192b thru 0x0262ab (0x4981 bytes) 000737 1787099221.537853 mapwrite 0x01e35d thru 0x026742 (0x83e6 bytes) 000738 1787099221.687741 mapwrite 0x0385df thru 0x03efee (0x6a10 bytes) 000740 1787099221.859691 mapread 0x0078a1 thru 0x01169a (0x9dfa bytes) 000742 1787099221.900053 mapwrite 0x021475 thru 0x02a251 (0x8ddd bytes) 000743 1787099221.955412 mapwrite 0x02a0ac thru 0x034dd9 (0xad2e bytes) 000745 1787099222.019553 read 0x03c60f thru 0x03efee (0x29e0 bytes) 000746 1787099222.020904 trunc from 0x03efef to 0x0010f5 (0x3def9 bytes) 000747 1787099222.043006 trunc from 0x0010f6 to 0x00b161 (0xa06c bytes) 000749 1787099222.062648 mapwrite 0x014566 thru 0x01fa7e (0xb519 bytes) 000750 1787099222.208052 read 0x00e489 thru 0x0147aa (0x6322 bytes) 000754 1787099222.213946 mapwrite 0x018535 thru 0x01abf9 (0x26c5 bytes) 000756 1787099222.283491 write 0x009e95 thru 0x00d8b5 (0x3a21 bytes) 000757 1787099222.298906 write 0x018f40 thru 0x01d335 (0x43f6 bytes) 000758 1787099222.301345 trunc from 0x01fa7f to 0x02c239 (0xc7bb bytes) 000767 1787099222.335849 mapread 0x01070f thru 0x017b95 (0x7487 bytes) 000769 1787099222.348134 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c23a): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c23a): Mirror 2: 0x7bc63c32 (stale) Mirror 6: 0x7bc63c32 (stale) Mirror 7: 0x1378db2a (stale) CRC-32 checksum value for chunk [0, 0x2c23a): Mirror 1: 0x58b416f8 Mirror 8: 0x58b416f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c23a: skipped 000772 1787099223.323881 trunc from 0x02c23a to 0x03bb99 (0xf960 bytes) 000773 1787099223.421822 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bb9a): Mirror 2: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb9a): Mirror 2: 0x4860f23c (stale) Mirror 6: 0x4860f23c (stale) Mirror 7: 0x2d5b375b (stale) Mirror 8: 0xcdc9a5f9 (stale) CRC-32 checksum value for chunk [0, 0x3bb9a): Mirror 1: 0xcdc9a5f9 Mirror 9: 0xcdc9a5f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb9a: skipped 000776 1787099224.641951 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb9a): Mirror 2: 0x4860f23c (stale) Mirror 7: 0x2d5b375b (stale) Mirror 8: 0xcdc9a5f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb9a): Mirror 2: 0x4860f23c (stale) Mirror 7: 0x2d5b375b (stale) Mirror 8: 0xcdc9a5f9 (stale) CRC-32 checksum value for chunk [0, 0x3bb9a): Mirror 1: 0xcdc9a5f9 Mirror 9: 0xcdc9a5f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb9a: skipped 000777 1787099225.468009 trunc from 0x03bb9a to 0x038757 (0x3442 bytes) 000778 1787099225.619207 read 0x02a518 thru 0x02a61c (0x105 bytes) 000780 1787099225.633658 mapwrite 0x00d9c4 thru 0x0189fb (0xb038 bytes) 000781 1787099225.786655 read 0x021c05 thru 0x02a107 (0x8503 bytes) 000783 1787099225.812282 read 0x00dc4e thru 0x01bd34 (0xe0e7 bytes) 000784 1787099225.813220 write 0x007553 thru 0x01282c (0xb2da bytes) 000787 1787099225.823956 write 0x00ad26 thru 0x00d97d (0x2c58 bytes) 000788 1787099225.828940 mapwrite 0x027af8 thru 0x02c8e9 (0x4df2 bytes) 000789 1787099225.916757 mapread 0x00927c thru 0x012272 (0x8ff7 bytes) 000790 1787099225.918846 mapwrite 0x010e42 thru 0x01b1fb (0xa3ba bytes) 000792 1787099225.984400 trunc from 0x038758 to 0x010cf1 (0x27a66 bytes) 000794 1787099226.018716 read 0x007efb thru 0x010cf1 (0x8df7 bytes) 000797 1787099226.023323 write 0x001dfc thru 0x00bf11 (0xa116 bytes) 000798 1787099226.025590 write 0x023035 thru 0x0255b2 (0x257e bytes) 000800 1787099226.028736 mapread 0x01f726 thru 0x0255b2 (0x5e8d bytes) 000801 1787099226.040361 mapwrite 0x027c90 thru 0x02d30f (0x5680 bytes) 000803 1787099226.222750 MIRROR_RESYNC 5 mirrors 000805 1787099227.489766 mapread 0x02365d thru 0x02d30f (0x9cb3 bytes) 000807 1787099227.519964 mapread 0x01d263 thru 0x029315 (0xc0b3 bytes) 000808 1787099227.528052 trunc from 0x02d310 to 0x0302c9 (0x2fba bytes) 000812 1787099227.608978 read 0x013179 thru 0x021b3b (0xe9c3 bytes) 000814 1787099227.618872 mapwrite 0x009adc thru 0x018723 (0xec48 bytes) 000815 1787099227.679866 write 0x025da6 thru 0x02e184 (0x83df bytes) 000816 1787099227.697265 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x302ca): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302ca): Mirror 2: 0x2b0ea83e (stale) Mirror 7: 0x2b0ea83e (stale) Mirror 8: 0x2b0ea83e (stale) Mirror 9: 0x2b0ea83e (stale) CRC-32 checksum value for chunk [0, 0x302ca): Mirror 1: 0xd5c53ce9 Mirror 10: 0xd5c53ce9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302ca: skipped 000819 1787099228.680298 trunc from 0x0302ca to 0x023cb3 (0xc616 bytes) 000821 1787099228.783177 mapwrite 0x037595 thru 0x03ffff (0x8a6b bytes) 000822 1787099228.866825 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x747b285a (stale) Mirror 7: 0x747b285a (stale) Mirror 8: 0x747b285a (stale) Mirror 9: 0x747b285a (stale) Mirror 10: 0xc7f260a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfc1f29b9 Mirror 11: 0xfc1f29b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000823 1787099229.822121 write 0x039daf thru 0x03ffff (0x6251 bytes) 000825 1787099229.911733 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x747b285a (stale) Mirror 7: 0x747b285a (stale) Mirror 8: 0x747b285a (stale) Mirror 9: 0x747b285a (stale) Mirror 10: 0xc7f260a0 (stale) Mirror 11: 0xfc1f29b9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2009452e Mirror 12: 0x2009452e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000826 1787099231.049803 mapwrite 0x017087 thru 0x020098 (0x9012 bytes) 000827 1787099231.311229 mapread 0x037ec8 thru 0x03fbd8 (0x7d11 bytes) 000828 1787099231.332883 MIRROR_RESYNC 8 mirrors 000831 1787099232.980203 mapwrite 0x0194c2 thru 0x01f2cb (0x5e0a bytes) 000832 1787099233.165384 write 0x034339 thru 0x03e430 (0xa0f8 bytes) 000833 1787099233.176432 mapwrite 0x02da74 thru 0x02fb40 (0x20cd bytes) 000835 1787099233.235595 write 0x0264f2 thru 0x03097f (0xa48e bytes) 000836 1787099233.237656 mapread 0x0292e3 thru 0x02b8e8 (0x2606 bytes) 000837 1787099233.239305 trunc from 0x040000 to 0x02d2d4 (0x12d2b bytes) 000838 1787099233.262038 MIRROR_RESYNC 8 mirrors 000839 1787099235.043234 read 0x012aab thru 0x015553 (0x2aa9 bytes) 000840 1787099235.073819 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) 000843 1787099235.480687 read 0x000832 thru 0x00fc8c (0xf45b bytes) 000844 1787099235.492339 mapwrite 0x03732d thru 0x03ffff (0x8cd3 bytes) 000846 1787099235.659190 read 0x0255e1 thru 0x02bc98 (0x66b8 bytes) 000847 1787099235.675610 MIRROR_RESYNC 7 mirrors 000849 1787099236.828425 read 0x015796 thru 0x01ba64 (0x62cf bytes) 000852 1787099236.854460 write 0x0102ba thru 0x01793c (0x7683 bytes) 000854 1787099236.953276 mapread 0x03a347 thru 0x03ffff (0x5cb9 bytes) 000856 1787099236.980809 read 0x007331 thru 0x014406 (0xd0d6 bytes) 000857 1787099236.995589 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa2cd5a2f (stale) Mirror 7: 0xa2cd5a2f (stale) Mirror 8: 0xa2cd5a2f (stale) Mirror 9: 0xa2cd5a2f (stale) Mirror 10: 0xa2cd5a2f (stale) Mirror 12: 0xa2cd5a2f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa65050fb Mirror 13: 0xa65050fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000859 1787099238.407827 mapwrite 0x03ac65 thru 0x03dc48 (0x2fe4 bytes) 000860 1787099238.575868 write 0x0111a1 thru 0x0128d2 (0x1732 bytes) 000862 1787099238.586973 read 0x01430e thru 0x01a8ca (0x65bd bytes) 000863 1787099238.600011 read 0x01d8c2 thru 0x02714d (0x988c bytes) 000865 1787099238.612442 read 0x025f5e thru 0x02af05 (0x4fa8 bytes) 000868 1787099238.615443 mapwrite 0x03f9f2 thru 0x03ffff (0x60e bytes) 000870 1787099238.707707 write 0x03a2ac thru 0x03ffff (0x5d54 bytes) 000871 1787099238.708943 trunc from 0x040000 to 0x026718 (0x198e7 bytes) 000872 1787099238.745565 write 0x016a3e thru 0x016bd3 (0x196 bytes) 000873 1787099238.751124 MIRROR_RESYNC 8 mirrors 000874 1787099240.225640 write 0x032181 thru 0x032c56 (0xad6 bytes) 000876 1787099240.321131 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32c57): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c57): Mirror 2: 0x6a05c214 (stale) Mirror 7: 0x6a05c214 (stale) Mirror 8: 0x6a05c214 (stale) Mirror 9: 0x6a05c214 (stale) Mirror 10: 0x6a05c214 (stale) Mirror 12: 0x6a05c214 (stale) Mirror 13: 0x6a05c214 (stale) CRC-32 checksum value for chunk [0, 0x32c57): Mirror 1: 0x5bb2a5b8 Mirror 14: 0x5bb2a5b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c57: skipped 000878 1787099241.209825 MIRROR_RESYNC 9 mirrors 000879 1787099243.152944 write 0x00266a thru 0x00794f (0x52e6 bytes) 000880 1787099243.340008 write 0x00294b thru 0x002bb1 (0x267 bytes) 000882 1787099243.342826 write 0x03c46a thru 0x03ffff (0x3b96 bytes) 000883 1787099243.354879 mapwrite 0x00c935 thru 0x011722 (0x4dee bytes) 000884 1787099243.535203 write 0x03c10d thru 0x03ffff (0x3ef3 bytes) 000885 1787099243.536815 mapread 0x01e9fb thru 0x02999a (0xafa0 bytes) 000886 1787099243.571629 write 0x004ca8 thru 0x008482 (0x37db bytes) 000888 1787099243.594169 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x65b95a69 (stale) Mirror 7: 0x65b95a69 (stale) Mirror 8: 0x65b95a69 (stale) Mirror 9: 0x65b95a69 (stale) Mirror 10: 0x65b95a69 (stale) Mirror 12: 0x65b95a69 (stale) Mirror 13: 0x65b95a69 (stale) Mirror 14: 0x65b95a69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x65058685 Mirror 15: 0x65058685 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000890 1787099245.005837 mapread 0x005263 thru 0x011c89 (0xca27 bytes) 000891 1787099245.050224 MIRROR_RESYNC 10 mirrors 000896 1787099246.999585 mapwrite 0x015e7f thru 0x0175a6 (0x1728 bytes) 000897 1787099247.426050 trunc from 0x040000 to 0x024ca1 (0x1b35e bytes) 000898 1787099247.492509 write 0x037156 thru 0x03ffff (0x8eaa bytes) 000899 1787099247.495035 mapwrite 0x00fcc2 thru 0x01ac44 (0xaf83 bytes) 000904 1787099247.605953 write 0x024f96 thru 0x0250e2 (0x14d bytes) 000905 1787099247.632497 write 0x02f217 thru 0x03831e (0x9108 bytes) 000907 1787099247.663837 MIRROR_RESYNC 10 mirrors 000910 1787099249.693136 read 0x03ec35 thru 0x03ffff (0x13cb bytes) 000911 1787099249.702665 trunc from 0x040000 to 0x03b92c (0x46d3 bytes) 000912 1787099249.869624 trunc from 0x03b92d to 0x010cac (0x2ac80 bytes) 000913 1787099249.945839 read 0x001364 thru 0x008cb4 (0x7951 bytes) 000915 1787099249.974789 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x10cad): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10cad): Mirror 2: 0x943a57a3 (stale) Mirror 7: 0x943a57a3 (stale) Mirror 8: 0x943a57a3 (stale) Mirror 9: 0x943a57a3 (stale) Mirror 10: 0x943a57a3 (stale) Mirror 12: 0x943a57a3 (stale) Mirror 13: 0x943a57a3 (stale) Mirror 14: 0x943a57a3 (stale) Mirror 15: 0x943a57a3 (stale) CRC-32 checksum value for chunk [0, 0x10cad): Mirror 1: 0x943a57a3 Mirror 16: 0x943a57a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10cad: skipped 000916 1787099251.618416 mapread 0x004cae thru 0x00980d (0x4b60 bytes) 000917 1787099251.671090 trunc from 0x010cad to 0x03bb18 (0x2ae6c bytes) 000918 1787099251.894791 mapwrite 0x00df14 thru 0x01b93c (0xda29 bytes) 000921 1787099252.129051 read 0x004bed thru 0x00ff83 (0xb397 bytes) 000922 1787099252.147638 mapread 0x01b882 thru 0x029982 (0xe101 bytes) 000923 1787099252.233396 mapread 0x02cb64 thru 0x032a1e (0x5ebb bytes) 000925 1787099252.274419 mapwrite 0x0375f8 thru 0x03ab5f (0x3568 bytes) 000926 1787099252.467030 mapread 0x016854 thru 0x0204fd (0x9caa bytes) 000929 1787099252.472794 write 0x031d57 thru 0x037435 (0x56df bytes) 000930 1787099252.477734 write 0x015b8a thru 0x018cea (0x3161 bytes) 000931 1787099252.482790 write 0x003780 thru 0x00d7ae (0xa02f bytes) 000932 1787099252.498904 mapread 0x00e803 thru 0x00f95f (0x115d bytes) 000933 1787099252.502790 write 0x02c9eb thru 0x02dcad (0x12c3 bytes) 000938 1787099252.515706 mapwrite 0x00ad6a thru 0x00e8f0 (0x3b87 bytes) 000939 1787099252.629288 trunc from 0x03bb19 to 0x00c80f (0x2f309 bytes) 000940 1787099252.665413 write 0x02e815 thru 0x03dfda (0xf7c6 bytes) 000941 1787099252.668565 mapwrite 0x01ab42 thru 0x0295d9 (0xea98 bytes) 000943 1787099252.737284 mapwrite 0x015677 thru 0x0247c9 (0xf153 bytes) 000944 1787099252.785681 trunc from 0x03dfdb to 0x025734 (0x188a6 bytes) 000945 1787099252.808054 trunc from 0x025735 to 0x005221 (0x20513 bytes) 000948 1787099252.838854 trunc from 0x005222 to 0x029298 (0x24077 bytes) 000951 1787099252.865331 mapwrite 0x03224d thru 0x03ef9b (0xcd4f bytes) 000952 1787099252.972855 write 0x032594 thru 0x03ffff (0xda6c bytes) 000953 1787099252.977166 read 0x0174a7 thru 0x0269b8 (0xf512 bytes) 000954 1787099252.995049 trunc from 0x040000 to 0x016dd6 (0x29229 bytes) 000958 1787099253.019383 write 0x03ce80 thru 0x03ffff (0x3180 bytes) 000960 1787099253.022115 mapwrite 0x030a70 thru 0x034b5f (0x40f0 bytes) 000961 1787099253.152361 mapwrite 0x0261d9 thru 0x029be0 (0x3a08 bytes) 000962 1787099253.292696 trunc from 0x040000 to 0x015df7 (0x2a208 bytes) 000963 1787099253.331331 write 0x008115 thru 0x00aa27 (0x2913 bytes) 000965 1787099253.335197 write 0x03cc05 thru 0x03e113 (0x150f bytes) 000966 1787099253.337417 mapread 0x036bb1 thru 0x03e113 (0x7563 bytes) 000968 1787099253.425404 trunc from 0x03e114 to 0x01113b (0x2cfd8 bytes) 000970 1787099253.452510 trunc from 0x01113c to 0x009193 (0x7fa8 bytes) 000972 1787099253.535561 MIRROR_RESYNC 11 mirrors 000973 1787099255.143244 trunc from 0x009194 to 0x01d906 (0x14773 bytes) 000975 1787099255.204554 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d907): Mirror 2: 0x3cc766b0 (stale) Mirror 7: 0x3cc766b0 (stale) Mirror 9: 0x3cc766b0 (stale) Mirror 10: 0x3cc766b0 (stale) Mirror 12: 0x3cc766b0 (stale) Mirror 13: 0x3cc766b0 (stale) Mirror 14: 0x3cc766b0 (stale) Mirror 15: 0x3cc766b0 (stale) Mirror 16: 0x3cc766b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d907): Mirror 2: 0x3cc766b0 (stale) Mirror 7: 0x3cc766b0 (stale) Mirror 9: 0x3cc766b0 (stale) Mirror 10: 0x3cc766b0 (stale) Mirror 12: 0x3cc766b0 (stale) Mirror 13: 0x3cc766b0 (stale) Mirror 14: 0x3cc766b0 (stale) Mirror 15: 0x3cc766b0 (stale) Mirror 16: 0x3cc766b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d907: skipped 000978 1787099255.863262 mapread 0x00be06 thru 0x016d27 (0xaf22 bytes) 000979 1787099255.935792 mapread 0x01d2dd thru 0x01d906 (0x62a bytes) 000981 1787099255.938785 write 0x00b8f3 thru 0x0155ec (0x9cfa bytes) 000982 1787099255.963208 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1d907): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d907): Mirror 2: 0x3cc766b0 (stale) Mirror 7: 0x3cc766b0 (stale) Mirror 9: 0x3cc766b0 (stale) Mirror 10: 0x3cc766b0 (stale) Mirror 12: 0x3cc766b0 (stale) Mirror 13: 0x3cc766b0 (stale) Mirror 14: 0x3cc766b0 (stale) Mirror 15: 0x3cc766b0 (stale) Mirror 16: 0x3cc766b0 (stale) CRC-32 checksum value for chunk [0, 0x1d907): Mirror 1: 0x241451dc Mirror 17: 0x241451dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d907: skipped 000983 1787099257.155820 trunc from 0x01d907 to 0x00c55c (0x113aa bytes) 000984 1787099257.370586 read 0x005d84 thru 0x00c55c (0x67d9 bytes) 000986 1787099257.381531 write 0x022dee thru 0x02ff82 (0xd195 bytes) 000991 1787099257.393121 MIRROR_RESYNC 11 mirrors 000992 1787099260.987970 MIRROR_RESYNC 11 mirrors 000993 1787099261.826588 mapwrite 0x013d4c thru 0x01d89f (0x9b54 bytes) 000995 1787099262.134019 MIRROR_RESYNC 11 mirrors 000996 1787099264.148737 mapwrite 0x00f743 thru 0x0114fa (0x1db8 bytes) 000999 1787099264.481923 write 0x01a790 thru 0x026093 (0xb904 bytes) All operations completed A-OK! PASS 70a (101s) == sanity-flr test 71: check mirror extend parent fid ==== 20:27:49 (1787099269) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:27:53 (1787099273) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.258058 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173478 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 20:28:01 (1787099281) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4569440 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3149536 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6754464 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7626752 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6952896 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8194624 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8468288 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9062560 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8971104 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57365 57366 57368 57370 57373 failed 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) 192.168.204.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:29:29 (1787099369) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6164416 .. Extending file size to 993568 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7340896 .. failed Extending file size to 5289792 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6970400 .. Extending file size to 7699968 .. Extending file size to 9061600 .. Extending file size to 7474336 .. Extending file size to 7869152 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8486112 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8694880 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58784 58785 58786 58790 58791 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) 192.168.204.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:30:52 (1787099452) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.078542 s, 26.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.279905 s, 7.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x42d27646 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:31:11 (1787099471) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0891145 s, 23.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 20:31:19 (1787099479) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:31:26 (1787099486) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182315 s, 11.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:31:33 (1787099493) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25626 s, 8.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x302:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x303:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e7:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:31:41 (1787099501) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225716 s, 9.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:32:14 (1787099534) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979572 s, 10.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:33:18 (1787099598) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334542 s, 6.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:34:24 (1787099664) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. Creating new pool test_204f oleg412-server: Pool lustre.test_204f created Adding targets to pool oleg412-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg412-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208801 s, 10.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg412-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg412-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg412-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:35:51 (1787099751) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:35:58 (1787099758) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:36:06 (1787099766) oleg412-server: Pool lustre.test_206 created Adding targets to pool oleg412-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg412-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg412-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg412-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg412-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:36:31 (1787099791) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121684 s, 8.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640081 s, 16.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x45487874 Mirror 3: 0x45487874 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:36:42 (1787099802) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:36:47 (1787099807) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:36:51 (1787099811) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.1479 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xe20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xba0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xba0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xba0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1440000 Read this (4 bytes): BBBB 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:37:34 (1787099854) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0334428 s, 245 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:37:40 (1787099860) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768217 s, 13.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20d lmm_fid: [0x200000404:0x20d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834445 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-18 20:37:42.000000000 -0400 Modify: 2026-08-18 20:37:42.000000000 -0400 Change: 2026-08-18 20:37:44.000000000 -0400 Birth: 2026-08-18 20:37:42.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:37:49 (1787099869) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:37:52 (1787099872) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.596741 s, 68.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0794966 s, 51.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 18 20:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 18 20:37 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:38:01 (1787099881) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:38:08 (1787099888) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1769 sec =========== 20:38:17 (1787099897) === sanity-flr: start cleanup 20:38:18 (1787099898) === === sanity-flr: finish cleanup 20:38:26 (1787099906) ===