-----============= acceptance-small: sanity-flr ============----- Sat Aug 8 01:15:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg158-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:15:44 (1786166144) === oleg158-client.virtnet: executing check_config_client /mnt/lustre oleg158-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg158-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cc9c8d13000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cc9c8d13000.idle_timeout=debug disable quota as required oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:16:12 (1786166172) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7659] == sanity-flr test 0a: lfs mirror create with -N option == 01:16:14 (1786166174) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:16:26 (1786166186) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:16:29 (1786166189) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:16:32 (1786166192) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:16:58 (1786166218) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:17:02 (1786166222) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:17:06 (1786166226) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9cc9c8d13000.stats=clear osc.lustre-OST0001-osc-ffff9cc9c8d13000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:17:14 (1786166234) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:17:27 (1786166247) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:17:37 (1786166257) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:17:41 (1786166261) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:17:51 (1786166271) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 01:18:01 (1786166281) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:18:08 (1786166288) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:18:12 (1786166292) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:18:22 (1786166302) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:18:33 (1786166313) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:18:40 (1786166320) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.892111 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110948 s, 9.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:18:51 (1786166331) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.361264 s, 29.0 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:18:59 (1786166339) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:19:09 (1786166349) Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg158-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg158-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (134s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:21:22 (1786166482) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.615504 s, 17.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:22:04 (1786166524) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg158-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:23:16 (1786166596) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.28889 s, 12.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 8 01:23 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.001s user 0m0.003s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cc9c8d13000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.062s user 0m0.000s sys 0m0.183s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg158-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg158-server oleg158-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.206s user 0m0.001s sys 0m0.165s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg158-server: oleg158-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg158-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cc9c8d13000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:24:51 (1786166691) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:24:55 (1786166695) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:24:58 (1786166698) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 01:25:01 (1786166701) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 01:25:10 (1786166710) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.635035 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:25:22 (1786166722) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317317 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327524 s, 32.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:25:29 (1786166729) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:25:38 (1786166738) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0053523 s, 191 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112167 s, 913 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000875681 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00187715 s, 546 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 01:25:48 (1786166748) create test files with size 15746587 .. 453411477da5d0035e1e1f9fe1641503 - 8d77f016a3a30d73c036e321bb2dfed5 - 3834672259b5591b25c8fc6a23dc868d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg158-client.virtnet /mnt/lustre (opts:) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 01:26:10 (1786166770) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:26:23 (1786166783) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:26:30 (1786166790) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965391 s, 10.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406017 s, 25.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 01:26:40 (1786166800) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115474 s, 18.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.343088 s, 12.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135557 s, 15.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 01:26:55 (1786166815) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:26:58 (1786166818) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:27:01 (1786166821) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421527 s, 24.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 01:27:09 (1786166829) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:27:12 (1786166832) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xbf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:27:22 (1786166842) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.288679 s, 36.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg158-client.virtnet /mnt/lustre (opts:) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:27:36 (1786166856) llite.lustre-ffff9cc9c37d4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.419193 s, 25.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg158-client.virtnet /mnt/lustre (opts:) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9cc9c84e9000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:27:51 (1786166871) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:28:30 (1786166910) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:29:11 (1786166951) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 01:29:15 (1786166955) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.208134 s, 15.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.173878 s, 18.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd130c28a (nosync,stale) Mirror 2: 0xd130c28a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 42a9e7a1fb1be7c9ee34a98c86ee35f6 - mirror 1 checksum: 12140ae5b96d94c9c928ba422e2a55eb - mirror 2 checksum: 42a9e7a1fb1be7c9ee34a98c86ee35f6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T01:29:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-08T01:29:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 42a9e7a1fb1be7c9ee34a98c86ee35f6 - mirror 1 checksum: 12140ae5b96d94c9c928ba422e2a55eb - mirror 2 checksum: 12140ae5b96d94c9c928ba422e2a55eb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T01:29:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-08T01:29:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:29:28 (1786166968) Failing mds1 on oleg158-server Stopping /mnt/lustre-mds1 (opts:) on oleg158-server 01:29:35 (1786166975) shut down facet: mds1 facet_host: oleg158-server facet_failover_host: oleg158-server Failover mds1 to oleg158-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 01:30:07 (1786167007) targets are mounted 01:30:07 (1786167007) facet_failover done oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302609 s, 34.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } PASS 50A (55s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:30:23 (1786167023) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:30:27 (1786167027) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:30:31 (1786167031) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00611489 s, 2.7 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:30:39 (1786167039) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:30:43 (1786167043) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0112216 s, 365 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0554723 s, 73.8 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:30:51 (1786167051) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:31:12 (1786167072) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:31:42 (1786167102) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:32:06 (1786167126) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:32:31 (1786167151) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 01:32:39 (1786167159) mirror create pid 54343 mirror split pid 54344 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 54343 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 54344 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 01:33:48 (1786167228) Seed set to 7230 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1786167230.952636 mapwrite 0x022e77 thru 0x02a738 (0x78c2 bytes) 000006 1786167231.145214 read 0x007ec9 thru 0x015ad4 (0xdc0c bytes) 000007 1786167231.153646 mapread 0x003bb3 thru 0x00ff80 (0xc3ce bytes) 000009 1786167231.170372 mapwrite 0x030ff9 thru 0x03e610 (0xd618 bytes) 000010 1786167231.314380 write 0x02d6cd thru 0x03152c (0x3e60 bytes) 000011 1786167231.316269 read 0x03dc13 thru 0x03e610 (0x9fe bytes) truncating to largest ever: 0x1acd8 000012 1786167231.319671 trunc from 0x03e611 to 0x01acd7 (0x23939 bytes) 000013 1786167231.342124 mapwrite 0x026bd7 thru 0x03182d (0xac57 bytes) 000016 1786167231.438133 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x30ddb 000019 1786167232.244058 trunc from 0x03182e to 0x030dda (0xa53 bytes) 000020 1786167232.391370 write 0x013439 thru 0x01b37e (0x7f46 bytes) 000021 1786167232.411828 read 0x00a6f5 thru 0x00a9be (0x2ca bytes) 000022 1786167232.420822 write 0x009f74 thru 0x0163c1 (0xc44e bytes) 000027 1786167232.437970 mapwrite 0x039896 thru 0x03ffff (0x676a bytes) 000030 1786167232.594001 mapread 0x01c489 thru 0x0286e9 (0xc261 bytes) 000031 1786167232.625997 read 0x000e0a thru 0x003660 (0x2857 bytes) 000032 1786167232.641091 read 0x02b782 thru 0x0351ee (0x9a6d bytes) 000033 1786167232.643329 mapwrite 0x005440 thru 0x00c58c (0x714d bytes) 000034 1786167232.727652 mapread 0x03542a thru 0x03965b (0x4232 bytes) 000035 1786167232.728834 trunc from 0x040000 to 0x006294 (0x39d6b bytes) 000036 1786167232.761608 mapread 0x002059 thru 0x006294 (0x423c bytes) 000038 1786167232.764801 write 0x03bfde thru 0x03f0c6 (0x30e9 bytes) 000040 1786167232.774934 mapread 0x0001cc thru 0x00f727 (0xf55c bytes) 000041 1786167232.796864 read 0x00dcd9 thru 0x00e762 (0xa8a bytes) 000042 1786167232.798010 write 0x013242 thru 0x016315 (0x30d4 bytes) 000043 1786167232.801011 read 0x013147 thru 0x01f62a (0xc4e4 bytes) truncating to largest ever: 0x33dd3 000044 1786167232.803244 trunc from 0x03f0c7 to 0x033dd2 (0xb2f4 bytes) 000045 1786167232.821565 trunc from 0x033dd3 to 0x0054c4 (0x2e90e bytes) 000046 1786167232.854006 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000047 1786167233.059702 mapwrite 0x03c04f thru 0x03ffff (0x3fb1 bytes) 000048 1786167233.169253 MIRROR_EXTEND to 2 mirrors 000049 1786167234.222557 mapread 0x039a01 thru 0x03ffff (0x65ff bytes) 000050 1786167234.266395 mapwrite 0x017b46 thru 0x0219c9 (0x9e84 bytes) truncating to largest ever: 0x3f5d9 000051 1786167234.466660 trunc from 0x040000 to 0x03f5d8 (0xa27 bytes) 000055 1786167234.546687 trunc from 0x03f5d9 to 0x00253f (0x3d099 bytes) 000058 1786167234.577186 mapwrite 0x02ebab thru 0x037cfe (0x9154 bytes) 000059 1786167234.708538 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000061 1786167234.884059 trunc from 0x037cff to 0x0329ff (0x52ff bytes) 000063 1786167234.899716 mapwrite 0x01259d thru 0x013ae0 (0x1544 bytes) 000070 1786167234.972747 write 0x01909d thru 0x01c175 (0x30d9 bytes) 000071 1786167234.993387 write 0x00430d thru 0x00c72b (0x841f bytes) 000074 1786167235.010961 mapread 0x001d5a thru 0x00beca (0xa171 bytes) 000079 1786167235.070768 mapread 0x01b4dc thru 0x029bc1 (0xe6e6 bytes) 000080 1786167235.118963 mapwrite 0x0320e1 thru 0x037f5e (0x5e7e bytes) 000082 1786167235.217892 write 0x0163a8 thru 0x01ec0a (0x8863 bytes) 000084 1786167235.224753 trunc from 0x037f5f to 0x02554a (0x12a14 bytes) 000085 1786167235.243710 write 0x032cb4 thru 0x035e63 (0x31b0 bytes) 000086 1786167235.248406 trunc from 0x035e64 to 0x008c45 (0x2d21e bytes) 000095 1786167235.307605 trunc from 0x008c46 to 0x007a23 (0x1222 bytes) 000096 1786167235.326967 mapwrite 0x011792 thru 0x012405 (0xc74 bytes) 000100 1786167235.465293 mapwrite 0x03e7ef thru 0x03ffff (0x1811 bytes) 000101 1786167235.580797 trunc from 0x040000 to 0x02b001 (0x14ffe bytes) 000104 1786167235.618442 write 0x03c0d8 thru 0x03ffff (0x3f28 bytes) 000106 1786167235.622372 mapread 0x0011f9 thru 0x004229 (0x3031 bytes) 000107 1786167235.623951 trunc from 0x040000 to 0x006727 (0x398d8 bytes) 000108 1786167235.648094 trunc from 0x006728 to 0x0179a5 (0x1127e bytes) 000109 1786167235.665006 mapread 0x001cb0 thru 0x00a7d2 (0x8b23 bytes) 000110 1786167235.678056 trunc from 0x0179a6 to 0x03b7a1 (0x23dfc bytes) 000113 1786167235.708943 read 0x03b560 thru 0x03b7a1 (0x242 bytes) 000114 1786167235.715629 write 0x023ed4 thru 0x02b673 (0x77a0 bytes) 000115 1786167235.734677 read 0x01f6b2 thru 0x02d3f5 (0xdd44 bytes) 000116 1786167235.751876 write 0x0117c7 thru 0x018533 (0x6d6d bytes) 000119 1786167235.757195 trunc from 0x03b7a2 to 0x021322 (0x1a47f bytes) 000120 1786167235.785636 mapwrite 0x0249e8 thru 0x02dee9 (0x9502 bytes) 000122 1786167236.002133 MIRROR_EXTEND to 2 mirrors 000123 1786167237.150007 trunc from 0x02deea to 0x035f8f (0x80a6 bytes) 000124 1786167237.251118 mapread 0x03128a thru 0x0312df (0x56 bytes) 000125 1786167237.259274 write 0x033111 thru 0x038ce3 (0x5bd3 bytes) 000126 1786167237.265227 trunc from 0x038ce4 to 0x00b19e (0x2db45 bytes) 000127 1786167237.291585 mapwrite 0x007373 thru 0x011ca9 (0xa937 bytes) 000130 1786167237.431863 mapwrite 0x02af89 thru 0x030e6d (0x5ee5 bytes) 000131 1786167237.521607 write 0x020440 thru 0x021c9a (0x185b bytes) 000132 1786167237.545040 mapread 0x01e455 thru 0x023b4b (0x56f7 bytes) 000133 1786167237.574931 mapread 0x0210a9 thru 0x030a64 (0xf9bc bytes) 000137 1786167237.581905 mapread 0x0045f7 thru 0x00c852 (0x825c bytes) 000138 1786167237.585208 read 0x01448d thru 0x01d9ed (0x9561 bytes) 000140 1786167237.593275 read 0x028d73 thru 0x030e6d (0x80fb bytes) 000141 1786167237.599306 trunc from 0x030e6e to 0x01aead (0x15fc0 bytes) 000143 1786167237.652222 read 0x00758d thru 0x00e77e (0x71f2 bytes) 000144 1786167237.661472 mapwrite 0x011725 thru 0x015219 (0x3af5 bytes) 000145 1786167237.855060 trunc from 0x01aeae to 0x027913 (0xca66 bytes) 000148 1786167237.896557 write 0x02bc3f thru 0x032854 (0x6c16 bytes) 000152 1786167237.916207 mapwrite 0x007199 thru 0x00d0d2 (0x5f3a bytes) 000153 1786167238.004979 mapread 0x02c657 thru 0x032854 (0x61fe bytes) 000157 1786167238.014234 trunc from 0x032855 to 0x03cffa (0xa7a6 bytes) 000158 1786167238.062957 MIRROR_RESYNC 2 mirrors 000159 1786167239.244401 write 0x039eb0 thru 0x03ffff (0x6150 bytes) 000162 1786167239.445651 mapwrite 0x0362ad thru 0x03ffff (0x9d53 bytes) 000163 1786167239.682275 write 0x0162bc thru 0x021668 (0xb3ad bytes) 000166 1786167239.760884 write 0x03dcaa thru 0x03ffff (0x2356 bytes) 000168 1786167239.786210 mapwrite 0x0262ee thru 0x034ac2 (0xe7d5 bytes) 000170 1786167239.980222 mapwrite 0x02a45f thru 0x02bdde (0x1980 bytes) 000173 1786167240.066416 read 0x019eac thru 0x025d3d (0xbe92 bytes) 000174 1786167240.068099 read 0x0296d1 thru 0x036b1b (0xd44b bytes) 000175 1786167240.069061 mapwrite 0x03d004 thru 0x03d529 (0x526 bytes) 000176 1786167240.115971 mapread 0x00dc9a thru 0x01696c (0x8cd3 bytes) 000177 1786167240.127059 write 0x01de11 thru 0x020e8f (0x307f bytes) 000178 1786167240.128220 mapwrite 0x018e56 thru 0x01c45c (0x3607 bytes) 000179 1786167240.170181 mapread 0x024cf3 thru 0x02701d (0x232b bytes) 000180 1786167240.171280 mapwrite 0x01498b thru 0x023e4d (0xf4c3 bytes) 000182 1786167240.219099 trunc from 0x040000 to 0x02240b (0x1dbf4 bytes) 000184 1786167240.250483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000185 1786167240.492371 mapread 0x01666d thru 0x0223e5 (0xbd79 bytes) 000187 1786167240.554161 read 0x0142c0 thru 0x01e458 (0xa199 bytes) 000189 1786167240.555472 trunc from 0x02240c to 0x03f053 (0x1cc48 bytes) 000190 1786167240.604531 write 0x03d908 thru 0x03ffff (0x26f8 bytes) 000191 1786167240.619092 trunc from 0x040000 to 0x03ddfb (0x2204 bytes) 000193 1786167240.697967 trunc from 0x03ddfc to 0x00d0b7 (0x30d44 bytes) 000194 1786167240.752459 mapread 0x0062fd thru 0x009a91 (0x3795 bytes) 000197 1786167240.790354 mapwrite 0x038375 thru 0x03ffff (0x7c8b bytes) 000199 1786167241.022624 read 0x033e76 thru 0x03ebb7 (0xad42 bytes) 000201 1786167241.053316 mapread 0x03ca0b thru 0x03ffff (0x35f5 bytes) 000203 1786167241.055071 trunc from 0x040000 to 0x014804 (0x2b7fb bytes) 000204 1786167241.090306 write 0x012851 thru 0x01871d (0x5ecd bytes) 000205 1786167241.132920 read 0x00524c thru 0x00dafa (0x88af bytes) 000206 1786167241.145342 trunc from 0x01871e to 0x000394 (0x18389 bytes) 000207 1786167241.185076 write 0x024084 thru 0x028e4a (0x4dc7 bytes) 000210 1786167241.186979 trunc from 0x028e4b to 0x00fe50 (0x18ffa bytes) 000211 1786167241.226214 mapwrite 0x013b99 thru 0x02219f (0xe607 bytes) 000214 1786167241.378197 mapread 0x01e525 thru 0x02219f (0x3c7b bytes) 000217 1786167241.382606 trunc from 0x0221a0 to 0x023f82 (0x1de3 bytes) 000219 1786167241.425758 trunc from 0x023f83 to 0x02892a (0x49a8 bytes) 000220 1786167241.452209 mapwrite 0x012c05 thru 0x01e1b9 (0xb5b5 bytes) 000221 1786167241.517476 write 0x0258fb thru 0x0261c6 (0x8cc bytes) 000222 1786167241.530814 MIRROR_EXTEND to 2 mirrors 000226 1786167242.279002 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000231 1786167242.430273 mapread 0x01a31f thru 0x02892a (0xe60c bytes) 000232 1786167242.461719 trunc from 0x02892b to 0x03bc33 (0x13309 bytes) 000233 1786167242.498455 mapread 0x02ff06 thru 0x03bc33 (0xbd2e bytes) 000235 1786167242.519712 trunc from 0x03bc34 to 0x00ffd5 (0x2bc5e bytes) 000240 1786167242.550120 read 0x008a9f thru 0x00b336 (0x2898 bytes) 000242 1786167242.555114 read 0x002272 thru 0x007bf0 (0x597f bytes) 000243 1786167242.566143 write 0x02885a thru 0x034b2c (0xc2d3 bytes) 000244 1786167242.568203 write 0x03df8a thru 0x03ffff (0x2076 bytes) 000247 1786167242.572741 mapread 0x00d0ee thru 0x01b3ab (0xe2be bytes) 000248 1786167242.582116 read 0x0322db thru 0x03f813 (0xd539 bytes) 000249 1786167242.583058 mapread 0x029bf2 thru 0x033e01 (0xa210 bytes) 000250 1786167242.585005 mapwrite 0x03e5d2 thru 0x03ffff (0x1a2e bytes) 000251 1786167242.684817 trunc from 0x040000 to 0x02e8a2 (0x1175d bytes) 000252 1786167242.721433 read 0x02108c thru 0x029606 (0x857b bytes) 000255 1786167242.729758 mapwrite 0x014c66 thru 0x016cd3 (0x206e bytes) 000257 1786167242.823162 MIRROR_EXTEND to 2 mirrors 000259 1786167243.718799 read 0x01bcd2 thru 0x0283e2 (0xc711 bytes) 000261 1786167243.788807 read 0x021d0a thru 0x02b2f5 (0x95ec bytes) 000262 1786167243.819667 write 0x027ed3 thru 0x033e66 (0xbf94 bytes) 000263 1786167243.924223 mapwrite 0x0216b8 thru 0x02ba5c (0xa3a5 bytes) 000265 1786167244.016788 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000266 1786167244.151663 write 0x032ae3 thru 0x039061 (0x657f bytes) 000267 1786167244.158699 mapread 0x033de6 thru 0x039061 (0x527c bytes) 000269 1786167244.162062 trunc from 0x039062 to 0x00a733 (0x2e92e bytes) 000270 1786167244.184467 mapwrite 0x02a023 thru 0x0328b2 (0x8890 bytes) 000271 1786167244.270600 mapread 0x01690f thru 0x01eae6 (0x81d8 bytes) 000274 1786167244.282403 mapwrite 0x01937b thru 0x01bee5 (0x2b6b bytes) 000277 1786167244.354093 write 0x0329da thru 0x03bf3c (0x9563 bytes) 000278 1786167244.357131 mapread 0x005763 thru 0x00bf28 (0x67c6 bytes) 000279 1786167244.372157 trunc from 0x03bf3d to 0x028df7 (0x13145 bytes) 000282 1786167244.401177 mapread 0x012e0e thru 0x01e1f6 (0xb3e9 bytes) 000283 1786167244.404792 mapread 0x027856 thru 0x027902 (0xad bytes) 000285 1786167244.408419 mapread 0x0079bd thru 0x01404d (0xc691 bytes) 000288 1786167244.412209 trunc from 0x028df8 to 0x0037d1 (0x25626 bytes) 000291 1786167244.432470 write 0x008bd7 thru 0x017439 (0xe863 bytes) 000292 1786167244.435558 mapwrite 0x021cb4 thru 0x02b538 (0x9885 bytes) 000293 1786167244.613124 trunc from 0x02b539 to 0x02df9b (0x2a63 bytes) 000294 1786167244.632974 mapwrite 0x01bf83 thru 0x01e350 (0x23ce bytes) 000295 1786167244.672422 mapwrite 0x005e5e thru 0x014bcb (0xed6e bytes) 000297 1786167244.810089 write 0x002f00 thru 0x007d48 (0x4e49 bytes) 000298 1786167244.813370 read 0x00e608 thru 0x00f032 (0xa2b bytes) 000299 1786167244.820127 mapwrite 0x034b83 thru 0x03ffff (0xb47d bytes) 000300 1786167244.947392 mapread 0x003fc0 thru 0x012e10 (0xee51 bytes) 000301 1786167244.959669 MIRROR_EXTEND to 2 mirrors 000304 1786167245.690631 mapread 0x03776c thru 0x03ffff (0x8894 bytes) 000306 1786167245.769458 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000307 1786167245.908935 read 0x00a35f thru 0x0101c0 (0x5e62 bytes) 000308 1786167245.917715 mapwrite 0x015c51 thru 0x023335 (0xd6e5 bytes) 000310 1786167246.027695 mapwrite 0x013a2b thru 0x01ab3b (0x7111 bytes) 000311 1786167246.077331 write 0x037325 thru 0x03ce4d (0x5b29 bytes) 000317 1786167246.093104 mapread 0x003bad thru 0x00d35e (0x97b2 bytes) 000318 1786167246.103055 mapread 0x00c7ca thru 0x0169c6 (0xa1fd bytes) 000319 1786167246.104279 write 0x00b33a thru 0x0136c8 (0x838f bytes) 000322 1786167246.113201 read 0x03fa83 thru 0x03ffff (0x57d bytes) 000325 1786167246.125309 mapwrite 0x010299 thru 0x011635 (0x139d bytes) 000326 1786167246.206909 trunc from 0x040000 to 0x03b200 (0x4dff bytes) 000327 1786167246.225887 read 0x028552 thru 0x0366e3 (0xe192 bytes) 000328 1786167246.228802 write 0x034f0c thru 0x03b512 (0x6607 bytes) 000329 1786167246.235018 write 0x01f475 thru 0x027068 (0x7bf4 bytes) 000333 1786167246.248077 mapwrite 0x011854 thru 0x020a98 (0xf245 bytes) 000334 1786167246.357311 write 0x02bcde thru 0x02f40c (0x372f bytes) 000337 1786167246.360873 mapread 0x0298d5 thru 0x02db72 (0x429e bytes) 000339 1786167246.365991 mapread 0x032d28 thru 0x03b512 (0x87eb bytes) 000340 1786167246.367272 write 0x02a49a thru 0x02e4e3 (0x404a bytes) 000344 1786167246.371105 write 0x03d930 thru 0x03ffff (0x26d0 bytes) 000345 1786167246.373125 trunc from 0x040000 to 0x02a216 (0x15de9 bytes) 000348 1786167246.393417 mapwrite 0x003de0 thru 0x0047de (0x9ff bytes) 000350 1786167246.439042 trunc from 0x02a217 to 0x02104c (0x91ca bytes) 000354 1786167246.476452 trunc from 0x02104d to 0x03b993 (0x1a947 bytes) 000355 1786167246.495423 read 0x01bfae thru 0x01e27e (0x22d1 bytes) 000358 1786167246.499757 write 0x023751 thru 0x02a265 (0x6b15 bytes) 000359 1786167246.513104 read 0x0249d9 thru 0x02be9b (0x74c3 bytes) 000360 1786167246.514196 mapread 0x006b37 thru 0x00b30a (0x47d4 bytes) 000362 1786167246.517306 trunc from 0x03b994 to 0x0286e9 (0x132aa bytes) 000364 1786167246.540063 read 0x00c360 thru 0x019ffd (0xdc9e bytes) 000368 1786167246.544084 mapwrite 0x00a0bf thru 0x00dcb3 (0x3bf5 bytes) 000371 1786167246.588194 trunc from 0x0286ea to 0x0004b8 (0x28231 bytes) 000372 1786167246.606664 write 0x00f438 thru 0x017125 (0x7cee bytes) 000373 1786167246.615469 MIRROR_EXTEND to 2 mirrors 000374 1786167247.445487 write 0x039d11 thru 0x03ca7e (0x2d6e bytes) 000376 1786167247.543057 write 0x018513 thru 0x0222f0 (0x9dde bytes) 000378 1786167247.563240 mapread 0x03c353 thru 0x03ca7e (0x72c bytes) 000380 1786167247.570476 trunc from 0x03ca7f to 0x00d93b (0x2f143 bytes) 000381 1786167247.609752 read 0x003155 thru 0x00d93b (0xa7e7 bytes) 000382 1786167247.638618 mapread 0x00328a thru 0x00d8cd (0xa644 bytes) 000385 1786167247.661561 MIRROR_RESYNC 2 mirrors 000386 1786167248.874541 MIRROR_EXTEND to 3 mirrors 000387 1786167249.849688 read 0x004975 thru 0x00a132 (0x57be bytes) 000388 1786167249.861493 trunc from 0x00d93c to 0x029173 (0x1b838 bytes) 000389 1786167249.959020 trunc from 0x029174 to 0x021597 (0x7bdc bytes) 000391 1786167249.976003 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21598): Mirror 3: 0xe1f8858c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21598): Mirror 3: 0xe1f8858c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21598: skipped 000392 1786167250.214344 mapread 0x0191fd thru 0x021597 (0x839b bytes) 000394 1786167250.240964 mapwrite 0x030cb5 thru 0x036511 (0x585d bytes) 000395 1786167250.334034 trunc from 0x036512 to 0x00d5c1 (0x28f50 bytes) 000397 1786167250.365023 read 0x003983 thru 0x00d2ba (0x9938 bytes) 000398 1786167250.380992 trunc from 0x00d5c2 to 0x01289b (0x52da bytes) 000403 1786167250.426200 write 0x005b6c thru 0x008b3c (0x2fd1 bytes) 000404 1786167250.427725 read 0x00466b thru 0x01289b (0xe231 bytes) 000405 1786167250.469743 write 0x010680 thru 0x01ba7d (0xb3fe bytes) 000406 1786167250.471299 write 0x0315e9 thru 0x0366a4 (0x50bc bytes) 000408 1786167250.484587 mapwrite 0x014dc1 thru 0x023d62 (0xefa2 bytes) 000410 1786167250.590391 trunc from 0x0366a5 to 0x011ce0 (0x249c4 bytes) 000412 1786167250.629589 mapread 0x00d521 thru 0x011ce0 (0x47c0 bytes) 000413 1786167250.630723 read 0x010919 thru 0x011ce0 (0x13c8 bytes) 000416 1786167250.639428 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11ce1): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11ce1): Mirror 3: 0xacbb1fbe (stale) CRC-32 checksum value for chunk [0, 0x11ce1): Mirror 1: 0xe3312b66 Mirror 4: 0xe3312b66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11ce1: skipped 000421 1786167251.738242 write 0x0048fb thru 0x00c873 (0x7f79 bytes) 000423 1786167251.821945 mapread 0x00e9af thru 0x011ce0 (0x3332 bytes) 000429 1786167251.848103 mapread 0x00ae8f thru 0x011ce0 (0x6e52 bytes) 000430 1786167251.849631 mapread 0x01069e thru 0x011ce0 (0x1643 bytes) 000431 1786167251.851262 write 0x006e5c thru 0x00ebe2 (0x7d87 bytes) 000433 1786167251.853587 mapread 0x00c47c thru 0x011ce0 (0x5865 bytes) 000435 1786167251.856069 trunc from 0x011ce1 to 0x01f711 (0xda31 bytes) 000436 1786167251.887361 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f712): Mirror 3: 0x16151503 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f712): Mirror 3: 0x16151503 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f712: skipped 000437 1786167252.186604 read 0x016919 thru 0x01f711 (0x8df9 bytes) 000440 1786167252.195728 read 0x01a176 thru 0x01f711 (0x559c bytes) 000441 1786167252.196230 mapread 0x01784b thru 0x01c8a3 (0x5059 bytes) 000443 1786167252.202416 MIRROR_RESYNC 2 mirrors 000444 1786167252.941833 trunc from 0x01f712 to 0x004375 (0x1b39c bytes) 000445 1786167253.024351 mapwrite 0x03a49c thru 0x03f95b (0x54c0 bytes) 000446 1786167253.113654 trunc from 0x03f95c to 0x03a182 (0x57d9 bytes) 000447 1786167253.133764 mapwrite 0x037366 thru 0x03c02e (0x4cc9 bytes) 000448 1786167253.210346 mapwrite 0x01f610 thru 0x02990b (0xa2fc bytes) 000450 1786167253.318854 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c02f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c02f): Mirror 3: 0x85512c0d (stale) CRC-32 checksum value for chunk [0, 0x3c02f): Mirror 1: 0x31f61769 Mirror 4: 0x31f61769 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c02f: skipped 000453 1786167254.414627 mapread 0x01dcc3 thru 0x021cb1 (0x3fef bytes) 000454 1786167254.459238 MIRROR_RESYNC 3 mirrors 000455 1786167255.537378 write 0x02580f thru 0x02c628 (0x6e1a bytes) 000458 1786167255.641833 mapread 0x01d5ff thru 0x023c5f (0x6661 bytes) 000460 1786167255.667949 mapwrite 0x02c527 thru 0x035947 (0x9421 bytes) 000461 1786167255.752786 read 0x030320 thru 0x032461 (0x2142 bytes) 000462 1786167255.753489 read 0x016273 thru 0x01764b (0x13d9 bytes) 000463 1786167255.770725 trunc from 0x03c02f to 0x000590 (0x3ba9e bytes) 000464 1786167255.800801 read 0x0004a3 thru 0x000590 (0xee bytes) 000466 1786167255.809945 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x591): Mirror 4: 0x62427922 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x591): Mirror 4: 0x62427922 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x591: skipped 000467 1786167256.181672 mapwrite 0x00115e thru 0x001412 (0x2b5 bytes) 000469 1786167256.256569 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000470 1786167256.441632 MIRROR_EXTEND to 2 mirrors 000471 1786167257.115303 write 0x00c795 thru 0x014c9f (0x850b bytes) 000475 1786167257.200391 mapread 0x0088a9 thru 0x010f25 (0x867d bytes) 000476 1786167257.231777 mapread 0x0146b5 thru 0x014c9f (0x5eb bytes) 000477 1786167257.234390 mapwrite 0x0072e2 thru 0x0140b8 (0xcdd7 bytes) 000481 1786167257.301376 write 0x03a4f9 thru 0x03ffff (0x5b07 bytes) 000483 1786167257.306052 trunc from 0x040000 to 0x023c4b (0x1c3b4 bytes) 000485 1786167257.338445 write 0x020fc6 thru 0x026967 (0x59a2 bytes) 000487 1786167257.347678 mapread 0x007731 thru 0x00f9ad (0x827d bytes) 000488 1786167257.355334 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26968): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26968): Mirror 2: 0xb9939a7a (stale) CRC-32 checksum value for chunk [0, 0x26968): Mirror 1: 0x16ed52f1 Mirror 3: 0x16ed52f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26968: skipped 000490 1786167258.231481 trunc from 0x026968 to 0x006e55 (0x1fb12 bytes) 000491 1786167258.395823 mapwrite 0x0122eb thru 0x01d763 (0xb479 bytes) 000493 1786167258.530449 read 0x00f5e3 thru 0x01283e (0x325c bytes) 000494 1786167258.534060 write 0x008333 thru 0x013a91 (0xb75f bytes) 000496 1786167258.571732 read 0x017eeb thru 0x01c1ee (0x4304 bytes) 000497 1786167258.594307 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1d764): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d764): Mirror 2: 0x10244ea8 (stale) Mirror 3: 0x17009e86 (stale) CRC-32 checksum value for chunk [0, 0x1d764): Mirror 1: 0xa0c9666f Mirror 4: 0xa0c9666f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d764: skipped 000498 1786167259.688751 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d764): Mirror 2: 0x10244ea8 (stale) Mirror 3: 0x17009e86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d764): Mirror 2: 0x10244ea8 (stale) Mirror 3: 0x17009e86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d764: skipped 000500 1786167260.029300 read 0x001c0b thru 0x002dde (0x11d4 bytes) 000501 1786167260.044532 mapread 0x011d76 thru 0x0134c1 (0x174c bytes) 000502 1786167260.066078 read 0x00d27d thru 0x00e996 (0x171a bytes) 000505 1786167260.075386 write 0x032dbf thru 0x03ffff (0xd241 bytes) 000506 1786167260.100896 mapread 0x034118 thru 0x038e9d (0x4d86 bytes) 000508 1786167260.111775 MIRROR_RESYNC 3 mirrors 000510 1786167261.567220 read 0x004042 thru 0x0089b4 (0x4973 bytes) 000511 1786167261.579072 read 0x008489 thru 0x00c986 (0x44fe bytes) 000512 1786167261.601488 MIRROR_RESYNC 3 mirrors 000513 1786167262.067301 MIRROR_EXTEND to 4 mirrors 000514 1786167263.519712 write 0x00e445 thru 0x0124ec (0x40a8 bytes) 000515 1786167263.613373 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22524e09 (stale) Mirror 3: 0x22524e09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22524e09 (stale) Mirror 3: 0x22524e09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000518 1786167263.935353 write 0x024165 thru 0x0301f3 (0xc08f bytes) 000521 1786167263.953895 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22524e09 (stale) Mirror 3: 0x22524e09 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x76d20654 Mirror 4: 0x76d20654 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000524 1786167264.984082 mapread 0x0308d7 thru 0x033dc0 (0x34ea bytes) 000525 1786167265.070209 MIRROR_RESYNC 4 mirrors 000530 1786167265.938356 trunc from 0x040000 to 0x009da1 (0x3625e bytes) 000532 1786167266.166817 write 0x0102e9 thru 0x0201c9 (0xfee1 bytes) 000533 1786167266.168370 read 0x012d0f thru 0x01c337 (0x9629 bytes) 000534 1786167266.169817 write 0x02d45e thru 0x03cfa1 (0xfb44 bytes) 000540 1786167266.178403 mapwrite 0x016b60 thru 0x021f47 (0xb3e8 bytes) 000541 1786167266.290973 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfa2): Mirror 2: 0x8d737eef (stale) Mirror 3: 0x8d737eef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfa2): Mirror 2: 0x8d737eef (stale) Mirror 3: 0x8d737eef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfa2: skipped 000543 1786167266.684400 mapwrite 0x020267 thru 0x02da3c (0xd7d6 bytes) 000545 1786167266.858622 read 0x030115 thru 0x034ff3 (0x4edf bytes) 000546 1786167266.868271 read 0x031543 thru 0x038f1e (0x79dc bytes) 000548 1786167266.896687 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cfa2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfa2): Mirror 2: 0x8d737eef (stale) Mirror 3: 0x8d737eef (stale) CRC-32 checksum value for chunk [0, 0x3cfa2): Mirror 1: 0x16e00d62 Mirror 4: 0x16e00d62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfa2: skipped 000550 1786167268.143299 MIRROR_RESYNC 4 mirrors 000551 1786167269.082720 trunc from 0x03cfa2 to 0x032cb9 (0xa2e8 bytes) 000553 1786167269.196209 write 0x03806e thru 0x03a7c4 (0x2757 bytes) 000555 1786167269.201119 write 0x033ef9 thru 0x03e74d (0xa855 bytes) 000557 1786167269.228422 MIRROR_RESYNC 4 mirrors 000559 1786167270.443694 mapwrite 0x0226a0 thru 0x0315b1 (0xef12 bytes) 000562 1786167270.820990 mapread 0x0173f8 thru 0x01b99b (0x45a4 bytes) 000563 1786167270.843277 read 0x026c26 thru 0x034f02 (0xe2dd bytes) 000564 1786167270.848768 read 0x0041e4 thru 0x00a282 (0x609f bytes) 000565 1786167270.866491 trunc from 0x03e74e to 0x007d06 (0x36a47 bytes) 000566 1786167270.896264 trunc from 0x007d07 to 0x026556 (0x1e850 bytes) 000567 1786167270.924138 mapread 0x0077f5 thru 0x013f08 (0xc714 bytes) 000571 1786167271.092679 read 0x01db84 thru 0x023614 (0x5a91 bytes) 000572 1786167271.108719 mapwrite 0x0369ca thru 0x03ffff (0x9636 bytes) 000573 1786167271.212100 read 0x008ed8 thru 0x01391e (0xaa47 bytes) 000574 1786167271.218613 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe1bdce90 (stale) Mirror 3: 0xe1bdce90 (stale) Mirror 4: 0xe1bdce90 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x124197d Mirror 5: 0x124197d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000575 1786167272.639199 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe1bdce90 (stale) Mirror 3: 0xe1bdce90 (stale) Mirror 4: 0xe1bdce90 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x124197d Mirror 5: 0x124197d Mirror 6: 0x124197d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000579 1786167273.893467 read 0x024ba2 thru 0x0282ad (0x370c bytes) 000580 1786167273.904227 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe1bdce90 (stale) Mirror 4: 0xe1bdce90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe1bdce90 (stale) Mirror 4: 0xe1bdce90 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x124197d Mirror 5: 0x124197d Mirror 6: 0x124197d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000582 1786167274.526165 trunc from 0x040000 to 0x0293ae (0x16c51 bytes) 000588 1786167274.642892 read 0x024322 thru 0x028f27 (0x4c06 bytes) 000589 1786167274.655665 mapwrite 0x012388 thru 0x012b62 (0x7db bytes) 000590 1786167274.711380 mapread 0x01775d thru 0x01f799 (0x803d bytes) 000598 1786167274.737961 read 0x018273 thru 0x021c46 (0x99d4 bytes) 000599 1786167274.747717 read 0x0073c8 thru 0x010246 (0x8e7f bytes) 000600 1786167274.754595 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x293af): Mirror 2: 0xbf2ea2f8 (stale) Mirror 4: 0xbf2ea2f8 (stale) Mirror 5: 0xcfae94af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293af): Mirror 2: 0xbf2ea2f8 (stale) Mirror 4: 0xbf2ea2f8 (stale) Mirror 5: 0xcfae94af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293af: skipped 000601 1786167275.125105 write 0x005f30 thru 0x014631 (0xe702 bytes) 000602 1786167275.145156 read 0x01be42 thru 0x0293ae (0xd56d bytes) 000603 1786167275.162020 MIRROR_RESYNC 4 mirrors 000604 1786167276.224002 mapread 0x00cd27 thru 0x012e21 (0x60fb bytes) 000606 1786167276.262639 read 0x015f39 thru 0x01c711 (0x67d9 bytes) 000608 1786167276.279598 read 0x02755e thru 0x0293ae (0x1e51 bytes) 000610 1786167276.290035 mapwrite 0x00dd85 thru 0x010d12 (0x2f8e bytes) 000611 1786167276.612764 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x293af): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293af): Mirror 2: 0x4d563cf2 (stale) Mirror 4: 0x4d563cf2 (stale) Mirror 5: 0x4d563cf2 (stale) CRC-32 checksum value for chunk [0, 0x293af): Mirror 1: 0xa00c563b Mirror 6: 0xa00c563b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293af: skipped 000612 1786167277.491972 mapread 0x011f92 thru 0x01f1f7 (0xd266 bytes) 000614 1786167277.543375 trunc from 0x0293af to 0x021ef6 (0x74b8 bytes) 000620 1786167277.621543 read 0x019836 thru 0x021ef6 (0x86c1 bytes) 000622 1786167277.628579 write 0x03678a thru 0x0394e9 (0x2d60 bytes) 000623 1786167277.629432 mapwrite 0x027463 thru 0x029b12 (0x26b0 bytes) 000625 1786167277.733755 mapwrite 0x026064 thru 0x035d62 (0xfcff bytes) 000626 1786167277.765184 read 0x009f26 thru 0x0164b4 (0xc58f bytes) 000627 1786167277.775310 mapread 0x0021d5 thru 0x004727 (0x2553 bytes) 000628 1786167277.776271 mapwrite 0x02ac37 thru 0x0356a4 (0xaa6e bytes) 000631 1786167277.897998 mapread 0x00dfb9 thru 0x011504 (0x354c bytes) 000632 1786167277.907270 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x394ea): Mirror 2: 0x93058be4 (stale) Mirror 4: 0x93058be4 (stale) Mirror 5: 0x93058be4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394ea): Mirror 2: 0x93058be4 (stale) Mirror 4: 0x93058be4 (stale) Mirror 5: 0x93058be4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394ea: skipped 000633 1786167278.485994 mapread 0x03383c thru 0x0394e9 (0x5cae bytes) 000635 1786167278.504193 read 0x004f1f thru 0x0119c1 (0xcaa3 bytes) 000636 1786167278.516301 MIRROR_RESYNC 4 mirrors 000637 1786167279.253487 write 0x03d94b thru 0x03ffff (0x26b5 bytes) 000638 1786167279.362362 mapread 0x009013 thru 0x00b52a (0x2518 bytes) 000641 1786167279.406454 write 0x00c629 thru 0x00ee9d (0x2875 bytes) 000642 1786167279.411076 write 0x03531d thru 0x03d89a (0x857e bytes) 000643 1786167279.425573 read 0x009d63 thru 0x00cca0 (0x2f3e bytes) 000644 1786167279.426245 read 0x01a565 thru 0x01c252 (0x1cee bytes) 000645 1786167279.442366 trunc from 0x040000 to 0x00ec44 (0x313bb bytes) 000646 1786167279.481001 mapwrite 0x03018a thru 0x039268 (0x90df bytes) 000647 1786167279.649393 trunc from 0x039269 to 0x03180b (0x7a5d bytes) 000648 1786167279.670669 trunc from 0x03180c to 0x0145a0 (0x1d26b bytes) 000649 1786167279.698496 trunc from 0x0145a1 to 0x0063e6 (0xe1ba bytes) 000650 1786167279.733390 read 0x002a21 thru 0x0036b0 (0xc90 bytes) 000653 1786167279.746786 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x63e7): Mirror 4: 0xd2a6ba21 (stale) Mirror 5: 0xd2a6ba21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x63e7): Mirror 4: 0xd2a6ba21 (stale) Mirror 5: 0xd2a6ba21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x63e7: skipped 000654 1786167280.108761 read 0x0053c0 thru 0x0063e6 (0x1027 bytes) 000655 1786167280.117280 read 0x00265f thru 0x0063e6 (0x3d88 bytes) 000656 1786167280.150814 mapread 0x0057fe thru 0x0063e6 (0xbe9 bytes) 000657 1786167280.152097 write 0x01dab6 thru 0x02d976 (0xfec1 bytes) 000658 1786167280.159242 trunc from 0x02d977 to 0x005926 (0x28050 bytes) 000660 1786167280.184488 mapwrite 0x02e079 thru 0x0387d6 (0xa75e bytes) 000663 1786167280.312948 mapread 0x001390 thru 0x009b22 (0x8793 bytes) 000664 1786167280.329055 mapwrite 0x02723e thru 0x02a005 (0x2dc8 bytes) 000667 1786167280.401235 write 0x00c49d thru 0x00f070 (0x2bd4 bytes) 000668 1786167280.408451 read 0x00901d thru 0x0159c5 (0xc9a9 bytes) 000669 1786167280.411601 mapwrite 0x002d8e thru 0x0055f2 (0x2865 bytes) 000670 1786167280.572041 mapwrite 0x036377 thru 0x03e9ff (0x8689 bytes) 000674 1786167280.741221 mapread 0x0323e3 thru 0x038040 (0x5c5e bytes) 000675 1786167280.752311 read 0x03623a thru 0x03e9ff (0x87c6 bytes) 000676 1786167280.753197 mapread 0x01059f thru 0x01c7bd (0xc21f bytes) 000680 1786167280.780347 read 0x027309 thru 0x02b645 (0x433d bytes) 000682 1786167280.786604 trunc from 0x03ea00 to 0x0065a6 (0x38459 bytes) 000684 1786167280.832728 mapread 0x003217 thru 0x0065a6 (0x3390 bytes) 000685 1786167280.837409 trunc from 0x0065a7 to 0x02984e (0x232a8 bytes) 000686 1786167280.879670 write 0x01c7c9 thru 0x026c67 (0xa49f bytes) 000688 1786167280.914354 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2984f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2984f): Mirror 4: 0xaedfb1d5 (stale) Mirror 5: 0xaedfb1d5 (stale) CRC-32 checksum value for chunk [0, 0x2984f): Mirror 1: 0x81d68ce4 Mirror 6: 0x81d68ce4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2984f: skipped 000689 1786167282.524949 mapread 0x00a249 thru 0x017264 (0xd01c bytes) 000690 1786167282.594678 read 0x00362d thru 0x00bd50 (0x8724 bytes) 000691 1786167282.607592 mapread 0x013cb6 thru 0x01e5d5 (0xa920 bytes) 000692 1786167282.632780 write 0x01678a thru 0x023364 (0xcbdb bytes) 000693 1786167282.741044 mapwrite 0x027366 thru 0x028068 (0xd03 bytes) 000694 1786167282.803080 trunc from 0x02984f to 0x03d313 (0x13ac5 bytes) 000696 1786167282.855308 MIRROR_RESYNC 4 mirrors 000697 1786167284.461834 trunc from 0x03d314 to 0x0076b0 (0x35c63 bytes) 000698 1786167284.518176 trunc from 0x0076b1 to 0x02c3f2 (0x24d42 bytes) 000699 1786167284.539105 mapread 0x028150 thru 0x02c3f2 (0x42a3 bytes) 000700 1786167284.562302 mapwrite 0x00b376 thru 0x01b06f (0xfcfa bytes) 000701 1786167284.636783 read 0x001d01 thru 0x00aa82 (0x8d82 bytes) 000703 1786167284.643965 trunc from 0x02c3f3 to 0x02ec10 (0x281e bytes) 000706 1786167284.687989 MIRROR_RESYNC 4 mirrors 000707 1786167286.342783 write 0x03972c thru 0x03ffff (0x68d4 bytes) 000708 1786167286.493210 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6a47faf6 (stale) Mirror 5: 0x6a47faf6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6a47faf6 (stale) Mirror 5: 0x6a47faf6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000709 1786167287.118788 read 0x003742 thru 0x012c72 (0xf531 bytes) 000711 1786167287.148407 read 0x03a4b1 thru 0x03ffff (0x5b4f bytes) 000713 1786167287.203340 mapread 0x007f5c thru 0x00ee2a (0x6ecf bytes) 000714 1786167287.215400 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6a47faf6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x6a47faf6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000717 1786167287.712932 write 0x01b6d3 thru 0x01da1c (0x234a bytes) 000718 1786167287.754039 write 0x025e31 thru 0x031a20 (0xbbf0 bytes) 000720 1786167287.802458 MIRROR_RESYNC 2 mirrors 000721 1786167289.124768 mapread 0x03b25b thru 0x03fe2d (0x4bd3 bytes) 000724 1786167289.173324 mapwrite 0x0165e7 thru 0x020457 (0x9e71 bytes) 000725 1786167289.484193 mapwrite 0x0180c1 thru 0x02406f (0xbfaf bytes) 000727 1786167289.618273 MIRROR_RESYNC 2 mirrors 000728 1786167290.888346 write 0x03d578 thru 0x03ffff (0x2a88 bytes) 000729 1786167290.952854 mapwrite 0x035803 thru 0x038e9d (0x369b bytes) 000734 1786167291.020598 trunc from 0x040000 to 0x022c86 (0x1d379 bytes) 000735 1786167291.059392 mapwrite 0x000f75 thru 0x001d7b (0xe07 bytes) 000739 1786167291.113617 write 0x01a013 thru 0x026514 (0xc502 bytes) 000740 1786167291.120250 write 0x005ad8 thru 0x0126cf (0xcbf8 bytes) 000742 1786167291.129441 mapread 0x02261d thru 0x026514 (0x3ef8 bytes) 000743 1786167291.136321 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26515): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26515): Mirror 5: 0x6c0098bd (stale) CRC-32 checksum value for chunk [0, 0x26515): Mirror 1: 0x27fc6bb3 Mirror 6: 0x27fc6bb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26515: skipped 000744 1786167292.050838 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26515): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26515): Mirror 5: 0x6c0098bd (stale) CRC-32 checksum value for chunk [0, 0x26515): Mirror 1: 0x27fc6bb3 Mirror 6: 0x27fc6bb3 Mirror 7: 0x27fc6bb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26515: skipped 000745 1786167293.389042 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000746 1786167293.709365 trunc from 0x026515 to 0x038f4f (0x12a3b bytes) 000747 1786167293.821184 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38f50): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f50): Mirror 6: 0x1f3d889a (stale) Mirror 7: 0x1f3d889a (stale) CRC-32 checksum value for chunk [0, 0x38f50): Mirror 1: 0x1f3d889a Mirror 8: 0x1f3d889a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f50: skipped 000749 1786167294.906468 mapwrite 0x031d29 thru 0x03805c (0x6334 bytes) 000750 1786167295.111803 trunc from 0x038f50 to 0x002468 (0x36ae7 bytes) 000751 1786167295.156247 read 0x001eba thru 0x00224b (0x392 bytes) 000752 1786167295.164539 write 0x006225 thru 0x013119 (0xcef5 bytes) 000755 1786167295.174985 write 0x0250ec thru 0x0302d5 (0xb1ea bytes) 000757 1786167295.185246 read 0x025632 thru 0x02edf8 (0x97c7 bytes) 000758 1786167295.191010 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302d6): Mirror 6: 0x62fd45e0 (stale) Mirror 7: 0x62fd45e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302d6): Mirror 6: 0x62fd45e0 (stale) Mirror 7: 0x62fd45e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302d6: skipped 000759 1786167295.719878 read 0x01e579 thru 0x027b31 (0x95b9 bytes) 000761 1786167295.733356 trunc from 0x0302d6 to 0x01dcd4 (0x12601 bytes) 000762 1786167295.768873 trunc from 0x01dcd5 to 0x0082cc (0x15a08 bytes) 000765 1786167295.794100 read 0x007147 thru 0x0082cc (0x1186 bytes) 000771 1786167295.814761 read 0x000907 thru 0x006db7 (0x64b1 bytes) 000772 1786167295.815835 read 0x0009b3 thru 0x000f9a (0x5e8 bytes) 000773 1786167295.817024 mapread 0x00297e thru 0x0029c8 (0x4b bytes) 000777 1786167295.824243 mapread 0x006149 thru 0x0082cc (0x2184 bytes) 000778 1786167295.827154 mapwrite 0x00ae2d thru 0x01229b (0x746f bytes) 000780 1786167296.083237 trunc from 0x01229c to 0x021507 (0xf26c bytes) 000781 1786167296.124266 trunc from 0x021508 to 0x0273ac (0x5ea5 bytes) 000785 1786167296.145927 mapwrite 0x02f377 thru 0x0304ca (0x1154 bytes) 000789 1786167296.231447 read 0x02e5e0 thru 0x0304ca (0x1eeb bytes) 000791 1786167296.233583 trunc from 0x0304cb to 0x022513 (0xdfb7 bytes) 000792 1786167296.252047 trunc from 0x022514 to 0x02d51f (0xb00c bytes) 000793 1786167296.293161 read 0x005329 thru 0x012f9e (0xdc76 bytes) 000798 1786167296.306231 trunc from 0x02d520 to 0x00a0a1 (0x2347e bytes) 000799 1786167296.352810 read 0x004488 thru 0x00a0a1 (0x5c1a bytes) 000800 1786167296.357586 mapread 0x0059e0 thru 0x005cf5 (0x316 bytes) 000803 1786167296.361782 trunc from 0x00a0a2 to 0x002ac6 (0x75db bytes) 000804 1786167296.390066 trunc from 0x002ac7 to 0x03c941 (0x39e7b bytes) 000805 1786167296.421615 write 0x02bfb1 thru 0x031c30 (0x5c80 bytes) 000807 1786167296.438813 mapwrite 0x033ca1 thru 0x038ff3 (0x5353 bytes) 000810 1786167296.563280 mapwrite 0x022625 thru 0x02f08e (0xca6a bytes) 000811 1786167296.643144 mapread 0x01c51c thru 0x023a67 (0x754c bytes) 000813 1786167296.649925 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c942): Mirror 7: 0xffeb1c8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c942): Mirror 7: 0xffeb1c8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c942: skipped 000815 1786167297.089103 trunc from 0x03c942 to 0x003795 (0x391ac bytes) 000816 1786167297.128417 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000817 1786167297.428641 mapread 0x0035c4 thru 0x003795 (0x1d2 bytes) 000818 1786167297.442725 write 0x034d73 thru 0x03b392 (0x6620 bytes) 000819 1786167297.444072 read 0x01c9f5 thru 0x02c195 (0xf7a1 bytes) 000824 1786167297.472739 mapwrite 0x035297 thru 0x03ffff (0xad69 bytes) 000825 1786167297.656459 trunc from 0x040000 to 0x03e9e9 (0x1616 bytes) 000826 1786167297.702437 write 0x036a36 thru 0x03ffff (0x95ca bytes) 000828 1786167297.704351 write 0x02e039 thru 0x039647 (0xb60f bytes) 000829 1786167297.735154 trunc from 0x040000 to 0x00de8d (0x32172 bytes) 000830 1786167297.796779 MIRROR_EXTEND to 2 mirrors 000832 1786167298.812546 mapread 0x009b2c thru 0x00de8d (0x4362 bytes) 000833 1786167298.923130 read 0x00d9eb thru 0x00de8d (0x4a3 bytes) 000835 1786167298.982962 read 0x005321 thru 0x0059f7 (0x6d7 bytes) 000836 1786167298.996767 trunc from 0x00de8e to 0x026fbd (0x19130 bytes) 000837 1786167299.248230 mapread 0x01f889 thru 0x023ee9 (0x4661 bytes) 000838 1786167299.296186 trunc from 0x026fbe to 0x00da85 (0x19538 bytes) 000840 1786167299.364993 MIRROR_RESYNC 2 mirrors 000841 1786167300.674384 mapwrite 0x00c417 thru 0x014b96 (0x8780 bytes) 000844 1786167301.074687 mapwrite 0x010881 thru 0x01d757 (0xced7 bytes) 000845 1786167301.283350 read 0x00c650 thru 0x010628 (0x3fd9 bytes) 000847 1786167301.292797 write 0x028bfc thru 0x02ebdb (0x5fe0 bytes) 000848 1786167301.300287 trunc from 0x02ebdc to 0x025198 (0x9a43 bytes) 000850 1786167301.335992 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25199): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25199): Mirror 2: 0x8b8fb4a7 (stale) CRC-32 checksum value for chunk [0, 0x25199): Mirror 1: 0x9772dc24 Mirror 3: 0x9772dc24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25199: skipped 000854 1786167302.703703 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000856 1786167303.163417 trunc from 0x025199 to 0x00a634 (0x1ab64 bytes) 000858 1786167303.320092 read 0x008a28 thru 0x00a634 (0x1c0d bytes) 000859 1786167303.353182 write 0x012b52 thru 0x0193d5 (0x6884 bytes) 000861 1786167303.376164 read 0x011fc8 thru 0x018953 (0x698c bytes) 000862 1786167303.416602 write 0x027bb5 thru 0x0337fa (0xbc46 bytes) 000863 1786167303.418191 read 0x008199 thru 0x00f8ed (0x7755 bytes) 000864 1786167303.424325 mapwrite 0x037f53 thru 0x03f1fc (0x72aa bytes) 000865 1786167303.645412 read 0x010946 thru 0x01164f (0xd0a bytes) 000866 1786167303.648123 write 0x005585 thru 0x0135eb (0xe067 bytes) 000869 1786167303.654209 read 0x0167d7 thru 0x020d07 (0xa531 bytes) 000874 1786167303.661156 trunc from 0x03f1fd to 0x009c78 (0x35584 bytes) 000875 1786167303.694411 mapread 0x00602f thru 0x009c78 (0x3c4a bytes) 000876 1786167303.695459 mapread 0x0044ae thru 0x009c78 (0x57cb bytes) 000877 1786167303.696535 mapread 0x001409 thru 0x009c78 (0x8870 bytes) 000880 1786167303.719631 trunc from 0x009c79 to 0x03eee7 (0x3526f bytes) 000882 1786167303.743717 trunc from 0x03eee8 to 0x01252b (0x2c9bc bytes) 000883 1786167303.762544 read 0x00a34b thru 0x01252b (0x81e1 bytes) 000884 1786167303.781674 MIRROR_RESYNC 2 mirrors 000885 1786167304.471475 mapwrite 0x03fb13 thru 0x03ffff (0x4ed bytes) 000886 1786167304.582452 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b7ffaa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbe232926 Mirror 4: 0xbe232926 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000889 1786167305.804370 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x65b7ffaa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbe232926 Mirror 4: 0xbe232926 Mirror 5: 0xbe232926 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000891 1786167307.298695 read 0x022b0b thru 0x027ef2 (0x53e8 bytes) 000893 1786167307.383659 mapread 0x011ca4 thru 0x016cf3 (0x5050 bytes) 000894 1786167307.513248 mapwrite 0x01f6d5 thru 0x028a47 (0x9373 bytes) 000897 1786167307.895755 read 0x019fb2 thru 0x01ad10 (0xd5f bytes) 000898 1786167307.896589 read 0x020f0b thru 0x0225b2 (0x16a8 bytes) 000899 1786167307.897250 mapwrite 0x01b82d thru 0x024278 (0x8a4c bytes) 000904 1786167308.021273 MIRROR_RESYNC 4 mirrors 000905 1786167309.910056 write 0x006e86 thru 0x012915 (0xba90 bytes) 000906 1786167310.020632 read 0x00afbd thru 0x010520 (0x5564 bytes) 000907 1786167310.035550 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x10cba27 (stale) Mirror 5: 0x10cba27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x10cba27 (stale) Mirror 5: 0x10cba27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000909 1786167310.487242 mapwrite 0x01eb01 thru 0x0291f3 (0xa6f3 bytes) 000910 1786167310.585961 trunc from 0x040000 to 0x01ff84 (0x2007b bytes) 000911 1786167310.620089 read 0x000dcd thru 0x00283b (0x1a6f bytes) 000912 1786167310.638457 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ff85): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff85): Mirror 3: 0xc3149631 (stale) Mirror 5: 0xc3149631 (stale) CRC-32 checksum value for chunk [0, 0x1ff85): Mirror 1: 0x610be227 Mirror 6: 0x610be227 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff85: skipped 000913 1786167311.926027 write 0x031cce thru 0x03dd01 (0xc034 bytes) 000914 1786167312.043983 mapread 0x0226a8 thru 0x03266a (0xffc3 bytes) 000918 1786167312.089179 mapread 0x039943 thru 0x03aff1 (0x16af bytes) 000922 1786167312.106673 MIRROR_RESYNC 4 mirrors 000923 1786167313.705649 read 0x03539a thru 0x03dd01 (0x8968 bytes) 000924 1786167313.720141 trunc from 0x03dd02 to 0x00ea02 (0x2f2ff bytes) 000925 1786167313.830303 mapread 0x0057bb thru 0x00ea02 (0x9248 bytes) 000927 1786167313.878724 MIRROR_RESYNC 4 mirrors 000928 1786167315.048195 mapread 0x004c7b thru 0x006696 (0x1a1c bytes) 000929 1786167315.076002 write 0x010887 thru 0x015abc (0x5236 bytes) 000930 1786167315.178166 read 0x00fc44 thru 0x015abc (0x5e79 bytes) 000932 1786167315.193172 mapread 0x003310 thru 0x00bdd7 (0x8ac8 bytes) 000934 1786167315.214275 mapread 0x008f9f thru 0x00f3bb (0x641d bytes) 000935 1786167315.215568 mapread 0x009048 thru 0x015abc (0xca75 bytes) 000938 1786167315.221282 mapwrite 0x00e48c thru 0x00f065 (0xbda bytes) 000939 1786167315.276665 write 0x02273e thru 0x02865a (0x5f1d bytes) 000940 1786167315.284537 mapwrite 0x03f40c thru 0x03ffff (0xbf4 bytes) 000943 1786167315.390044 mapwrite 0x001283 thru 0x00eb38 (0xd8b6 bytes) 000944 1786167315.442104 read 0x032faf thru 0x038e40 (0x5e92 bytes) 000947 1786167315.457101 read 0x00662c thru 0x00dec9 (0x789e bytes) 000949 1786167315.458323 trunc from 0x040000 to 0x02ca12 (0x135ed bytes) 000950 1786167315.477921 trunc from 0x02ca13 to 0x012725 (0x1a2ed bytes) 000951 1786167315.494983 write 0x00bb71 thru 0x00dc35 (0x20c5 bytes) 000952 1786167315.496249 mapwrite 0x01335d thru 0x0156b8 (0x235c bytes) 000954 1786167315.533170 write 0x01fadc thru 0x01ffa4 (0x4c9 bytes) 000957 1786167315.535751 read 0x01d66f thru 0x01ffa4 (0x2936 bytes) 000960 1786167315.545346 mapwrite 0x034482 thru 0x03ffff (0xbb7e bytes) 000962 1786167315.589838 mapwrite 0x03b764 thru 0x03ffff (0x489c bytes) 000963 1786167315.613005 read 0x001f99 thru 0x00e2c5 (0xc32d bytes) 000964 1786167315.613567 mapread 0x027169 thru 0x02b79f (0x4637 bytes) 000966 1786167315.614764 mapread 0x038191 thru 0x03ffff (0x7e6f bytes) 000967 1786167315.615711 mapread 0x01324e thru 0x01aac6 (0x7879 bytes) 000968 1786167315.616679 write 0x02f4ef thru 0x03a55a (0xb06c bytes) 000969 1786167315.618140 write 0x016a13 thru 0x017471 (0xa5f bytes) 000971 1786167315.619656 write 0x0148ee thru 0x01a1db (0x58ee bytes) 000972 1786167315.620798 write 0x015bca thru 0x01e33f (0x8776 bytes) 000973 1786167315.622297 mapread 0x029cbb thru 0x037408 (0xd74e bytes) 000974 1786167315.627070 MIRROR_RESYNC 4 mirrors 000975 1786167316.288949 trunc from 0x040000 to 0x03084b (0xf7b4 bytes) 000976 1786167316.356440 read 0x02b9c0 thru 0x03084b (0x4e8c bytes) 000977 1786167316.365680 trunc from 0x03084c to 0x019f1b (0x16930 bytes) 000978 1786167316.393171 mapread 0x019c52 thru 0x019f1b (0x2ca bytes) 000980 1786167316.409543 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19f1c): Mirror 5: 0x78d1aca4 (stale) Mirror 6: 0x78d1aca4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19f1c): Mirror 5: 0x78d1aca4 (stale) Mirror 6: 0x78d1aca4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19f1c: skipped 000981 1786167316.737844 mapwrite 0x01dc60 thru 0x024f3b (0x72dc bytes) 000982 1786167316.806653 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24f3c): Mirror 5: 0x343d9141 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f3c): Mirror 5: 0x343d9141 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f3c: skipped 000983 1786167317.188128 trunc from 0x024f3c to 0x000468 (0x24ad3 bytes) 000984 1786167317.227381 mapwrite 0x011519 thru 0x013dee (0x28d6 bytes) 000991 1786167317.461905 read 0x004531 thru 0x00f42a (0xaefa bytes) 000992 1786167317.487437 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c2:0x0]: Invalid argument (22) 000993 1786167317.683400 read 0x00121f thru 0x009a19 (0x87fb bytes) 000995 1786167317.694148 mapwrite 0x0142e1 thru 0x01f668 (0xb388 bytes) 000998 1786167317.792222 write 0x02f047 thru 0x038f4c (0x9f06 bytes) 000999 1786167317.794107 mapwrite 0x0239aa thru 0x031341 (0xd998 bytes) All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 01:35:23 (1786167323) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 01:35:26 (1786167326) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.285033 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155199 s, 6.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 01:35:34 (1786167334) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5093312 .. Extending file size to 4120992 .. done Extending file size to 6762016 .. Extending file size to 4527072 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6411488 .. Extending file size to 6793152 .. Extending file size to 8922880 .. Extending file size to 8317184 .. Extending file size to 9021120 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8736352 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9060064 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57425 57426 57428 57430 57431 done 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) 192.168.201.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:36:56 (1786167416) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6998368 .. Extending file size to 8506592 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7700160 .. Extending file size to 8029344 .. Extending file size to 8345472 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8421920 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9400096 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58995 58996 58998 58999 59000 done 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) 192.168.201.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (90s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:38:26 (1786167506) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138356 s, 15.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.515935 s, 4.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x58eaf4e6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:38:45 (1786167525) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0962497 s, 21.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 01:38:53 (1786167533) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ff:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:39:01 (1786167541) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.350241 s, 6.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:39:14 (1786167554) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239026 s, 8.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:39:23 (1786167563) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267059 s, 7.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:39:58 (1786167598) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg158-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116467 s, 9.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:41:00 (1786167660) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg158-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.359654 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:42:04 (1786167724) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg158-server mds-ost sync done. Creating new pool test_204f oleg158-server: Pool lustre.test_204f created Adding targets to pool oleg158-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg158-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.324665 s, 6.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg158-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg158-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg158-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:43:25 (1786167805) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:43:32 (1786167812) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:43:40 (1786167820) oleg158-server: Pool lustre.test_206 created Adding targets to pool oleg158-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg158-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg158-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg158-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg158-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:44:05 (1786167845) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117045 s, 9.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064272 s, 16.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbbf70b0a Mirror 3: 0xbbf70b0a PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:44:17 (1786167857) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:44:21 (1786167861) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:44:24 (1786167864) Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.144102 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2170000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2540000 Read this (4 bytes): BBBB 192.168.201.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg158-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:45:03 (1786167903) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0183049 s, 448 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:45:09 (1786167909) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108812 s, 9.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x230 lmm_fid: [0x200000404:0x230:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x230 lmm_fid: [0x200000404:0x230:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834480 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-08 01:45:10.000000000 -0400 Modify: 2026-08-08 01:45:11.000000000 -0400 Change: 2026-08-08 01:45:12.000000000 -0400 Birth: 2026-08-08 01:45:11.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:45:17 (1786167917) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:45:20 (1786167920) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.55362 s, 74.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0431912 s, 94.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 8 01:45 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x232:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 8 01:45 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:45:29 (1786167929) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:45:37 (1786167937) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1819 sec =========== 01:45:47 (1786167947) === sanity-flr: start cleanup 01:45:49 (1786167949) === === sanity-flr: finish cleanup 01:45:55 (1786167955) ===