-----============= acceptance-small: sanity-flr ============----- Thu Sep 10 08:19:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg634-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:20:15 (1789042815) === oleg634-client.virtnet: executing check_config_client /mnt/lustre oleg634-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg634-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8fece06ad000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8fece06ad000.idle_timeout=debug disable quota as required oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:20:50 (1789042850) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanity-flr test 0a: lfs mirror create with -N option == 08:20:53 (1789042853) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:21:12 (1789042872) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:21:15 (1789042875) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:21:18 (1789042878) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:21:45 (1789042905) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:21:48 (1789042908) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:21:51 (1789042911) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8fece06ad000.stats=clear osc.lustre-OST0001-osc-ffff8fece06ad000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:21:59 (1789042919) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:22:10 (1789042930) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:22:19 (1789042939) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:22:23 (1789042943) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:22:31 (1789042951) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 08:22:42 (1789042962) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:22:52 (1789042972) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:22:56 (1789042976) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:23:04 (1789042984) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:23:14 (1789042994) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:23:20 (1789043000) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.978798 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918144 s, 11.4 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:23:31 (1789043011) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.47637 s, 37.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:23:40 (1789043020) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:23:50 (1789043030) Stopping /mnt/lustre-ost1 (opts:) on oleg634-server oleg634-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg634-server: oleg634-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg634-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg634-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg634-server oleg634-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg634-server: oleg634-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg634-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (141s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:26:11 (1789043171) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.395236 s, 13.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg634-server oleg634-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg634-server: oleg634-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg634-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:26:58 (1789043218) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg634-server oleg634-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg634-server: oleg634-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg634-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg634-server oleg634-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg634-server: oleg634-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg634-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:28:20 (1789043300) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.53469 s, 8.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 10 08:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.666s user 0m0.000s sys 0m0.186s Stopping /mnt/lustre-ost1 (opts:) on oleg634-server oleg634-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fece06ad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.325s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg634-server: oleg634-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg634-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg634-server oleg634-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.102s user 0m0.001s sys 0m0.172s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg634-server: oleg634-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg634-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg634-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fece06ad000.ost_server_uuid in FULL state after 0 sec PASS 33b (107s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:30:07 (1789043407) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:30:11 (1789043411) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:30:15 (1789043415) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:30:18 (1789043418) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 08:30:26 (1789043426) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.88915 s, 14.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:30:42 (1789043442) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354925 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332595 s, 31.5 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:30:50 (1789043450) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:30:58 (1789043458) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00462417 s, 221 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00242445 s, 422 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000911734 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00201879 s, 507 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 08:31:09 (1789043469) create test files with size 15732387 .. 5d748614f5f30a62ffd65951ec613637 - 34fa8be81710c731265c1e1f2429a893 - 3c59b3ed9e47789f82efe2b8090c6b4e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg634-client.virtnet /mnt/lustre (opts:) Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 08:31:40 (1789043500) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:31:59 (1789043519) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:32:06 (1789043526) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117881 s, 8.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325403 s, 32.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 08:32:15 (1789043535) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237379 s, 8.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.337645 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.29908 s, 7.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 08:32:33 (1789043553) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 08:32:38 (1789043558) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:32:43 (1789043563) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569426 s, 18.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 08:32:54 (1789043574) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:32:58 (1789043578) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:33:08 (1789043588) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.203617 s, 51.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg634-client.virtnet /mnt/lustre (opts:) Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:33:21 (1789043601) llite.lustre-ffff8fecdef38000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.275122 s, 38.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg634-client.virtnet /mnt/lustre (opts:) Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8fecc51fc800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:33:37 (1789043617) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:34:14 (1789043654) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (40s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:34:54 (1789043694) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 08:34:58 (1789043698) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.319191 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.117558 s, 26.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8e96641 (nosync,stale) Mirror 2: 0x8e96641 (nosync,stale) ** verify mirror 2 doesn't change original checksum: ba01dbe16e3da8834186d91a9c86f44f - mirror 1 checksum: c76c9635de1c0789dece7e18cb3478b1 - mirror 2 checksum: ba01dbe16e3da8834186d91a9c86f44f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T08:35:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T08:35:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ba01dbe16e3da8834186d91a9c86f44f - mirror 1 checksum: c76c9635de1c0789dece7e18cb3478b1 - mirror 2 checksum: c76c9635de1c0789dece7e18cb3478b1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T08:35:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T08:35:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:35:10 (1789043710) Failing mds1 on oleg634-server Stopping /mnt/lustre-mds1 (opts:) on oleg634-server 08:35:17 (1789043717) shut down facet: mds1 facet_host: oleg634-server facet_failover_host: oleg634-server Failover mds1 to oleg634-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg634-server: oleg634-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg634-client: oleg634-server: ssh exited with exit code 1 Started lustre-MDT0000 08:35:50 (1789043750) targets are mounted 08:35:50 (1789043750) facet_failover done oleg634-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312528 s, 33.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:36:13 (1789043773) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:36:17 (1789043777) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:36:21 (1789043781) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.021785 s, 752 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:36:29 (1789043789) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:36:32 (1789043792) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0249001 s, 164 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.132346 s, 30.9 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:36:40 (1789043800) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:37:03 (1789043823) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:37:33 (1789043853) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:37:56 (1789043876) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:38:20 (1789043900) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 08:38:28 (1789043908) mirror create pid 54333 mirror split pid 54334 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54333 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54334 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 08:39:35 (1789043975) Seed set to 3977 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1789043978.276984 write 0x022544 thru 0x028497 (0x5f54 bytes) 000002 1789043978.292105 mapread 0x020c68 thru 0x028497 (0x7830 bytes) truncating to largest ever: 0x2a082 000004 1789043978.307711 trunc from 0x028498 to 0x02a081 (0x1bea bytes) 000006 1789043978.328075 mapwrite 0x00a958 thru 0x00e262 (0x390b bytes) 000007 1789043978.454658 mapwrite 0x020767 thru 0x0240a7 (0x3941 bytes) 000011 1789043978.558813 punch from 0x01e22e to 0x02a081 (0xbe54 bytes) 000013 1789043978.575252 punch from 0x01d954 to 0x02a081 (0xc72e bytes) 000015 1789043978.602447 write 0x03bfb9 thru 0x03ffff (0x4047 bytes) 000016 1789043978.604732 read 0x015c0d thru 0x01f011 (0x9405 bytes) 000018 1789043978.629297 punch from 0x005694 to 0x013d95 (0xe702 bytes) 000020 1789043978.664654 mapread 0x028dc4 thru 0x029a45 (0xc82 bytes) 000021 1789043978.687680 trunc from 0x040000 to 0x00fb5e (0x304a1 bytes) 000022 1789043978.714259 trunc from 0x00fb5f to 0x01efa2 (0xf444 bytes) 000023 1789043978.739016 mapread 0x011d45 thru 0x016bd4 (0x4e90 bytes) 000024 1789043978.750194 read 0x01819d thru 0x01efa2 (0x6e06 bytes) 000025 1789043978.761302 mapwrite 0x007e35 thru 0x0141df (0xc3ab bytes) 000026 1789043978.847266 mapwrite 0x03f768 thru 0x03ffff (0x898 bytes) 000027 1789043978.976508 punch from 0x00b616 to 0x00d31b (0x1d06 bytes) 000028 1789043978.990290 punch from 0x00bcc1 to 0x0108f5 (0x4c35 bytes) 000029 1789043979.000051 write 0x013e25 thru 0x01b2cf (0x74ab bytes) 000032 1789043979.004495 mapwrite 0x01fb0a thru 0x02bfbe (0xc4b5 bytes) 000033 1789043979.149252 trunc from 0x040000 to 0x0089ad (0x37652 bytes) 000034 1789043979.169722 read 0x002f9e thru 0x0089ad (0x5a10 bytes) 000035 1789043979.170406 punch from 0x003728 to 0x003c01 (0x4da bytes) 000036 1789043979.178259 read 0x001397 thru 0x001633 (0x29d bytes) 000038 1789043979.180664 mapwrite 0x01b684 thru 0x022f7f (0x78fc bytes) 000039 1789043979.307849 punch from 0x01613b to 0x0198a2 (0x3768 bytes) 000040 1789043979.315017 mapwrite 0x006be4 thru 0x00d119 (0x6536 bytes) 000044 1789043979.389476 mapwrite 0x013b39 thru 0x0202ba (0xc782 bytes) 000046 1789043979.533259 MIRROR_EXTEND to 2 mirrors 000047 1789043980.243858 write 0x02f4c5 thru 0x032af7 (0x3633 bytes) 000048 1789043980.369115 write 0x017c9a thru 0x025e4f (0xe1b6 bytes) 000050 1789043980.380308 mapwrite 0x03909a thru 0x03ffff (0x6f66 bytes) 000051 1789043980.541140 read 0x020e29 thru 0x02d1ed (0xc3c5 bytes) 000054 1789043980.561158 mapread 0x029474 thru 0x02a4d9 (0x1066 bytes) 000055 1789043980.564753 mapread 0x03675c thru 0x03d466 (0x6d0b bytes) 000056 1789043980.565646 mapwrite 0x00c870 thru 0x01b388 (0xeb19 bytes) 000058 1789043980.681374 trunc from 0x040000 to 0x023ee0 (0x1c11f bytes) 000059 1789043980.702716 read 0x021992 thru 0x023ee0 (0x254f bytes) 000060 1789043980.709847 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000061 1789043980.908168 mapread 0x01aaef thru 0x02022b (0x573d bytes) truncating to largest ever: 0x31b4c 000062 1789043980.935806 trunc from 0x023ee1 to 0x031b4b (0xdc6b bytes) 000063 1789043980.951387 mapread 0x008f99 thru 0x011786 (0x87ee bytes) 000066 1789043980.975616 mapwrite 0x030148 thru 0x0364cd (0x6386 bytes) 000068 1789043981.061223 punch from 0x00a15b to 0x0127c2 (0x8668 bytes) 000069 1789043981.069622 mapwrite 0x0015ac thru 0x00bf1d (0xa972 bytes) 000070 1789043981.125974 trunc from 0x0364ce to 0x01b580 (0x1af4d bytes) 000071 1789043981.149560 punch from 0x002212 to 0x00322e (0x101d bytes) 000072 1789043981.158040 punch from 0x0030a3 to 0x009188 (0x60e6 bytes) 000073 1789043981.165804 trunc from 0x01b581 to 0x014840 (0x6d40 bytes) 000074 1789043981.191486 punch from 0x002edb to 0x00352f (0x655 bytes) 000075 1789043981.197644 punch from 0x006c7a to 0x00c046 (0x53cd bytes) 000076 1789043981.202055 mapread 0x001795 thru 0x00848a (0x6cf6 bytes) 000077 1789043981.214833 trunc from 0x014841 to 0x00d982 (0x6ebe bytes) 000078 1789043981.235056 write 0x03735b thru 0x03ad40 (0x39e6 bytes) 000079 1789043981.238404 write 0x02c849 thru 0x032f71 (0x6729 bytes) 000080 1789043981.254863 read 0x02ae6c thru 0x03036a (0x54ff bytes) 000081 1789043981.267702 punch from 0x0259b2 to 0x02d41c (0x7a6b bytes) 000082 1789043981.292008 write 0x035309 thru 0x038e4f (0x3b47 bytes) 000085 1789043981.304087 punch from 0x02c003 to 0x0328bd (0x68bb bytes) 000087 1789043981.351198 read 0x0390be thru 0x03ad40 (0x1c83 bytes) 000089 1789043981.360980 mapwrite 0x02d186 thru 0x030c49 (0x3ac4 bytes) 000090 1789043981.465809 mapread 0x036d53 thru 0x03ad40 (0x3fee bytes) 000091 1789043981.467145 mapread 0x002115 thru 0x00f664 (0xd550 bytes) 000093 1789043981.469171 mapwrite 0x008e10 thru 0x00d296 (0x4487 bytes) 000095 1789043981.536551 mapwrite 0x0281c9 thru 0x032b5f (0xa997 bytes) 000101 1789043981.601950 mapread 0x01071e thru 0x01f480 (0xed63 bytes) 000102 1789043981.603817 mapread 0x003d46 thru 0x00f586 (0xb841 bytes) 000105 1789043981.619834 punch from 0x036acd to 0x03ad40 (0x4274 bytes) 000106 1789043981.631595 write 0x024138 thru 0x024f07 (0xdd0 bytes) 000108 1789043981.633648 write 0x0277a1 thru 0x035750 (0xdfb0 bytes) 000109 1789043981.635655 mapwrite 0x03fba9 thru 0x03ffff (0x457 bytes) 000110 1789043981.735122 write 0x003819 thru 0x0070c6 (0x38ae bytes) 000111 1789043981.740596 mapread 0x0118a7 thru 0x01c2c7 (0xaa21 bytes) 000113 1789043981.743513 mapwrite 0x00863e thru 0x00d57b (0x4f3e bytes) 000114 1789043981.836235 mapread 0x00c4e4 thru 0x00f21a (0x2d37 bytes) 000116 1789043981.838734 write 0x009253 thru 0x014370 (0xb11e bytes) 000117 1789043981.842419 punch from 0x015887 to 0x01f77a (0x9ef4 bytes) truncating to largest ever: 0x3de10 000119 1789043981.862561 trunc from 0x040000 to 0x03de0f (0x21f0 bytes) 000120 1789043981.898200 trunc from 0x03de10 to 0x02c273 (0x11b9c bytes) 000121 1789043981.934961 mapwrite 0x01736c thru 0x01e588 (0x721d bytes) 000122 1789043982.066452 MIRROR_EXTEND to 2 mirrors 000124 1789043982.941653 trunc from 0x02c274 to 0x03aba7 (0xe934 bytes) 000125 1789043983.122403 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000126 1789043983.409772 read 0x029f91 thru 0x02f663 (0x56d3 bytes) 000127 1789043983.421247 punch from 0x004e24 to 0x011be3 (0xcdc0 bytes) 000128 1789043983.436359 mapwrite 0x037b27 thru 0x03ffff (0x84d9 bytes) 000129 1789043983.603968 mapwrite 0x01c549 thru 0x028440 (0xbef8 bytes) 000132 1789043983.757103 trunc from 0x040000 to 0x006abc (0x39543 bytes) 000136 1789043983.789556 MIRROR_EXTEND to 2 mirrors 000137 1789043984.856650 trunc from 0x006abd to 0x017f24 (0x11468 bytes) 000138 1789043984.972019 MIRROR_RESYNC 2 mirrors 000139 1789043985.737475 mapwrite 0x027834 thru 0x02b17c (0x3949 bytes) 000140 1789043985.966484 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b17d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b17d): Mirror 2: 0xb2b7276a (stale) CRC-32 checksum value for chunk [0, 0x2b17d): Mirror 1: 0xdb813766 Mirror 3: 0xdb813766 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b17d: skipped 000141 1789043987.305870 trunc from 0x02b17d to 0x023289 (0x7ef3 bytes) 000143 1789043987.419193 read 0x002d34 thru 0x00ddc0 (0xb08d bytes) 000146 1789043987.438244 mapread 0x01dac9 thru 0x023289 (0x57c1 bytes) 000147 1789043987.474649 punch from 0x00b39b to 0x0123a8 (0x700e bytes) 000148 1789043987.480609 mapwrite 0x00535a thru 0x010161 (0xae08 bytes) 000149 1789043987.626288 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2328a): Mirror 3: 0x72745ec1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2328a): Mirror 3: 0x72745ec1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2328a: skipped 000150 1789043988.075699 trunc from 0x02328a to 0x015caf (0xd5da bytes) 000152 1789043988.101642 mapread 0x008698 thru 0x01420a (0xbb73 bytes) 000153 1789043988.160714 trunc from 0x015cb0 to 0x0171bc (0x150d bytes) 000155 1789043988.186671 trunc from 0x0171bd to 0x02fb4a (0x1898e bytes) 000158 1789043988.218008 mapwrite 0x022eec thru 0x02a4ee (0x7603 bytes) 000159 1789043988.359089 mapwrite 0x01e00c thru 0x029cee (0xbce3 bytes) 000161 1789043988.455667 read 0x009371 thru 0x00a186 (0xe16 bytes) 000163 1789043988.457788 mapwrite 0x00c4fa thru 0x013165 (0x6c6c bytes) 000164 1789043988.517725 punch from 0x01bc1f to 0x025d66 (0xa148 bytes) 000165 1789043988.523928 write 0x0008fd thru 0x002294 (0x1998 bytes) 000166 1789043988.549031 read 0x02f364 thru 0x02fb4a (0x7e7 bytes) 000167 1789043988.570394 trunc from 0x02fb4b to 0x021b02 (0xe048 bytes) 000168 1789043988.587790 trunc from 0x021b03 to 0x01d06d (0x4a95 bytes) 000169 1789043988.610266 read 0x012845 thru 0x019b2e (0x72ea bytes) 000170 1789043988.611575 punch from 0x00ee5a to 0x019132 (0xa2d9 bytes) 000171 1789043988.624881 mapwrite 0x0036fe thru 0x0038cc (0x1cf bytes) 000172 1789043988.678012 mapread 0x019daf thru 0x01d06d (0x32bf bytes) 000173 1789043988.696957 punch from 0x017b9c to 0x01cfc8 (0x542d bytes) 000175 1789043988.711454 mapwrite 0x028cac thru 0x0378a0 (0xebf5 bytes) 000176 1789043988.861302 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000177 1789043989.080692 mapwrite 0x016c7d thru 0x017d31 (0x10b5 bytes) 000179 1789043989.160834 write 0x001ab2 thru 0x00a549 (0x8a98 bytes) 000184 1789043989.188839 mapwrite 0x03703a thru 0x03ffff (0x8fc6 bytes) 000185 1789043989.374153 read 0x00ea8e thru 0x017f91 (0x9504 bytes) 000186 1789043989.388884 trunc from 0x040000 to 0x03b5bf (0x4a40 bytes) 000188 1789043989.408017 trunc from 0x03b5c0 to 0x0292e6 (0x122d9 bytes) 000191 1789043989.425275 trunc from 0x0292e7 to 0x00c7d8 (0x1cb0e bytes) 000192 1789043989.444446 punch from 0x00c16c to 0x00c7d8 (0x66d bytes) 000194 1789043989.449614 punch from 0x00529d to 0x0078f4 (0x2658 bytes) 000195 1789043989.455300 mapread 0x0054a2 thru 0x0098a5 (0x4404 bytes) 000197 1789043989.466216 read 0x00b403 thru 0x00c7d8 (0x13d6 bytes) 000198 1789043989.471774 MIRROR_EXTEND to 2 mirrors 000199 1789043990.248090 punch from 0x00bbd1 to 0x00c7d8 (0xc08 bytes) 000200 1789043990.378031 mapread 0x004c5a thru 0x00c7d8 (0x7b7f bytes) 000201 1789043990.405068 mapwrite 0x0387f3 thru 0x03ffff (0x780d bytes) 000202 1789043990.537952 mapread 0x026a02 thru 0x036495 (0xfa94 bytes) 000203 1789043990.559849 mapread 0x004c0a thru 0x00c3e9 (0x77e0 bytes) 000206 1789043990.563886 trunc from 0x040000 to 0x037b7e (0x8481 bytes) 000207 1789043990.588007 mapread 0x007bac thru 0x00aab4 (0x2f09 bytes) 000208 1789043990.590148 trunc from 0x037b7f to 0x00bf4d (0x2bc31 bytes) 000209 1789043990.610975 read 0x0051f4 thru 0x007e02 (0x2c0f bytes) 000210 1789043990.611616 write 0x02c0a7 thru 0x02f05c (0x2fb6 bytes) 000211 1789043990.614960 read 0x00a49c thru 0x015a31 (0xb596 bytes) 000212 1789043990.624920 mapwrite 0x0292f5 thru 0x034237 (0xaf43 bytes) 000213 1789043990.738012 mapread 0x011920 thru 0x019150 (0x7831 bytes) 000216 1789043990.744728 punch from 0x0018b7 to 0x002bda (0x1324 bytes) 000217 1789043990.756331 trunc from 0x034238 to 0x03a523 (0x62ec bytes) 000219 1789043990.787082 write 0x02ec28 thru 0x02ec4e (0x27 bytes) 000221 1789043990.789835 trunc from 0x03a524 to 0x010347 (0x2a1dc bytes) 000222 1789043990.829875 read 0x00f0d7 thru 0x010347 (0x1271 bytes) 000224 1789043990.842067 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10348): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10348): Mirror 2: 0x5e815ea5 (stale) CRC-32 checksum value for chunk [0, 0x10348): Mirror 1: 0x27e8de7a Mirror 3: 0x27e8de7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10348: skipped 000225 1789043991.912534 punch from 0x00d658 to 0x010347 (0x2cf0 bytes) 000226 1789043992.049500 mapread 0x00080b thru 0x004220 (0x3a16 bytes) 000228 1789043992.090323 write 0x0334bc thru 0x03a9cb (0x7510 bytes) 000229 1789043992.103868 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a9cc): Mirror 2: 0x46250e09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a9cc): Mirror 2: 0x46250e09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a9cc: skipped 000231 1789043992.528753 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a9cc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a9cc): Mirror 2: 0x46250e09 (stale) CRC-32 checksum value for chunk [0, 0x3a9cc): Mirror 1: 0xd954f5b5 Mirror 3: 0xd954f5b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a9cc: skipped 000232 1789043993.701695 trunc from 0x03a9cc to 0x0160c1 (0x2490a bytes) 000233 1789043993.790779 mapread 0x01597d thru 0x0160c1 (0x745 bytes) 000235 1789043993.806798 trunc from 0x0160c2 to 0x02aa36 (0x14975 bytes) 000236 1789043993.840404 mapread 0x0191e1 thru 0x01dd7a (0x4b9a bytes) 000240 1789043993.870478 MIRROR_RESYNC 3 mirrors 000241 1789043994.939038 punch from 0x0050ca to 0x00e63f (0x9576 bytes) 000243 1789043995.024734 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aa37): Mirror 3: 0x72534113 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa37): Mirror 3: 0x72534113 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa37: skipped 000245 1789043995.347182 write 0x010470 thru 0x01dc58 (0xd7e9 bytes) 000246 1789043995.372866 read 0x007312 thru 0x01556e (0xe25d bytes) 000247 1789043995.383016 write 0x03da6b thru 0x03ffff (0x2595 bytes) 000248 1789043995.384360 mapread 0x01173c thru 0x01678a (0x504f bytes) truncating to largest ever: 0x3dff5 000249 1789043995.386953 trunc from 0x040000 to 0x03dff4 (0x200b bytes) 000250 1789043995.402702 mapread 0x01e577 thru 0x0239ac (0x5436 bytes) 000252 1789043995.423675 punch from 0x01dc4c to 0x0266c6 (0x8a7b bytes) 000253 1789043995.436981 punch from 0x01de67 to 0x02a547 (0xc6e1 bytes) 000254 1789043995.442069 read 0x023cee thru 0x031c6a (0xdf7d bytes) 000255 1789043995.460724 MIRROR_RESYNC 2 mirrors 000257 1789043996.307108 trunc from 0x03dff5 to 0x01756b (0x26a89 bytes) 000258 1789043996.516954 MIRROR_RESYNC 2 mirrors 000259 1789043997.270049 mapread 0x00ed07 thru 0x0106ed (0x19e7 bytes) 000262 1789043997.315803 MIRROR_EXTEND to 3 mirrors 000263 1789043998.234962 mapread 0x014a19 thru 0x01756b (0x2b53 bytes) 000264 1789043998.269541 mapread 0x008813 thru 0x00981f (0x100d bytes) 000267 1789043998.321649 trunc from 0x01756c to 0x0019f8 (0x15b73 bytes) 000269 1789043998.416062 write 0x038285 thru 0x03ffff (0x7d7b bytes) 000270 1789043998.429700 MIRROR_RESYNC 3 mirrors 000272 1789043999.742520 trunc from 0x040000 to 0x018a9f (0x27560 bytes) 000273 1789043999.845923 write 0x035d5a thru 0x03ffff (0xa2a6 bytes) 000275 1789043999.852718 punch from 0x008ae3 to 0x0153df (0xc8fd bytes) 000276 1789043999.895541 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5c65dc99 (stale) Mirror 4: 0x5c65dc99 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6586e1e4 Mirror 5: 0x6586e1e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1789044001.031486 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5c65dc99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5c65dc99 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6586e1e4 Mirror 5: 0x6586e1e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000278 1789044001.757368 write 0x012134 thru 0x0153c4 (0x3291 bytes) 000281 1789044001.904842 mapwrite 0x00337e thru 0x008cbd (0x5940 bytes) 000282 1789044002.020756 mapread 0x019e36 thru 0x01d717 (0x38e2 bytes) 000283 1789044002.035508 MIRROR_RESYNC 3 mirrors 000284 1789044003.227546 MIRROR_EXTEND to 4 mirrors 000287 1789044004.418606 punch from 0x038283 to 0x03ffff (0x7d7d bytes) 000289 1789044004.538533 write 0x02e790 thru 0x0349d7 (0x6248 bytes) 000290 1789044004.554871 write 0x03bb3b thru 0x03ffff (0x44c5 bytes) 000291 1789044004.570836 trunc from 0x040000 to 0x02bb48 (0x144b7 bytes) 000292 1789044004.597147 punch from 0x02572a to 0x027a56 (0x232d bytes) 000293 1789044004.602670 mapread 0x013a1b thru 0x021733 (0xdd19 bytes) 000294 1789044004.633808 mapwrite 0x00884d thru 0x00e58d (0x5d41 bytes) 000295 1789044004.696734 trunc from 0x02bb49 to 0x01ad30 (0x10e18 bytes) 000296 1789044004.720951 punch from 0x011d10 to 0x01ad30 (0x9021 bytes) 000298 1789044004.734255 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ad31): Mirror 5: 0xf3ab04fe (stale) Mirror 6: 0xf3ab04fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ad31): Mirror 5: 0xf3ab04fe (stale) Mirror 6: 0xf3ab04fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ad31: skipped 000300 1789044005.170932 write 0x02619e thru 0x032635 (0xc498 bytes) 000301 1789044005.177620 mapread 0x018994 thru 0x027ad0 (0xf13d bytes) 000302 1789044005.237320 punch from 0x012cb4 to 0x014022 (0x136f bytes) 000303 1789044005.274299 MIRROR_RESYNC 3 mirrors 000304 1789044006.391080 punch from 0x023dc5 to 0x032512 (0xe74e bytes) 000305 1789044006.460188 write 0x01ce6c thru 0x01d111 (0x2a6 bytes) 000306 1789044006.469936 trunc from 0x032636 to 0x0137e5 (0x1ee50 bytes) 000307 1789044006.493147 mapread 0x00aa52 thru 0x00e1d9 (0x3788 bytes) 000309 1789044006.517137 mapwrite 0x013aa7 thru 0x02284a (0xeda4 bytes) 000310 1789044006.628297 write 0x0313d6 thru 0x034d10 (0x393b bytes) 000311 1789044006.629595 trunc from 0x034d11 to 0x021465 (0x138ab bytes) 000312 1789044006.662714 trunc from 0x021466 to 0x03c6b1 (0x1b24c bytes) 000314 1789044006.702569 punch from 0x02b9dc to 0x0381a5 (0xc7ca bytes) 000315 1789044006.708561 punch from 0x02a671 to 0x037618 (0xcfa8 bytes) 000317 1789044006.716238 mapread 0x02fbbb thru 0x03b50a (0xb950 bytes) 000318 1789044006.734742 mapwrite 0x005b0f thru 0x0150ad (0xf59f bytes) 000322 1789044006.810744 write 0x017bf2 thru 0x01bf62 (0x4371 bytes) 000323 1789044006.812987 write 0x02013c thru 0x029321 (0x91e6 bytes) 000325 1789044006.815930 trunc from 0x03c6b2 to 0x035b35 (0x6b7c bytes) 000326 1789044006.831479 trunc from 0x035b36 to 0x00adcc (0x2ad69 bytes) 000328 1789044006.860608 mapread 0x000871 thru 0x006eba (0x664a bytes) 000331 1789044006.871811 mapread 0x0016d9 thru 0x005677 (0x3f9f bytes) 000333 1789044006.873816 read 0x00a701 thru 0x00adcc (0x6cc bytes) 000335 1789044006.875169 read 0x005627 thru 0x00adcc (0x57a6 bytes) 000336 1789044006.876856 trunc from 0x00adcd to 0x007833 (0x3599 bytes) 000337 1789044006.895863 punch from 0x000473 to 0x007833 (0x73c1 bytes) 000338 1789044006.903680 mapwrite 0x01d198 thru 0x027e45 (0xacae bytes) 000339 1789044007.017993 mapread 0x005056 thru 0x00987e (0x4829 bytes) 000340 1789044007.021109 trunc from 0x027e46 to 0x009cc7 (0x1e17e bytes) 000341 1789044007.054766 mapwrite 0x006a2e thru 0x00e0e1 (0x76b4 bytes) 000342 1789044007.142950 read 0x0077ed thru 0x00e0e1 (0x68f5 bytes) 000344 1789044007.144019 trunc from 0x00e0e2 to 0x03418d (0x260ac bytes) 000345 1789044007.160069 trunc from 0x03418e to 0x03d9a2 (0x9815 bytes) 000346 1789044007.181010 write 0x00edd0 thru 0x01b130 (0xc361 bytes) 000347 1789044007.188067 punch from 0x01253e to 0x01b683 (0x9146 bytes) 000348 1789044007.202511 punch from 0x0135eb to 0x01ae81 (0x7897 bytes) 000349 1789044007.212450 MIRROR_RESYNC 3 mirrors 000351 1789044008.160114 trunc from 0x03d9a3 to 0x03a7af (0x31f3 bytes) 000352 1789044008.223883 trunc from 0x03a7b0 to 0x0057db (0x34fd4 bytes) 000353 1789044008.248519 mapread 0x001c0b thru 0x0039d8 (0x1dce bytes) 000354 1789044008.268747 write 0x006bd8 thru 0x0088fe (0x1d27 bytes) 000355 1789044008.269871 trunc from 0x0088ff to 0x01a72f (0x11e31 bytes) 000356 1789044008.289492 write 0x022e39 thru 0x025603 (0x27cb bytes) 000357 1789044008.292138 write 0x0398b2 thru 0x03b5ab (0x1cfa bytes) 000358 1789044008.294267 punch from 0x032043 to 0x03810d (0x60cb bytes) 000361 1789044008.326499 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b5ac): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5ac): Mirror 5: 0x4b250503 (stale) Mirror 6: 0x4b250503 (stale) CRC-32 checksum value for chunk [0, 0x3b5ac): Mirror 1: 0xdad1e193 Mirror 7: 0xdad1e193 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5ac: skipped 000362 1789044009.751505 read 0x03254c thru 0x03263b (0xf0 bytes) 000363 1789044009.781819 punch from 0x034fc4 to 0x03ab92 (0x5bcf bytes) 000364 1789044009.913792 read 0x02c108 thru 0x03461b (0x8514 bytes) 000365 1789044009.933681 write 0x031e40 thru 0x03ece5 (0xcea6 bytes) 000366 1789044009.945035 MIRROR_RESYNC 4 mirrors 000367 1789044011.732117 punch from 0x01161e to 0x014521 (0x2f04 bytes) 000374 1789044011.827695 punch from 0x02cd2c to 0x037e10 (0xb0e5 bytes) 000378 1789044011.833245 mapread 0x026bc9 thru 0x02ddab (0x71e3 bytes) 000379 1789044011.855256 mapread 0x001f6c thru 0x010e27 (0xeebc bytes) 000382 1789044011.882853 mapread 0x0172d2 thru 0x020dbc (0x9aeb bytes) 000384 1789044011.897802 trunc from 0x03ece6 to 0x00dc01 (0x310e4 bytes) 000385 1789044011.923723 mapread 0x004e52 thru 0x00dc01 (0x8db0 bytes) 000386 1789044011.925104 mapwrite 0x000d62 thru 0x009b2d (0x8dcc bytes) 000387 1789044012.012888 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdc02): Mirror 6: 0x7c0530a6 (stale) Mirror 7: 0x7c0530a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdc02): Mirror 6: 0x7c0530a6 (stale) Mirror 7: 0x7c0530a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdc02: skipped 000388 1789044012.434088 punch from 0x00a183 to 0x00dc01 (0x3a7f bytes) 000390 1789044012.446805 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xdc02): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdc02): Mirror 6: 0x7c0530a6 (stale) Mirror 7: 0x7c0530a6 (stale) CRC-32 checksum value for chunk [0, 0xdc02): Mirror 1: 0xb95351a7 Mirror 8: 0xb95351a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdc02: skipped 000392 1789044013.485469 read 0x00d2f9 thru 0x00dc01 (0x909 bytes) 000393 1789044013.497530 mapwrite 0x0309df thru 0x03793f (0x6f61 bytes) 000394 1789044013.729074 trunc from 0x037940 to 0x01dd37 (0x19c08 bytes) 000396 1789044013.749605 mapread 0x00329b thru 0x005b70 (0x28d6 bytes) 000400 1789044013.780699 punch from 0x00953f to 0x0161b7 (0xcc79 bytes) 000401 1789044013.791836 mapread 0x015120 thru 0x01dd37 (0x8c18 bytes) 000403 1789044013.808785 punch from 0x01b607 to 0x01dd37 (0x2731 bytes) 000404 1789044013.817796 punch from 0x01bbaa to 0x01dd37 (0x218e bytes) 000407 1789044013.831219 read 0x008b90 thru 0x00bba7 (0x3018 bytes) 000408 1789044013.847292 read 0x009709 thru 0x00ca1c (0x3314 bytes) 000409 1789044013.848266 mapwrite 0x006cbf thru 0x00f873 (0x8bb5 bytes) 000410 1789044013.910561 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1dd38): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dd38): Mirror 6: 0xbfcbee7e (stale) Mirror 7: 0xbfcbee7e (stale) Mirror 8: 0x92bd5426 (stale) CRC-32 checksum value for chunk [0, 0x1dd38): Mirror 1: 0x9c972286 Mirror 9: 0x9c972286 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dd38: skipped truncating to largest ever: 0x3ee48 000413 1789044014.991423 trunc from 0x01dd38 to 0x03ee47 (0x21110 bytes) 000414 1789044015.192049 trunc from 0x03ee48 to 0x024d92 (0x1a0b5 bytes) 000416 1789044015.213100 punch from 0x019be8 to 0x020f40 (0x7359 bytes) 000419 1789044015.219386 mapread 0x024aea thru 0x024d92 (0x2a9 bytes) 000421 1789044015.226067 read 0x022e76 thru 0x024d92 (0x1f1d bytes) 000422 1789044015.232157 read 0x023d00 thru 0x024d92 (0x1093 bytes) 000423 1789044015.232697 write 0x03cd82 thru 0x03e1fb (0x147a bytes) 000424 1789044015.233637 mapwrite 0x0225d2 thru 0x02ecd8 (0xc707 bytes) 000425 1789044015.299596 mapread 0x01160d thru 0x018e42 (0x7836 bytes) 000426 1789044015.301214 trunc from 0x03e1fc to 0x01242b (0x2bdd0 bytes) truncating to largest ever: 0x3f226 000427 1789044015.322275 trunc from 0x01242c to 0x03f225 (0x2cdfa bytes) 000429 1789044015.349627 punch from 0x002ce6 to 0x008d54 (0x606f bytes) 000431 1789044015.355749 trunc from 0x03f226 to 0x01676e (0x28ab7 bytes) 000432 1789044015.375421 mapread 0x0132fc thru 0x01676e (0x3473 bytes) 000434 1789044015.407762 mapwrite 0x02be63 thru 0x036f90 (0xb12e bytes) 000435 1789044015.558973 mapwrite 0x02b31b thru 0x02fa7f (0x4765 bytes) 000436 1789044015.617739 punch from 0x01a626 to 0x01ca04 (0x23df bytes) 000437 1789044015.622499 trunc from 0x036f91 to 0x015330 (0x21c60 bytes) 000438 1789044015.640219 mapread 0x00a36f thru 0x015330 (0xafc2 bytes) 000439 1789044015.641460 read 0x00bb98 thru 0x00c956 (0xdbf bytes) 000441 1789044015.645006 mapwrite 0x01d3cd thru 0x02a7d8 (0xd40c bytes) 000442 1789044015.714978 trunc from 0x02a7d9 to 0x01c802 (0xdfd6 bytes) 000443 1789044015.733949 mapwrite 0x005cc2 thru 0x0096c7 (0x3a06 bytes) 000445 1789044015.822515 MIRROR_RESYNC 5 mirrors 000446 1789044017.567485 mapwrite 0x007e19 thru 0x00dead (0x6095 bytes) 000448 1789044017.813263 read 0x013911 thru 0x01c802 (0x8ef2 bytes) 000450 1789044017.837023 mapread 0x0097ef thru 0x014499 (0xacab bytes) 000451 1789044017.852418 read 0x008f61 thru 0x0187be (0xf85e bytes) 000452 1789044017.853249 punch from 0x013451 to 0x016c02 (0x37b2 bytes) 000453 1789044017.878303 mapwrite 0x011de4 thru 0x016968 (0x4b85 bytes) 000454 1789044018.032596 punch from 0x0100b7 to 0x012666 (0x25b0 bytes) 000456 1789044018.038497 punch from 0x0111f4 to 0x012d3c (0x1b49 bytes) 000458 1789044018.044485 read 0x016cc2 thru 0x01c802 (0x5b41 bytes) 000459 1789044018.045722 mapread 0x015a04 thru 0x018deb (0x33e8 bytes) 000460 1789044018.047465 mapread 0x01b1bc thru 0x01c802 (0x1647 bytes) 000461 1789044018.048686 mapwrite 0x0365e0 thru 0x038b3f (0x2560 bytes) 000464 1789044018.175888 punch from 0x02d090 to 0x0360bc (0x902d bytes) 000465 1789044018.183625 write 0x00f172 thru 0x01121a (0x20a9 bytes) 000466 1789044018.184751 mapwrite 0x015f78 thru 0x01b36f (0x53f8 bytes) 000467 1789044018.255401 punch from 0x02af8f to 0x032eb8 (0x7f2a bytes) 000468 1789044018.262052 write 0x021f60 thru 0x0286ed (0x678e bytes) 000470 1789044018.265873 write 0x03e50e thru 0x03ffff (0x1af2 bytes) 000471 1789044018.267138 punch from 0x0025e8 to 0x00d186 (0xab9f bytes) 000472 1789044018.287543 mapread 0x026e32 thru 0x02a26a (0x3439 bytes) 000473 1789044018.308747 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x2bf8c912 (stale) Mirror 8: 0x2bf8c912 (stale) Mirror 9: 0x2bf8c912 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x2bf8c912 (stale) Mirror 8: 0x2bf8c912 (stale) Mirror 9: 0x2bf8c912 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000474 1789044018.817360 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2bf8c912 (stale) Mirror 9: 0x2bf8c912 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2bf8c912 (stale) Mirror 9: 0x2bf8c912 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000477 1789044019.215993 read 0x03c88b thru 0x03ffff (0x3775 bytes) 000479 1789044019.225023 mapwrite 0x0376aa thru 0x038d3f (0x1696 bytes) 000484 1789044019.317590 punch from 0x036698 to 0x03ffff (0x9968 bytes) 000485 1789044019.326708 punch from 0x0131ac to 0x015a89 (0x28de bytes) 000487 1789044019.338011 mapwrite 0x00ff24 thru 0x01a5c0 (0xa69d bytes) 000488 1789044019.466650 punch from 0x03c1fb to 0x03c94c (0x752 bytes) 000489 1789044019.476523 write 0x03774b thru 0x03a581 (0x2e37 bytes) 000492 1789044019.508592 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2bf8c912 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2bf8c912 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000493 1789044019.854494 mapread 0x019368 thru 0x028876 (0xf50f bytes) 000494 1789044019.894943 trunc from 0x040000 to 0x0225af (0x1da50 bytes) 000495 1789044019.933523 write 0x01774c thru 0x0226ca (0xaf7f bytes) 000496 1789044019.936683 mapread 0x020d66 thru 0x0226ca (0x1965 bytes) 000497 1789044019.941084 mapwrite 0x018316 thru 0x0254cc (0xd1b7 bytes) 000498 1789044020.088600 read 0x0088c5 thru 0x012e8c (0xa5c8 bytes) 000499 1789044020.120767 mapread 0x017b04 thru 0x01ad57 (0x3254 bytes) 000500 1789044020.129717 MIRROR_RESYNC 2 mirrors 000501 1789044020.989615 mapwrite 0x03a3ff thru 0x03bb0a (0x170c bytes) 000502 1789044021.133018 read 0x00d335 thru 0x01821c (0xaee8 bytes) 000503 1789044021.142758 mapwrite 0x01e0fb thru 0x025d2c (0x7c32 bytes) 000504 1789044021.208486 write 0x03d692 thru 0x03ffff (0x296e bytes) 000505 1789044021.211262 punch from 0x001efe to 0x008755 (0x6858 bytes) 000506 1789044021.227557 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x2b61b237 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb42bd9b5 Mirror 10: 0xb42bd9b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1789044022.446257 mapwrite 0x01b0aa thru 0x02950e (0xe465 bytes) 000509 1789044022.779410 write 0x004de4 thru 0x00a515 (0x5732 bytes) 000513 1789044022.793359 trunc from 0x040000 to 0x018b4e (0x274b1 bytes) 000514 1789044022.861393 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18b4f): Mirror 10: 0x7bb1f78d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18b4f): Mirror 10: 0x7bb1f78d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18b4f: skipped 000516 1789044023.131677 punch from 0x004691 to 0x011de4 (0xd754 bytes) 000517 1789044023.137645 read 0x00f31b thru 0x018b4e (0x9834 bytes) 000518 1789044023.146643 read 0x00f304 thru 0x018b4e (0x984b bytes) 000519 1789044023.147334 write 0x020755 thru 0x026e30 (0x66dc bytes) 000520 1789044023.149713 write 0x00042b thru 0x005551 (0x5127 bytes) 000521 1789044023.164683 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26e31): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e31): Mirror 10: 0x62662ae9 (stale) CRC-32 checksum value for chunk [0, 0x26e31): Mirror 1: 0x1ee34d55 Mirror 11: 0x1ee34d55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e31: skipped 000522 1789044024.258348 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26e31): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e31): Mirror 10: 0x62662ae9 (stale) CRC-32 checksum value for chunk [0, 0x26e31): Mirror 1: 0x1ee34d55 Mirror 11: 0x1ee34d55 Mirror 12: 0x1ee34d55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e31: skipped 000523 1789044025.308853 read 0x002770 thru 0x00d846 (0xb0d7 bytes) 000524 1789044025.324623 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26e31): Mirror 10: 0x62662ae9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e31): Mirror 10: 0x62662ae9 (stale) CRC-32 checksum value for chunk [0, 0x26e31): Mirror 1: 0x1ee34d55 Mirror 12: 0x1ee34d55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e31: skipped 000526 1789044025.900335 read 0x000c10 thru 0x00c47c (0xb86d bytes) 000528 1789044026.000216 trunc from 0x026e31 to 0x00380f (0x23621 bytes) 000529 1789044026.187100 write 0x01db31 thru 0x029390 (0xb860 bytes) 000532 1789044026.202065 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29391): Mirror 12: 0x89834e1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29391): Mirror 12: 0x89834e1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29391: skipped 000533 1789044026.598223 write 0x0314ed thru 0x034bcb (0x36df bytes) 000535 1789044026.601943 mapread 0x02b6e0 thru 0x0345c6 (0x8ee7 bytes) 000536 1789044026.636726 write 0x033e58 thru 0x03c466 (0x860f bytes) 000538 1789044026.641346 mapread 0x009fad thru 0x00fae2 (0x5b36 bytes) 000540 1789044026.663699 mapwrite 0x02ee92 thru 0x0329ff (0x3b6e bytes) 000541 1789044026.728788 write 0x0370ac thru 0x03ffff (0x8f54 bytes) 000544 1789044026.740187 MIRROR_RESYNC 2 mirrors 000545 1789044027.596032 mapwrite 0x01eb36 thru 0x02d58e (0xea59 bytes) 000547 1789044027.825549 mapread 0x00f958 thru 0x01acba (0xb363 bytes) 000548 1789044027.837338 read 0x028a46 thru 0x03294e (0x9f09 bytes) 000549 1789044027.843136 mapread 0x00aedf thru 0x010465 (0x5587 bytes) 000554 1789044027.847217 write 0x011505 thru 0x01aee9 (0x99e5 bytes) 000556 1789044027.849697 mapread 0x033f38 thru 0x03d063 (0x912c bytes) 000557 1789044027.871034 MIRROR_RESYNC 2 mirrors 000558 1789044028.923443 MIRROR_EXTEND to 3 mirrors 000559 1789044029.739715 read 0x0384e9 thru 0x03ffff (0x7b17 bytes) 000560 1789044029.752758 punch from 0x024182 to 0x02a1fb (0x607a bytes) 000561 1789044029.828938 read 0x028d30 thru 0x032242 (0x9513 bytes) 000562 1789044029.849557 trunc from 0x040000 to 0x033642 (0xc9bd bytes) 000565 1789044029.874543 read 0x030596 thru 0x033642 (0x30ad bytes) 000567 1789044029.876606 trunc from 0x033643 to 0x00a31c (0x29326 bytes) 000568 1789044029.898446 mapwrite 0x01efba thru 0x024ebe (0x5f05 bytes) 000572 1789044029.985676 write 0x019241 thru 0x028115 (0xeed5 bytes) 000573 1789044029.987733 mapread 0x02561c thru 0x028115 (0x2afa bytes) 000574 1789044029.988999 trunc from 0x028116 to 0x014e98 (0x1327d bytes) 000575 1789044030.032819 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x14e99): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14e99): Mirror 12: 0xed66c42e (stale) Mirror 13: 0xed66c42e (stale) CRC-32 checksum value for chunk [0, 0x14e99): Mirror 1: 0x8f0d682d Mirror 14: 0x8f0d682d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14e99: skipped 000576 1789044031.261657 trunc from 0x014e99 to 0x00369f (0x117f9 bytes) 000578 1789044031.354449 write 0x02846d thru 0x032809 (0xa39d bytes) 000580 1789044031.358663 read 0x02c9b2 thru 0x0313fa (0x4a49 bytes) 000581 1789044031.366412 punch from 0x00c8f4 to 0x01c7fd (0xff0a bytes) 000582 1789044031.379390 trunc from 0x03280a to 0x01d715 (0x150f4 bytes) 000584 1789044031.402196 read 0x00eb07 thru 0x01d60a (0xeb04 bytes) 000585 1789044031.412802 mapread 0x0064f9 thru 0x010818 (0xa320 bytes) 000586 1789044031.413790 mapread 0x01bcd7 thru 0x01d715 (0x1a3f bytes) 000587 1789044031.414559 punch from 0x01c6c7 to 0x01d715 (0x104f bytes) 000589 1789044031.423016 punch from 0x00da34 to 0x00e9b8 (0xf85 bytes) 000590 1789044031.427804 read 0x008d8b thru 0x015219 (0xc48f bytes) 000592 1789044031.437433 trunc from 0x01d716 to 0x01adac (0x2969 bytes) 000594 1789044031.454502 mapwrite 0x02f860 thru 0x03510a (0x58ab bytes) 000595 1789044031.613069 mapread 0x00e1a0 thru 0x01d8d8 (0xf739 bytes) 000596 1789044031.643602 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3510b): Mirror 13: 0x9dbf68be (stale) Mirror 14: 0x78ee7cdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3510b): Mirror 13: 0x9dbf68be (stale) Mirror 14: 0x78ee7cdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3510b: skipped 000597 1789044032.160812 trunc from 0x03510b to 0x03452b (0xbdf bytes) 000598 1789044032.189242 read 0x02cec8 thru 0x02d62a (0x763 bytes) 000600 1789044032.211699 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3452c): Mirror 14: 0xe51c9865 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3452c): Mirror 14: 0xe51c9865 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3452c: skipped 000601 1789044032.630505 read 0x00f23f thru 0x0112c5 (0x2087 bytes) 000603 1789044032.651780 punch from 0x0250f1 to 0x02a764 (0x5674 bytes) 000604 1789044032.671598 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3452c): Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3452c): Mirror 14: 0xe51c9865 (stale) CRC-32 checksum value for chunk [0, 0x3452c): Mirror 1: 0xb915375c Mirror 15: 0xb915375c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3452c: skipped 000605 1789044033.998567 write 0x001399 thru 0x00ff48 (0xebb0 bytes) 000606 1789044034.134558 punch from 0x02808e to 0x02b408 (0x337b bytes) 000607 1789044034.147552 mapwrite 0x0162a2 thru 0x01dc26 (0x7985 bytes) 000608 1789044034.230237 punch from 0x01bfe2 to 0x02ace1 (0xed00 bytes) 000611 1789044034.238631 trunc from 0x03452c to 0x0137f9 (0x20d32 bytes) 000612 1789044034.269372 read 0x012d45 thru 0x0137f9 (0xab5 bytes) 000613 1789044034.275719 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x137fa): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x137fa): Mirror 14: 0x78a39a2d (stale) Mirror 15: 0x56c3d38e (stale) CRC-32 checksum value for chunk [0, 0x137fa): Mirror 1: 0x7f24f1b1 Mirror 16: 0x7f24f1b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x137fa: skipped 000614 1789044035.367018 mapread 0x009aab thru 0x00fb17 (0x606d bytes) 000615 1789044035.428190 trunc from 0x0137fa to 0x035d54 (0x2255b bytes) 000616 1789044035.664570 trunc from 0x035d55 to 0x023fd4 (0x11d80 bytes) 000617 1789044035.699825 write 0x00be47 thru 0x00f4d7 (0x3691 bytes) 000618 1789044035.749068 read 0x001b16 thru 0x011811 (0xfcfc bytes) 000619 1789044035.787467 read 0x0032a3 thru 0x003f97 (0xcf5 bytes) 000621 1789044035.788632 write 0x03cff5 thru 0x03e5cc (0x15d8 bytes) 000623 1789044035.791485 mapread 0x03073c thru 0x039c63 (0x9528 bytes) 000624 1789044035.828175 write 0x00d298 thru 0x00e26a (0xfd3 bytes) 000626 1789044035.839955 punch from 0x00764b to 0x00ac4b (0x3601 bytes) 000627 1789044035.893839 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e5cd): Mirror 14: 0x6d19a341 (stale) Mirror 16: 0x561d00af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e5cd): Mirror 14: 0x6d19a341 (stale) Mirror 16: 0x561d00af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e5cd: skipped 000629 1789044036.364051 mapwrite 0x020ba0 thru 0x024b23 (0x3f84 bytes) 000630 1789044036.568706 MIRROR_RESYNC 3 mirrors 000631 1789044038.535615 read 0x004d2d thru 0x00ac36 (0x5f0a bytes) 000633 1789044038.547964 trunc from 0x03e5cd to 0x03bb35 (0x2a97 bytes) 000635 1789044038.647302 MIRROR_RESYNC 3 mirrors 000636 1789044040.455601 write 0x00488f thru 0x0119dc (0xd14e bytes) 000637 1789044040.559770 trunc from 0x03bb36 to 0x003431 (0x38704 bytes) 000638 1789044040.610157 read 0x001e2e thru 0x003431 (0x1604 bytes) 000639 1789044040.619662 mapread 0x000742 thru 0x003431 (0x2cf0 bytes) 000641 1789044040.637080 MIRROR_RESYNC 3 mirrors 000642 1789044042.014784 MIRROR_RESYNC 3 mirrors 000643 1789044042.463736 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000644 1789044042.955969 trunc from 0x003432 to 0x02c044 (0x28c13 bytes) 000645 1789044043.119717 mapread 0x0130ac thru 0x01dc3a (0xab8f bytes) 000646 1789044043.157553 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000647 1789044043.372535 MIRROR_EXTEND to 2 mirrors 000648 1789044044.780219 read 0x01f3ff thru 0x0233a7 (0x3fa9 bytes) 000650 1789044044.831526 mapwrite 0x022081 thru 0x02600e (0x3f8e bytes) 000651 1789044045.155312 mapwrite 0x00ab93 thru 0x011f0a (0x7378 bytes) 000653 1789044045.262963 mapwrite 0x037f80 thru 0x03ffff (0x8080 bytes) 000654 1789044045.421556 write 0x014854 thru 0x023bbc (0xf369 bytes) 000655 1789044045.429405 write 0x029d50 thru 0x02f8d0 (0x5b81 bytes) 000656 1789044045.436106 trunc from 0x040000 to 0x01954f (0x26ab0 bytes) 000658 1789044045.466107 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x19550): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19550): Mirror 2: 0xfd0d6660 (stale) CRC-32 checksum value for chunk [0, 0x19550): Mirror 1: 0x8366073b Mirror 3: 0x8366073b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19550: skipped 000660 1789044046.974279 punch from 0x00dcf6 to 0x016379 (0x8684 bytes) 000661 1789044047.103531 read 0x008964 thru 0x00dc37 (0x52d4 bytes) 000663 1789044047.117595 trunc from 0x019550 to 0x005c00 (0x1394f bytes) 000664 1789044047.147775 read 0x001734 thru 0x005c00 (0x44cd bytes) 000666 1789044047.174454 write 0x01c42b thru 0x020d9c (0x4972 bytes) 000667 1789044047.178067 mapread 0x003d99 thru 0x005ba5 (0x1e0d bytes) 000668 1789044047.181859 mapread 0x01fdb2 thru 0x020d9c (0xfeb bytes) 000669 1789044047.183184 read 0x0118fa thru 0x01a182 (0x8889 bytes) 000670 1789044047.220687 write 0x0242ba thru 0x0303b4 (0xc0fb bytes) 000671 1789044047.237201 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x303b5): Mirror 2: 0x58172811 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303b5): Mirror 2: 0x58172811 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303b5: skipped 000673 1789044047.783480 trunc from 0x0303b5 to 0x00d235 (0x2317f bytes) 000674 1789044047.839797 mapwrite 0x02482f thru 0x02edc4 (0xa596 bytes) 000675 1789044048.089797 trunc from 0x02edc5 to 0x019320 (0x15aa4 bytes) 000676 1789044048.141526 trunc from 0x019321 to 0x02678f (0xd46f bytes) 000677 1789044048.164156 mapwrite 0x025539 thru 0x027d10 (0x27d8 bytes) 000678 1789044048.334371 punch from 0x0006e4 to 0x00485f (0x417c bytes) 000679 1789044048.348662 mapwrite 0x034677 thru 0x036502 (0x1e8c bytes) 000680 1789044048.486504 trunc from 0x036503 to 0x0336c7 (0x2e3b bytes) 000685 1789044048.532861 mapwrite 0x000c48 thru 0x0088b2 (0x7c6b bytes) 000686 1789044048.636370 MIRROR_RESYNC 2 mirrors 000687 1789044050.249523 read 0x023ab5 thru 0x02f5b2 (0xbafe bytes) 000690 1789044050.307522 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000691 1789044050.511667 trunc from 0x0336c8 to 0x018bac (0x1ab1b bytes) 000692 1789044050.561263 mapwrite 0x00e0f0 thru 0x01616a (0x807b bytes) 000694 1789044050.757366 write 0x020940 thru 0x025219 (0x48da bytes) 000696 1789044050.764287 MIRROR_EXTEND to 2 mirrors 000699 1789044051.818320 read 0x003ecf thru 0x012fe8 (0xf11a bytes) 000701 1789044051.858642 mapread 0x00049b thru 0x00f873 (0xf3d9 bytes) 000703 1789044051.893498 mapwrite 0x032c41 thru 0x03d576 (0xa936 bytes) 000704 1789044052.214944 punch from 0x015319 to 0x01573e (0x426 bytes) 000706 1789044052.234842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000707 1789044052.504754 punch from 0x016282 to 0x01ae40 (0x4bbf bytes) 000708 1789044052.508560 trunc from 0x03d577 to 0x030fea (0xc58c bytes) 000709 1789044052.526263 mapwrite 0x00f68e thru 0x014afb (0x546e bytes) 000711 1789044052.589802 write 0x03d701 thru 0x03ffff (0x28ff bytes) 000712 1789044052.591068 read 0x01527a thru 0x016ff6 (0x1d7d bytes) 000713 1789044052.592215 mapwrite 0x027440 thru 0x027f1a (0xadb bytes) 000714 1789044052.675721 write 0x022550 thru 0x02755a (0x500b bytes) 000715 1789044052.678409 trunc from 0x040000 to 0x004839 (0x3b7c6 bytes) 000716 1789044052.723123 punch from 0x001d68 to 0x004839 (0x2ad2 bytes) 000718 1789044052.733474 write 0x03c3a7 thru 0x03ffff (0x3c59 bytes) 000719 1789044052.735477 read 0x02f43c thru 0x0338aa (0x446f bytes) 000722 1789044052.750324 mapwrite 0x0303b2 thru 0x03e12b (0xdd7a bytes) 000723 1789044052.836496 punch from 0x002ad9 to 0x0112e3 (0xe80b bytes) 000724 1789044052.851453 punch from 0x01a135 to 0x02789e (0xd76a bytes) 000726 1789044052.877213 mapwrite 0x038037 thru 0x0381ae (0x178 bytes) 000727 1789044053.021423 mapwrite 0x01d170 thru 0x023ee8 (0x6d79 bytes) 000729 1789044053.201602 punch from 0x02a3c9 to 0x02dd4d (0x3985 bytes) 000731 1789044053.227384 punch from 0x023871 to 0x032951 (0xf0e1 bytes) 000733 1789044053.253043 read 0x02ddb4 thru 0x031346 (0x3593 bytes) 000739 1789044053.291215 read 0x0346ec thru 0x0350dd (0x9f2 bytes) 000740 1789044053.291795 mapread 0x0035d7 thru 0x0109a0 (0xd3ca bytes) 000741 1789044053.292712 mapread 0x03d507 thru 0x03ffff (0x2af9 bytes) 000743 1789044053.304483 mapwrite 0x0041e4 thru 0x005894 (0x16b1 bytes) 000744 1789044053.438180 write 0x017a10 thru 0x0251ba (0xd7ab bytes) 000746 1789044053.450498 write 0x0236a4 thru 0x02d945 (0xa2a2 bytes) 000747 1789044053.456803 read 0x015eef thru 0x01621a (0x32c bytes) 000748 1789044053.460804 write 0x032a98 thru 0x033d87 (0x12f0 bytes) 000750 1789044053.482313 mapread 0x012f50 thru 0x022b3e (0xfbef bytes) 000751 1789044053.489273 write 0x031d95 thru 0x03ffff (0xe26b bytes) 000753 1789044053.495631 read 0x03374e thru 0x03aa5f (0x7312 bytes) 000756 1789044053.499858 punch from 0x035b23 to 0x038b1b (0x2ff9 bytes) 000757 1789044053.546446 read 0x025f72 thru 0x0277b9 (0x1848 bytes) 000758 1789044053.547519 mapwrite 0x025873 thru 0x0356f1 (0xfe7f bytes) 000760 1789044053.710744 mapread 0x01d9d2 thru 0x02739e (0x99cd bytes) 000762 1789044053.718001 trunc from 0x040000 to 0x02d005 (0x12ffa bytes) 000764 1789044053.755042 punch from 0x00b0ce to 0x012c9e (0x7bd1 bytes) 000769 1789044053.770628 punch from 0x01f841 to 0x025e0a (0x65ca bytes) 000770 1789044053.784461 mapread 0x011114 thru 0x01d00e (0xbefb bytes) 000771 1789044053.812688 mapwrite 0x014182 thru 0x017604 (0x3483 bytes) 000773 1789044053.967610 write 0x03ec91 thru 0x03ffff (0x136f bytes) 000774 1789044053.985390 MIRROR_EXTEND to 2 mirrors 000775 1789044055.306002 read 0x00ed3e thru 0x01e7f5 (0xfab8 bytes) 000776 1789044055.323644 punch from 0x0083fa to 0x00f898 (0x749f bytes) 000777 1789044055.490185 write 0x0329ae thru 0x03ffff (0xd652 bytes) 000779 1789044055.510741 mapread 0x0278ec thru 0x02d11a (0x582f bytes) 000780 1789044055.538492 mapread 0x00c93b thru 0x018cad (0xc373 bytes) 000781 1789044055.563632 punch from 0x03bd61 to 0x03ffff (0x429f bytes) 000782 1789044055.591555 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000784 1789044055.756946 mapwrite 0x012734 thru 0x016c32 (0x44ff bytes) 000785 1789044055.911379 mapread 0x0335e1 thru 0x035298 (0x1cb8 bytes) 000786 1789044055.943920 read 0x00f0a0 thru 0x01e7e3 (0xf744 bytes) 000787 1789044055.957814 mapread 0x01fe3d thru 0x02d633 (0xd7f7 bytes) 000788 1789044055.985675 read 0x034ffd thru 0x03d8f3 (0x88f7 bytes) 000790 1789044056.012806 punch from 0x0346ea to 0x03aed7 (0x67ee bytes) 000792 1789044056.029349 write 0x0206b7 thru 0x0229ef (0x2339 bytes) 000794 1789044056.037329 write 0x031cc4 thru 0x03ffff (0xe33c bytes) 000796 1789044056.054130 trunc from 0x040000 to 0x00d88a (0x32775 bytes) 000799 1789044056.096686 read 0x003309 thru 0x00d88a (0xa582 bytes) 000800 1789044056.098247 trunc from 0x00d88b to 0x035e95 (0x2860b bytes) 000801 1789044056.121480 mapwrite 0x011138 thru 0x016b6d (0x5a36 bytes) 000802 1789044056.260337 read 0x015670 thru 0x02079c (0xb12d bytes) 000803 1789044056.271089 mapread 0x030243 thru 0x035e95 (0x5c53 bytes) 000805 1789044056.272808 punch from 0x013006 to 0x01ba74 (0x8a6f bytes) 000806 1789044056.287081 punch from 0x006018 to 0x00d47e (0x7467 bytes) 000808 1789044056.306973 trunc from 0x035e96 to 0x02ec5e (0x7237 bytes) 000811 1789044056.357133 mapread 0x027190 thru 0x02ec5e (0x7acf bytes) 000813 1789044056.360150 trunc from 0x02ec5f to 0x03afa0 (0xc342 bytes) 000814 1789044056.402050 mapread 0x01577c thru 0x01870c (0x2f91 bytes) 000815 1789044056.424069 write 0x008a80 thru 0x00eb80 (0x6101 bytes) 000816 1789044056.428567 trunc from 0x03afa1 to 0x00fc4f (0x2b351 bytes) 000820 1789044056.479970 MIRROR_EXTEND to 2 mirrors 000821 1789044057.365800 mapread 0x00fb96 thru 0x00fc4f (0xba bytes) 000822 1789044057.379851 mapread 0x00afc9 thru 0x00dda9 (0x2de1 bytes) 000823 1789044057.405163 read 0x00d58c thru 0x00d9c9 (0x43e bytes) 000824 1789044057.405530 mapread 0x008fef thru 0x00f247 (0x6259 bytes) 000825 1789044057.408441 mapread 0x00f01f thru 0x00fc4f (0xc31 bytes) 000826 1789044057.413259 write 0x018a51 thru 0x022ab8 (0xa068 bytes) 000827 1789044057.565024 write 0x0248e3 thru 0x02dd59 (0x9477 bytes) 000828 1789044057.566388 punch from 0x0260d3 to 0x02d4c4 (0x73f2 bytes) 000830 1789044057.585135 mapread 0x0135a9 thru 0x01e976 (0xb3ce bytes) 000831 1789044057.604830 read 0x020ba0 thru 0x023cee (0x314f bytes) 000833 1789044057.632474 MIRROR_RESYNC 2 mirrors 000834 1789044059.220383 mapwrite 0x02797d thru 0x027fba (0x63e bytes) 000835 1789044059.455771 punch from 0x002d65 to 0x004b43 (0x1ddf bytes) 000836 1789044059.472771 mapread 0x0286a0 thru 0x02dd59 (0x56ba bytes) 000837 1789044059.486309 trunc from 0x02dd5a to 0x000439 (0x2d920 bytes) 000841 1789044059.526980 punch from 0x000186 to 0x000439 (0x2b4 bytes) 000842 1789044059.542328 trunc from 0x00043a to 0x01f545 (0x1f10c bytes) 000843 1789044059.578443 punch from 0x009bbc to 0x00cfa6 (0x33eb bytes) 000844 1789044059.592526 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000846 1789044059.998062 mapwrite 0x0367e8 thru 0x03d2d9 (0x6af2 bytes) 000847 1789044060.196842 read 0x0194d4 thru 0x0235a4 (0xa0d1 bytes) 000848 1789044060.217242 read 0x007a28 thru 0x017619 (0xfbf2 bytes) 000849 1789044060.230786 write 0x0131bd thru 0x01b1c4 (0x8008 bytes) 000850 1789044060.233387 write 0x00af73 thru 0x019b22 (0xebb0 bytes) 000852 1789044060.238405 write 0x00af20 thru 0x00d21d (0x22fe bytes) 000855 1789044060.248829 mapwrite 0x0127c6 thru 0x013bab (0x13e6 bytes) 000856 1789044060.393370 punch from 0x00599f to 0x0125e5 (0xcc47 bytes) 000857 1789044060.407174 mapwrite 0x0349bc thru 0x037dce (0x3413 bytes) 000858 1789044060.511468 read 0x03041d thru 0x03893d (0x8521 bytes) 000859 1789044060.512288 write 0x03b77b thru 0x03ffff (0x4885 bytes) 000860 1789044060.531234 mapwrite 0x00e0ec thru 0x0141a8 (0x60bd bytes) 000861 1789044060.720764 mapwrite 0x00eb44 thru 0x016aeb (0x7fa8 bytes) 000862 1789044060.816102 trunc from 0x040000 to 0x022f28 (0x1d0d7 bytes) 000865 1789044060.913485 mapread 0x007d1c thru 0x00f53b (0x7820 bytes) 000867 1789044060.932711 mapwrite 0x02185f thru 0x027848 (0x5fea bytes) 000869 1789044061.127797 trunc from 0x027849 to 0x038065 (0x1081d bytes) 000871 1789044061.161575 punch from 0x02ade2 to 0x02ae3a (0x59 bytes) 000872 1789044061.169552 read 0x014802 thru 0x0238da (0xf0d9 bytes) 000873 1789044061.170959 mapread 0x02b827 thru 0x037dbf (0xc599 bytes) 000876 1789044061.193509 trunc from 0x038066 to 0x02aba8 (0xd4bd bytes) 000879 1789044061.231986 MIRROR_EXTEND to 2 mirrors 000880 1789044062.473297 write 0x02b561 thru 0x02c53e (0xfde bytes) 000881 1789044062.602605 trunc from 0x02c53f to 0x02cd26 (0x7e8 bytes) 000882 1789044062.619730 write 0x03c3a4 thru 0x03ffff (0x3c5c bytes) 000883 1789044062.620802 mapwrite 0x038fcc thru 0x03ffff (0x7034 bytes) 000884 1789044062.822276 mapread 0x01d9dd thru 0x021cf3 (0x4317 bytes) 000885 1789044062.844799 trunc from 0x040000 to 0x001af1 (0x3e50e bytes) 000886 1789044062.869036 mapwrite 0x01a242 thru 0x0275a0 (0xd35f bytes) 000887 1789044063.077726 mapwrite 0x0269b9 thru 0x0327c4 (0xbe0c bytes) 000888 1789044063.269456 write 0x00bd39 thru 0x00d0dc (0x13a4 bytes) 000889 1789044063.294278 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000890 1789044063.538398 mapwrite 0x01a2b4 thru 0x01caf4 (0x2841 bytes) 000891 1789044063.773409 write 0x03382a thru 0x03ffff (0xc7d6 bytes) 000892 1789044063.774933 trunc from 0x040000 to 0x02763c (0x189c3 bytes) 000896 1789044063.881398 trunc from 0x02763d to 0x0089e4 (0x1ec58 bytes) 000897 1789044063.934253 trunc from 0x0089e5 to 0x020de2 (0x183fe bytes) 000899 1789044063.965933 punch from 0x019b4a to 0x020de2 (0x7299 bytes) 000902 1789044063.985667 mapread 0x009e3f thru 0x018d0e (0xeed0 bytes) 000903 1789044064.016728 punch from 0x00bcef to 0x01714f (0xb461 bytes) 000904 1789044064.021428 punch from 0x0191f9 to 0x01c0c6 (0x2ece bytes) 000906 1789044064.026547 trunc from 0x020de3 to 0x018dbe (0x8024 bytes) 000907 1789044064.041540 mapread 0x00bb45 thru 0x010db9 (0x5275 bytes) 000908 1789044064.051170 punch from 0x00b8cb to 0x0121e5 (0x691b bytes) 000910 1789044064.056496 mapwrite 0x014d6a thru 0x0197ad (0x4a44 bytes) 000912 1789044064.188886 mapwrite 0x01735a thru 0x01e85c (0x7503 bytes) 000914 1789044064.396124 punch from 0x009861 to 0x016b26 (0xd2c6 bytes) 000916 1789044064.408729 mapwrite 0x03d9c9 thru 0x03ffff (0x2637 bytes) 000917 1789044064.591687 punch from 0x009aad to 0x00e61a (0x4b6e bytes) 000918 1789044064.612885 mapwrite 0x023026 thru 0x0242c1 (0x129c bytes) 000919 1789044064.739814 write 0x01d100 thru 0x027cac (0xabad bytes) 000920 1789044064.741563 write 0x018300 thru 0x027f3e (0xfc3f bytes) 000921 1789044064.745247 read 0x006e68 thru 0x00f494 (0x862d bytes) 000925 1789044064.778211 mapwrite 0x017aa9 thru 0x021883 (0x9ddb bytes) 000926 1789044064.851316 punch from 0x01e474 to 0x02271a (0x42a7 bytes) 000928 1789044064.863788 mapwrite 0x03783b thru 0x03ffff (0x87c5 bytes) 000929 1789044064.895444 punch from 0x037345 to 0x03d7ad (0x6469 bytes) 000932 1789044064.905018 write 0x008b40 thru 0x0183e5 (0xf8a6 bytes) 000933 1789044064.907384 read 0x036bc5 thru 0x03fe01 (0x923d bytes) 000934 1789044064.934789 mapread 0x0363c5 thru 0x03ffff (0x9c3b bytes) 000938 1789044064.944580 punch from 0x034d92 to 0x03ffff (0xb26e bytes) 000939 1789044065.003645 punch from 0x03123a to 0x039dae (0x8b75 bytes) 000940 1789044065.012824 read 0x02e723 thru 0x03383c (0x511a bytes) 000942 1789044065.031462 write 0x023f4c thru 0x031c9a (0xdd4f bytes) 000943 1789044065.033358 mapwrite 0x0201ec thru 0x0290a8 (0x8ebd bytes) 000944 1789044065.074577 mapwrite 0x023823 thru 0x02ac65 (0x7443 bytes) 000946 1789044065.147809 mapwrite 0x01ee81 thru 0x02e6bb (0xf83b bytes) 000947 1789044065.248165 punch from 0x012439 to 0x012793 (0x35b bytes) 000949 1789044065.260481 MIRROR_EXTEND to 2 mirrors 000951 1789044066.475124 MIRROR_RESYNC 2 mirrors 000952 1789044067.203255 write 0x008d8a thru 0x00c552 (0x37c9 bytes) 000953 1789044067.332448 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000956 1789044067.669355 MIRROR_EXTEND to 2 mirrors 000957 1789044068.729346 MIRROR_RESYNC 2 mirrors 000961 1789044069.273086 mapwrite 0x024084 thru 0x031761 (0xd6de bytes) 000962 1789044069.641750 mapwrite 0x03869f thru 0x03ffff (0x7961 bytes) 000965 1789044069.779480 mapwrite 0x0044a2 thru 0x0078b1 (0x3410 bytes) 000967 1789044069.926566 mapread 0x018ddf thru 0x02715d (0xe37f bytes) 000968 1789044069.958754 mapwrite 0x022646 thru 0x02dd7b (0xb736 bytes) 000971 1789044070.031762 punch from 0x023ab5 to 0x028de5 (0x5331 bytes) 000972 1789044070.043345 write 0x013440 thru 0x01cb12 (0x96d3 bytes) 000973 1789044070.045138 punch from 0x01b6fb to 0x01e8c9 (0x31cf bytes) 000974 1789044070.100823 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x19a:0x0]: Invalid argument (22) 000975 1789044070.253840 trunc from 0x040000 to 0x01ce7a (0x23185 bytes) 000977 1789044070.278870 mapwrite 0x0378ae thru 0x03ffff (0x8752 bytes) 000979 1789044070.475487 mapread 0x0168fa thru 0x02166e (0xad75 bytes) 000980 1789044070.507090 read 0x037988 thru 0x038427 (0xaa0 bytes) 000981 1789044070.507767 punch from 0x038966 to 0x03ffff (0x769a bytes) 000983 1789044070.532253 mapread 0x01a47d thru 0x01c2cc (0x1e50 bytes) 000984 1789044070.535987 write 0x02a1f4 thru 0x031dd9 (0x7be6 bytes) 000986 1789044070.569095 punch from 0x006613 to 0x00dda0 (0x778e bytes) 000987 1789044070.609418 read 0x03b01d thru 0x03ffff (0x4fe3 bytes) 000989 1789044070.626857 mapwrite 0x0067c4 thru 0x00ca38 (0x6275 bytes) 000991 1789044070.728819 punch from 0x00992b to 0x012dda (0x94b0 bytes) 000992 1789044070.737455 read 0x01f912 thru 0x02f5d9 (0xfcc8 bytes) 000993 1789044070.738348 write 0x0231da thru 0x0256ee (0x2515 bytes) 000995 1789044070.741532 punch from 0x0168cc to 0x0209fb (0xa130 bytes) 000996 1789044070.757526 write 0x039667 thru 0x03b1c6 (0x1b60 bytes) 000999 1789044070.803690 MIRROR_EXTEND to 2 mirrors 001000 1789044071.985759 mapread 0x032770 thru 0x0388f4 (0x6185 bytes) All operations completed A-OK! PASS 70a (102s) == sanity-flr test 71: check mirror extend parent fid ==== 08:41:17 (1789044077) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 08:41:20 (1789044080) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.322209 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246551 s, 4.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 08:41:31 (1789044091) Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre2 Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6793312 .. Extending file size to 3608192 .. Extending file size to 4688544 .. Extending file size to 6851008 .. Extending file size to 7189888 .. Extending file size to 5734624 .. Extending file size to 6156352 .. Extending file size to 7609440 .. done Extending file size to 8224864 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 6568832 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6947776 .. Extending file size to 7305056 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8668128 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8750624 .. Waiting 57309 57311 57312 57314 57316 done 192.168.206.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg634-client.virtnet /mnt/lustre2 (opts:) 192.168.206.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg634-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:42:56 (1789044176) Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre2 Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 753824 .. Extending file size to 7759872 .. failed Extending file size to 4201216 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8625696 .. Extending file size to 8648768 .. Extending file size to 8799648 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8802944 .. Extending file size to 8831424 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58834 58835 58837 58838 58839 done 192.168.206.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg634-client.virtnet /mnt/lustre2 (opts:) 192.168.206.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg634-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:44:17 (1789044257) Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0884584 s, 23.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2568 s, 8.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3044bac2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg634-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:44:33 (1789044273) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0940721 s, 22.3 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 08:44:39 (1789044279) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:44:46 (1789044286) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174705 s, 12.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:44:52 (1789044292) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192796 s, 10.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:45:00 (1789044300) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265186 s, 7.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:45:33 (1789044333) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg634-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833138 s, 12.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:46:29 (1789044389) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg634-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246234 s, 8.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:47:26 (1789044446) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg634-server mds-ost sync done. Creating new pool test_204f oleg634-server: Pool lustre.test_204f created Adding targets to pool oleg634-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg634-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240577 s, 8.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg634-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg634-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg634-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:48:50 (1789044530) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:48:58 (1789044538) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:49:07 (1789044547) oleg634-server: Pool lustre.test_206 created Adding targets to pool oleg634-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg634-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg634-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg634-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg634-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:49:35 (1789044575) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685336 s, 15.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288792 s, 36.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x44d50d5 Mirror 3: 0x44d50d5 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:49:49 (1789044589) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:49:53 (1789044593) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:49:57 (1789044597) Starting client: oleg634-client.virtnet: -o user_xattr,flock 192.168.206.134@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.150102 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x11d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x16b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x16b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x16b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1850000 Read this (4 bytes): BBBB 192.168.206.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg634-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:50:37 (1789044637) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0325295 s, 252 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:50:44 (1789044644) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527705 s, 19.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x209 lmm_fid: [0x200000404:0x209:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x209 lmm_fid: [0x200000404:0x209:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834441 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 08:50:46.000000000 -0400 Modify: 2026-09-10 08:50:46.000000000 -0400 Change: 2026-09-10 08:50:47.000000000 -0400 Birth: 2026-09-10 08:50:46.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:50:53 (1789044653) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:50:56 (1789044656) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.810127 s, 50.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0692308 s, 59.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 10 08:51 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 10 08:51 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:51:07 (1789044667) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:51:16 (1789044676) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1887 sec =========== 08:51:25 (1789044685) === sanity-flr: start cleanup 08:51:27 (1789044687) === === sanity-flr: finish cleanup 08:51:34 (1789044694) ===