-----============= acceptance-small: sanity-flr ============----- Tue Jun 23 09:45:01 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg337-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:45:22 (1782222322) === oleg337-client.virtnet: executing check_config_client /mnt/lustre oleg337-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg337-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d1845144800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d1845144800.idle_timeout=debug disable quota as required oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:46:04 (1782222364) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7615] == sanity-flr test 0a: lfs mirror create with -N option == 09:46:08 (1782222368) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:46:29 (1782222389) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:46:33 (1782222393) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:46:37 (1782222397) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:47:07 (1782222427) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:47:11 (1782222431) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:47:14 (1782222434) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d1845144800.stats=clear osc.lustre-OST0001-osc-ffff8d1845144800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:47:23 (1782222443) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:47:36 (1782222456) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:47:47 (1782222467) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:47:51 (1782222471) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:48:01 (1782222481) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 09:48:12 (1782222492) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:48:20 (1782222500) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:48:23 (1782222503) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:48:31 (1782222511) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:48:41 (1782222521) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:48:48 (1782222528) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.745389 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148745 s, 7.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:48:58 (1782222538) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.437033 s, 26.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:49:07 (1782222547) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:49:18 (1782222558) Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg337-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (129s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:51:28 (1782222688) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.19529 s, 16.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:52:15 (1782222735) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (73s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:53:29 (1782222809) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.6751 s, 11.2 MB/s -rw-r--r-- 1 root root 52428800 Jun 23 09:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.218s user 0m0.001s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d1845144800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.287s user 0m0.001s sys 0m0.164s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg337-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg337-server oleg337-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.488s user 0m0.001s sys 0m0.183s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg337-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg337-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d1845144800.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:55:02 (1782222902) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:55:06 (1782222906) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:55:09 (1782222909) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:55:12 (1782222912) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 09:55:20 (1782222920) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.443847 s, 28.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:55:31 (1782222931) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421394 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314761 s, 33.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:55:38 (1782222938) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:55:47 (1782222947) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0026671 s, 384 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00387757 s, 264 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00312386 s, 328 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00144217 s, 710 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 09:55:58 (1782222958) create test files with size 15733369 .. c6d1fd3fadc7602c0e55542697e0965f - 13d881eb5d308ea7925888cf629fa958 - 20c28d4d28d4861dd0926632d98003be - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 09:56:23 (1782222983) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:56:38 (1782222998) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:56:44 (1782223004) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163007 s, 6.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836028 s, 12.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 09:56:53 (1782223013) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119701 s, 17.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.393622 s, 10.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263598 s, 8.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 09:57:10 (1782223030) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 09:57:14 (1782223034) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:57:17 (1782223037) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302694 s, 34.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 09:57:26 (1782223046) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:57:29 (1782223049) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:57:39 (1782223059) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264746 s, 39.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:57:51 (1782223071) llite.lustre-ffff8d1844fe9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.261487 s, 40.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg337-client.virtnet /mnt/lustre (opts:) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d1866368800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:58:05 (1782223085) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:58:43 (1782223123) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:59:24 (1782223164) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 09:59:28 (1782223168) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.359153 s, 8.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.248305 s, 12.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x47b72945 (nosync,stale) Mirror 2: 0x47b72945 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 40d4ad23c6f197b0d0f78c9bf12f962b - mirror 1 checksum: 31a37cbd2ae25495011e2ae0ae226238 - mirror 2 checksum: 40d4ad23c6f197b0d0f78c9bf12f962b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T09:59:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-23T09:59:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 40d4ad23c6f197b0d0f78c9bf12f962b - mirror 1 checksum: 31a37cbd2ae25495011e2ae0ae226238 - mirror 2 checksum: 31a37cbd2ae25495011e2ae0ae226238 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T09:59:32-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-23T09:59:32-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:59:43 (1782223183) Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 09:59:50 (1782223190) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 10:00:24 (1782223224) targets are mounted 10:00:24 (1782223224) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287797 s, 36.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:00:47 (1782223247) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:00:51 (1782223251) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:00:54 (1782223254) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.012458 s, 1.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:01:03 (1782223263) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:01:06 (1782223266) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:01:10 (1782223270) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:01:39 (1782223299) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:02:09 (1782223329) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:02:35 (1782223355) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:03:01 (1782223381) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 10:03:09 (1782223389) mirror create pid 53787 mirror split pid 53788 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53787 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53788 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 10:04:16 (1782223456) Seed set to 3458 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000005 1782223458.626418 mapwrite 0x017e83 thru 0x01f34f (0x74cd bytes) 000007 1782223458.744032 read 0x0165c0 thru 0x01ab0d (0x454e bytes) 000010 1782223458.748622 read 0x01cb68 thru 0x01f34f (0x27e8 bytes) 000013 1782223458.751884 read 0x010eae thru 0x014200 (0x3353 bytes) 000014 1782223458.771361 mapread 0x013a7c thru 0x01f34f (0xb8d4 bytes) 000015 1782223458.772795 write 0x00b3ee thru 0x014c9e (0x98b1 bytes) 000017 1782223458.785338 mapread 0x00a867 thru 0x00d026 (0x27c0 bytes) truncating to largest ever: 0x2f71e 000018 1782223458.788442 trunc from 0x01f350 to 0x02f71d (0x103ce bytes) 000020 1782223458.816493 write 0x017c62 thru 0x01ddc9 (0x6168 bytes) 000021 1782223458.818484 read 0x008fe0 thru 0x01760c (0xe62d bytes) 000023 1782223458.828542 mapwrite 0x01cf6b thru 0x0228ee (0x5984 bytes) 000028 1782223458.944174 write 0x028323 thru 0x028fa6 (0xc84 bytes) 000030 1782223458.946044 mapread 0x01d430 thru 0x02b870 (0xe441 bytes) 000033 1782223458.979612 trunc from 0x02f71e to 0x01f077 (0x106a6 bytes) 000034 1782223459.040271 read 0x003a24 thru 0x00f7d3 (0xbdb0 bytes) 000038 1782223459.053203 write 0x02991e thru 0x02e3a4 (0x4a87 bytes) 000039 1782223459.057968 mapread 0x0260d1 thru 0x02e3a4 (0x82d4 bytes) 000043 1782223459.103574 mapread 0x01fb85 thru 0x026068 (0x64e4 bytes) 000044 1782223459.109843 mapread 0x010e08 thru 0x0141b3 (0x33ac bytes) 000052 1782223459.134390 mapwrite 0x012b5a thru 0x018a2d (0x5ed4 bytes) 000054 1782223459.215844 mapread 0x021592 thru 0x029593 (0x8002 bytes) 000057 1782223459.221112 mapread 0x0123a2 thru 0x01a6c0 (0x831f bytes) 000063 1782223459.227741 trunc from 0x02e3a5 to 0x00f9d9 (0x1e9cb bytes) 000064 1782223459.263780 write 0x011bd0 thru 0x01280a (0xc3b bytes) 000068 1782223459.270845 mapwrite 0x0344e3 thru 0x039c91 (0x57af bytes) 000072 1782223459.398724 mapread 0x001c24 thru 0x001d4a (0x127 bytes) 000074 1782223459.406324 mapread 0x00a97a thru 0x0122eb (0x7972 bytes) truncating to largest ever: 0x37f8f 000083 1782223459.432814 trunc from 0x039c92 to 0x037f8e (0x1d03 bytes) 000084 1782223459.476166 mapread 0x016186 thru 0x019324 (0x319f bytes) 000085 1782223459.502465 mapwrite 0x0001f9 thru 0x00ebc0 (0xe9c8 bytes) 000086 1782223459.585892 mapwrite 0x00846b thru 0x00eb2b (0x66c1 bytes) 000087 1782223459.637328 trunc from 0x037f8f to 0x0139bd (0x245d1 bytes) 000088 1782223459.657488 write 0x0008b3 thru 0x00e83f (0xdf8d bytes) 000089 1782223459.661478 read 0x00857b thru 0x00a982 (0x2408 bytes) 000091 1782223459.664325 mapwrite 0x02272d thru 0x02e035 (0xb909 bytes) 000093 1782223459.869926 read 0x020643 thru 0x02e035 (0xd9f3 bytes) 000095 1782223459.874448 write 0x007df4 thru 0x00a8e8 (0x2af5 bytes) 000096 1782223459.876115 write 0x007d40 thru 0x010a73 (0x8d34 bytes) 000097 1782223459.877479 read 0x012b5e thru 0x018525 (0x59c8 bytes) 000098 1782223459.885977 read 0x01abdd thru 0x0272d2 (0xc6f6 bytes) 000100 1782223459.896329 mapread 0x0029c1 thru 0x008b48 (0x6188 bytes) 000101 1782223459.900987 mapread 0x027b1b thru 0x02ad9c (0x3282 bytes) 000104 1782223459.911742 read 0x003023 thru 0x00406d (0x104b bytes) 000105 1782223459.915218 mapread 0x013099 thru 0x01bc24 (0x8b8c bytes) 000107 1782223459.925919 mapwrite 0x016cd3 thru 0x0257f0 (0xeb1e bytes) 000110 1782223460.029015 trunc from 0x02e036 to 0x019519 (0x14b1c bytes) 000111 1782223460.079927 write 0x014d2f thru 0x01808e (0x3360 bytes) 000113 1782223460.081785 mapwrite 0x0295db thru 0x03823e (0xec64 bytes) 000114 1782223460.272613 mapread 0x02593c thru 0x02bd09 (0x63ce bytes) 000117 1782223460.274905 read 0x00e8d2 thru 0x01aa41 (0xc170 bytes) 000121 1782223460.279133 mapread 0x02b449 thru 0x030205 (0x4dbd bytes) 000125 1782223460.286965 write 0x01be49 thru 0x023ebe (0x8076 bytes) 000126 1782223460.305742 MIRROR_EXTEND to 2 mirrors 000128 1782223460.856379 read 0x023ff8 thru 0x027979 (0x3982 bytes) 000130 1782223460.874829 write 0x0180b6 thru 0x020117 (0x8062 bytes) 000131 1782223461.029786 mapwrite 0x01e702 thru 0x024ea8 (0x67a7 bytes) 000132 1782223461.151840 write 0x03299c thru 0x035853 (0x2eb8 bytes) 000135 1782223461.182061 trunc from 0x03823f to 0x03234f (0x5eef bytes) 000136 1782223461.260854 MIRROR_RESYNC 2 mirrors 000137 1782223462.145301 write 0x0300fd thru 0x03b642 (0xb546 bytes) 000139 1782223462.244531 MIRROR_RESYNC 2 mirrors 000140 1782223463.104327 read 0x01c987 thru 0x02bd34 (0xf3ae bytes) 000142 1782223463.138470 read 0x012acd thru 0x01744f (0x4983 bytes) 000146 1782223463.157622 read 0x00e8c2 thru 0x01e5b8 (0xfcf7 bytes) 000148 1782223463.158789 mapwrite 0x0259f4 thru 0x02f074 (0x9681 bytes) 000149 1782223463.409740 mapread 0x01f319 thru 0x01fb03 (0x7eb bytes) 000152 1782223463.430584 MIRROR_RESYNC 2 mirrors 000154 1782223464.284947 mapread 0x00b954 thru 0x014752 (0x8dff bytes) 000156 1782223464.338900 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000158 1782223464.596090 write 0x00a17e thru 0x013d2a (0x9bad bytes) 000159 1782223464.634031 mapread 0x02d852 thru 0x037485 (0x9c34 bytes) 000160 1782223464.683792 write 0x003957 thru 0x0081c6 (0x4870 bytes) 000161 1782223464.708623 mapread 0x012d76 thru 0x0198da (0x6b65 bytes) 000162 1782223464.721293 mapread 0x03abd4 thru 0x03b642 (0xa6f bytes) 000164 1782223464.730975 mapread 0x02b972 thru 0x038ee4 (0xd573 bytes) 000167 1782223464.735890 mapwrite 0x0003c8 thru 0x00621d (0x5e56 bytes) 000169 1782223464.815557 read 0x0187d2 thru 0x020cd1 (0x8500 bytes) 000172 1782223464.827008 mapread 0x02976d thru 0x02cc9c (0x3530 bytes) 000175 1782223464.833449 read 0x02a892 thru 0x02d3b9 (0x2b28 bytes) 000177 1782223464.836025 trunc from 0x03b643 to 0x02abde (0x10a64 bytes) 000178 1782223464.869750 read 0x011528 thru 0x01670f (0x51e8 bytes) 000179 1782223464.870538 mapread 0x0213b5 thru 0x02abde (0x982a bytes) 000181 1782223464.884888 mapwrite 0x02010b thru 0x0289d8 (0x88ce bytes) 000182 1782223464.954178 read 0x01beaf thru 0x0251ee (0x9340 bytes) 000186 1782223464.963626 mapwrite 0x0212b7 thru 0x030023 (0xed6d bytes) 000187 1782223465.037758 write 0x03e830 thru 0x03ffff (0x17d0 bytes) 000188 1782223465.044384 MIRROR_EXTEND to 2 mirrors 000190 1782223465.989775 read 0x008f33 thru 0x00fe5b (0x6f29 bytes) 000192 1782223466.027164 mapread 0x00f91e thru 0x01921e (0x9901 bytes) 000195 1782223466.131589 read 0x03733a thru 0x03ffff (0x8cc6 bytes) 000196 1782223466.152541 mapread 0x038697 thru 0x03cc13 (0x457d bytes) 000197 1782223466.156822 mapwrite 0x01944b thru 0x01caa4 (0x365a bytes) 000198 1782223466.373953 mapwrite 0x0153e6 thru 0x017045 (0x1c60 bytes) 000200 1782223466.431106 mapread 0x019c2b thru 0x01e3e8 (0x47be bytes) 000204 1782223466.453377 trunc from 0x040000 to 0x03352f (0xcad0 bytes) 000205 1782223466.513891 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33530): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33530): Mirror 2: 0x798b5877 (stale) CRC-32 checksum value for chunk [0, 0x33530): Mirror 1: 0x25533e3c Mirror 3: 0x25533e3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33530: skipped 000206 1782223467.777781 read 0x0260c4 thru 0x02a061 (0x3f9e bytes) 000207 1782223467.811480 write 0x020304 thru 0x02c8d3 (0xc5d0 bytes) 000209 1782223467.984956 read 0x0169ce thru 0x02088a (0x9ebd bytes) 000211 1782223468.005306 read 0x007985 thru 0x008abf (0x113b bytes) 000212 1782223468.007325 write 0x025a8a thru 0x0334a3 (0xda1a bytes) 000213 1782223468.010435 write 0x0327a0 thru 0x03a753 (0x7fb4 bytes) 000214 1782223468.017153 write 0x000d5f thru 0x009f43 (0x91e5 bytes) 000215 1782223468.022316 trunc from 0x03a754 to 0x02d2d0 (0xd483 bytes) 000217 1782223468.063588 mapwrite 0x019033 thru 0x027eb2 (0xee80 bytes) 000218 1782223468.229977 mapread 0x00654d thru 0x0078a5 (0x1359 bytes) 000220 1782223468.236625 read 0x0248ff thru 0x0282bc (0x39be bytes) 000222 1782223468.240560 mapread 0x01582a thru 0x01e6e6 (0x8ebd bytes) 000224 1782223468.248502 trunc from 0x02d2d1 to 0x0292cc (0x4004 bytes) 000226 1782223468.295397 read 0x00d8c9 thru 0x013e01 (0x6539 bytes) 000227 1782223468.298983 trunc from 0x0292cd to 0x023b52 (0x577a bytes) 000229 1782223468.367797 trunc from 0x023b53 to 0x01d700 (0x6452 bytes) 000234 1782223468.413962 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d701): Mirror 3: 0x4f96a159 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d701): Mirror 3: 0x4f96a159 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d701: skipped 000235 1782223468.802432 mapread 0x003c5f thru 0x00a652 (0x69f4 bytes) 000236 1782223468.822675 write 0x007370 thru 0x010575 (0x9206 bytes) 000238 1782223468.824630 trunc from 0x01d701 to 0x033a6f (0x1636f bytes) 000239 1782223468.854694 mapread 0x028ebf thru 0x029198 (0x2da bytes) 000240 1782223468.872591 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33a70): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a70): Mirror 3: 0x74afa812 (stale) CRC-32 checksum value for chunk [0, 0x33a70): Mirror 1: 0x6b0c5dc4 Mirror 4: 0x6b0c5dc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a70: skipped 000241 1782223470.132996 write 0x030ea7 thru 0x033353 (0x24ad bytes) 000242 1782223470.207526 mapread 0x0107d3 thru 0x016dbe (0x65ec bytes) 000243 1782223470.227031 read 0x0256cd thru 0x028b66 (0x349a bytes) 000244 1782223470.237617 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33a70): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a70): Mirror 3: 0x74afa812 (stale) Mirror 4: 0x6b0c5dc4 (stale) CRC-32 checksum value for chunk [0, 0x33a70): Mirror 1: 0xdce3732d Mirror 5: 0xdce3732d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a70: skipped 000245 1782223471.209344 write 0x021881 thru 0x02216a (0x8ea bytes) 000249 1782223471.349067 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a70): Mirror 4: 0x6b0c5dc4 (stale) Mirror 5: 0xdce3732d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a70): Mirror 4: 0x6b0c5dc4 (stale) Mirror 5: 0xdce3732d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a70: skipped 000250 1782223471.949413 read 0x01342c thru 0x019fce (0x6ba3 bytes) 000255 1782223471.965891 read 0x00a285 thru 0x013c0a (0x9986 bytes) 000256 1782223471.992881 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a70): Mirror 5: 0xdce3732d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a70): Mirror 5: 0xdce3732d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a70: skipped 000257 1782223472.341408 mapread 0x0142e3 thru 0x02398e (0xf6ac bytes) 000261 1782223472.374666 read 0x02bc16 thru 0x033a6f (0x7e5a bytes) 000264 1782223472.386785 read 0x00f949 thru 0x01d495 (0xdb4d bytes) 000270 1782223472.395524 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000271 1782223472.535043 trunc from 0x033a70 to 0x01ce6e (0x16c01 bytes) 000272 1782223472.580284 write 0x02b3fe thru 0x02c4a9 (0x10ac bytes) 000273 1782223472.582240 read 0x021db5 thru 0x02c4a9 (0xa6f5 bytes) 000274 1782223472.589010 mapwrite 0x003931 thru 0x00c5ef (0x8cbf bytes) 000278 1782223472.661882 mapread 0x02717b thru 0x02c4a9 (0x532f bytes) 000279 1782223472.665018 read 0x00757b thru 0x010ad0 (0x9556 bytes) 000280 1782223472.666620 mapwrite 0x02aea4 thru 0x02aef0 (0x4d bytes) 000282 1782223472.711163 read 0x00597c thru 0x011cfd (0xc382 bytes) 000288 1782223472.727535 read 0x017f8c thru 0x022423 (0xa498 bytes) 000289 1782223472.728442 mapread 0x01b840 thru 0x02981b (0xdfdc bytes) 000290 1782223472.731463 mapread 0x01c036 thru 0x0283ba (0xc385 bytes) 000295 1782223472.737115 write 0x03cb0a thru 0x03ffff (0x34f6 bytes) truncating to largest ever: 0x3a55d 000297 1782223472.739399 trunc from 0x040000 to 0x03a55c (0x5aa3 bytes) 000299 1782223472.760771 trunc from 0x03a55d to 0x01136e (0x291ee bytes) 000300 1782223472.783721 mapread 0x006df8 thru 0x010d1c (0x9f25 bytes) 000303 1782223472.786738 write 0x00531c thru 0x01511c (0xfe01 bytes) 000304 1782223472.788531 mapread 0x014495 thru 0x01511c (0xc88 bytes) 000307 1782223472.791372 read 0x00c7ee thru 0x00e2f5 (0x1b08 bytes) 000308 1782223472.792296 write 0x029389 thru 0x0326a8 (0x9320 bytes) 000310 1782223472.794593 trunc from 0x0326a9 to 0x034fc0 (0x2918 bytes) 000311 1782223472.817710 read 0x022afd thru 0x02e9b6 (0xbeba bytes) 000312 1782223472.832035 MIRROR_EXTEND to 2 mirrors 000314 1782223473.629738 trunc from 0x034fc1 to 0x020d81 (0x1423f bytes) 000316 1782223473.732581 mapwrite 0x02bc2c thru 0x039935 (0xdd0a bytes) 000317 1782223473.914887 trunc from 0x039936 to 0x039764 (0x1d1 bytes) 000318 1782223473.944993 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000320 1782223474.132194 trunc from 0x039765 to 0x03967f (0xe5 bytes) 000321 1782223474.152565 write 0x0079d1 thru 0x00f37d (0x79ad bytes) 000324 1782223474.166241 read 0x000547 thru 0x00f3f2 (0xeeac bytes) 000325 1782223474.181376 MIRROR_EXTEND to 2 mirrors 000326 1782223474.942408 trunc from 0x039680 to 0x037786 (0x1ef9 bytes) 000330 1782223475.139631 read 0x01f91c thru 0x028ec8 (0x95ad bytes) 000333 1782223475.172248 mapread 0x02883a thru 0x02b4c2 (0x2c89 bytes) 000334 1782223475.197320 read 0x005dcd thru 0x00c61a (0x684e bytes) 000335 1782223475.221994 read 0x0090af thru 0x00f251 (0x61a3 bytes) 000336 1782223475.222994 trunc from 0x037787 to 0x00a3f8 (0x2d38e bytes) 000337 1782223475.284329 mapread 0x004041 thru 0x00a3f8 (0x63b8 bytes) 000340 1782223475.287748 mapread 0x0099f1 thru 0x00a3f8 (0xa08 bytes) 000341 1782223475.289485 mapread 0x004732 thru 0x00a3f8 (0x5cc7 bytes) 000342 1782223475.292976 mapread 0x007445 thru 0x00a3f8 (0x2fb4 bytes) 000344 1782223475.294725 write 0x03d41a thru 0x03ffff (0x2be6 bytes) 000346 1782223475.306063 trunc from 0x040000 to 0x0201da (0x1fe25 bytes) 000347 1782223475.338864 mapread 0x00438c thru 0x0061c1 (0x1e36 bytes) 000348 1782223475.340800 trunc from 0x0201db to 0x0210a6 (0xecc bytes) 000351 1782223475.373567 read 0x004739 thru 0x01210f (0xd9d7 bytes) 000352 1782223475.386089 write 0x03e225 thru 0x03ffff (0x1ddb bytes) 000353 1782223475.387152 read 0x004f95 thru 0x006192 (0x11fe bytes) 000355 1782223475.389109 write 0x0278d8 thru 0x02875b (0xe84 bytes) 000359 1782223475.398435 read 0x01ccb8 thru 0x024a82 (0x7dcb bytes) 000360 1782223475.400373 trunc from 0x040000 to 0x00199a (0x3e665 bytes) 000362 1782223475.418890 write 0x00e4d8 thru 0x013133 (0x4c5c bytes) 000363 1782223475.420428 mapread 0x0019cc thru 0x001fb3 (0x5e8 bytes) 000364 1782223475.429250 read 0x005d10 thru 0x00cfca (0x72bb bytes) 000365 1782223475.438731 mapwrite 0x022ca5 thru 0x024c3f (0x1f9b bytes) 000366 1782223475.560591 write 0x00f64e thru 0x0110cf (0x1a82 bytes) 000369 1782223475.563526 mapwrite 0x025590 thru 0x032d26 (0xd797 bytes) 000370 1782223475.664669 mapwrite 0x00656d thru 0x00f077 (0x8b0b bytes) 000374 1782223475.745932 read 0x020ced thru 0x026c51 (0x5f65 bytes) 000376 1782223475.748541 read 0x00bfa1 thru 0x011029 (0x5089 bytes) 000377 1782223475.749961 read 0x0236a0 thru 0x02b45f (0x7dc0 bytes) 000378 1782223475.751831 write 0x02e22e thru 0x03711f (0x8ef2 bytes) 000379 1782223475.754055 write 0x01406d thru 0x01411c (0xb0 bytes) 000380 1782223475.756921 mapread 0x00b9c0 thru 0x00d627 (0x1c68 bytes) 000382 1782223475.760021 read 0x02ee76 thru 0x036429 (0x75b4 bytes) 000384 1782223475.763367 write 0x02d7dc thru 0x03af2d (0xd752 bytes) 000385 1782223475.771903 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3af2e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af2e): Mirror 2: 0x15e1bb4e (stale) CRC-32 checksum value for chunk [0, 0x3af2e): Mirror 1: 0xedf14fef Mirror 3: 0xedf14fef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af2e: skipped 000389 1782223476.889932 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000390 1782223477.227668 write 0x009892 thru 0x00b9f9 (0x2168 bytes) 000393 1782223477.332050 read 0x03689d thru 0x03af2d (0x4691 bytes) 000395 1782223477.350955 write 0x0235af thru 0x02b5ba (0x800c bytes) 000398 1782223477.368555 mapwrite 0x0192e1 thru 0x01e81d (0x553d bytes) 000399 1782223477.464785 read 0x001083 thru 0x010162 (0xf0e0 bytes) truncating to largest ever: 0x3f366 000402 1782223477.475995 trunc from 0x03af2e to 0x03f365 (0x4438 bytes) 000405 1782223477.512369 mapread 0x01a7f9 thru 0x023bbc (0x93c4 bytes) 000407 1782223477.514391 mapwrite 0x01f6e4 thru 0x025c6f (0x658c bytes) 000410 1782223477.595323 write 0x037a79 thru 0x03f204 (0x778c bytes) 000412 1782223477.619693 read 0x006193 thru 0x00dec0 (0x7d2e bytes) 000415 1782223477.627962 write 0x01767a thru 0x01d0eb (0x5a72 bytes) 000417 1782223477.631612 read 0x0075ad thru 0x007d56 (0x7aa bytes) 000419 1782223477.642834 trunc from 0x03f366 to 0x005e8d (0x394d8 bytes) 000421 1782223477.705582 mapread 0x000c29 thru 0x005e8d (0x5265 bytes) 000422 1782223477.706885 write 0x023762 thru 0x0330a0 (0xf93f bytes) 000423 1782223477.712945 mapwrite 0x013df0 thru 0x0207b9 (0xc9ca bytes) 000424 1782223477.823491 write 0x004445 thru 0x0118aa (0xd466 bytes) 000426 1782223477.825437 mapwrite 0x03ec34 thru 0x03ffff (0x13cc bytes) 000428 1782223477.922358 mapread 0x00903e thru 0x016948 (0xd90b bytes) 000430 1782223477.927561 trunc from 0x040000 to 0x03b662 (0x499d bytes) 000431 1782223477.974055 trunc from 0x03b663 to 0x023e9e (0x177c4 bytes) 000433 1782223478.005402 read 0x002af3 thru 0x0113b8 (0xe8c6 bytes) 000435 1782223478.008758 write 0x02ccef thru 0x03bf2a (0xf23c bytes) 000437 1782223478.013219 trunc from 0x03bf2b to 0x008490 (0x33a9a bytes) 000438 1782223478.042962 read 0x0034fa thru 0x0053bd (0x1ec4 bytes) 000440 1782223478.045967 read 0x0004f9 thru 0x008490 (0x7f98 bytes) 000441 1782223478.047608 write 0x02b501 thru 0x02d256 (0x1d56 bytes) 000442 1782223478.048880 write 0x00a197 thru 0x019398 (0xf202 bytes) 000443 1782223478.061397 mapwrite 0x019aad thru 0x025679 (0xbbcd bytes) 000450 1782223478.142001 mapwrite 0x023da7 thru 0x026c00 (0x2e5a bytes) 000451 1782223478.295252 mapwrite 0x0193f5 thru 0x01c690 (0x329c bytes) 000454 1782223478.397122 mapwrite 0x031fc6 thru 0x03bec3 (0x9efe bytes) 000455 1782223478.586486 mapread 0x020468 thru 0x024df7 (0x4990 bytes) 000458 1782223478.594317 read 0x0272b0 thru 0x0286a3 (0x13f4 bytes) 000459 1782223478.597510 mapwrite 0x021c4d thru 0x0232fb (0x16af bytes) 000461 1782223478.655296 mapread 0x02758a thru 0x0349b1 (0xd428 bytes) 000462 1782223478.656528 mapwrite 0x03ab4c thru 0x03c1ed (0x16a2 bytes) 000463 1782223478.817625 write 0x013351 thru 0x022686 (0xf336 bytes) 000464 1782223478.823597 mapread 0x003062 thru 0x0124a4 (0xf443 bytes) 000465 1782223478.827807 mapread 0x0388c0 thru 0x03c1ed (0x392e bytes) 000466 1782223478.830793 mapwrite 0x018eed thru 0x01d324 (0x4438 bytes) 000467 1782223478.947153 write 0x02fe52 thru 0x03e338 (0xe4e7 bytes) 000468 1782223478.958072 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e339): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e339): Mirror 3: 0x22de47e6 (stale) CRC-32 checksum value for chunk [0, 0x3e339): Mirror 1: 0x73c077d9 Mirror 4: 0x73c077d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e339: skipped 000469 1782223480.216996 mapread 0x029a78 thru 0x033a04 (0x9f8d bytes) 000470 1782223480.273524 MIRROR_RESYNC 3 mirrors 000472 1782223481.106460 MIRROR_EXTEND to 4 mirrors 000473 1782223482.119673 mapwrite 0x00bd92 thru 0x00ef18 (0x3187 bytes) 000474 1782223482.406672 trunc from 0x03e339 to 0x0234de (0x1ae5a bytes) 000477 1782223482.467309 mapread 0x01a6fa thru 0x0234de (0x8de5 bytes) 000478 1782223482.509651 write 0x034f5e thru 0x03ffff (0xb0a2 bytes) 000481 1782223482.517775 mapread 0x0219f3 thru 0x02d236 (0xb844 bytes) 000482 1782223482.553283 trunc from 0x040000 to 0x02a818 (0x157e7 bytes) 000484 1782223482.604231 read 0x004523 thru 0x00fdec (0xb8ca bytes) 000485 1782223482.628954 trunc from 0x02a819 to 0x01f3fc (0xb41c bytes) 000488 1782223482.669396 mapread 0x017c55 thru 0x01f3fc (0x77a8 bytes) 000490 1782223482.675347 mapread 0x015e30 thru 0x01f3fc (0x95cd bytes) 000494 1782223482.687477 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f3fd): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f3fd): Mirror 3: 0xc872a842 (stale) Mirror 4: 0xc872a842 (stale) Mirror 5: 0xc872a842 (stale) CRC-32 checksum value for chunk [0, 0x1f3fd): Mirror 1: 0xa986da8f Mirror 6: 0xa986da8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f3fd: skipped 000495 1782223483.773325 mapwrite 0x01ffbd thru 0x02a5f2 (0xa636 bytes) 000499 1782223484.077287 MIRROR_RESYNC 5 mirrors 000500 1782223485.465420 MIRROR_EXTEND to 6 mirrors 000503 1782223486.447909 MIRROR_RESYNC 6 mirrors 000504 1782223487.102609 read 0x00530d thru 0x00d9bb (0x86af bytes) 000507 1782223487.116600 mapread 0x01239d thru 0x01f509 (0xd16d bytes) 000508 1782223487.178683 mapwrite 0x032ca6 thru 0x03b6ca (0x8a25 bytes) 000513 1782223487.481155 mapread 0x017ba1 thru 0x01c907 (0x4d67 bytes) 000516 1782223487.525559 trunc from 0x03b6cb to 0x0148c4 (0x26e06 bytes) 000517 1782223487.589619 read 0x011318 thru 0x0148c4 (0x35ad bytes) 000518 1782223487.590504 mapread 0x0030d1 thru 0x00ad1e (0x7c4e bytes) 000519 1782223487.627440 mapread 0x005800 thru 0x01339e (0xdb9f bytes) 000520 1782223487.628838 read 0x0040f2 thru 0x0136fe (0xf60d bytes) 000521 1782223487.629481 mapwrite 0x031268 thru 0x03d4f0 (0xc289 bytes) 000522 1782223487.832583 write 0x03bac9 thru 0x03ffff (0x4537 bytes) 000523 1782223487.834266 mapwrite 0x019331 thru 0x01db35 (0x4805 bytes) 000524 1782223487.892491 write 0x01a73d thru 0x01e3be (0x3c82 bytes) 000525 1782223487.894210 mapwrite 0x03c6ce thru 0x03ffff (0x3932 bytes) 000526 1782223487.956302 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbe3421ea (stale) Mirror 4: 0xbe3421ea (stale) Mirror 5: 0xbe3421ea (stale) Mirror 6: 0xbe3421ea (stale) Mirror 7: 0xbe3421ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc7868d08 Mirror 8: 0xc7868d08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000528 1782223489.402274 read 0x03c9c7 thru 0x03ffff (0x3639 bytes) 000529 1782223489.440378 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbe3421ea (stale) Mirror 5: 0xbe3421ea (stale) Mirror 6: 0xbe3421ea (stale) Mirror 7: 0xbe3421ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xbe3421ea (stale) Mirror 5: 0xbe3421ea (stale) Mirror 6: 0xbe3421ea (stale) Mirror 7: 0xbe3421ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc7868d08 Mirror 8: 0xc7868d08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000530 1782223490.382580 mapread 0x037e40 thru 0x03ffff (0x81c0 bytes) 000531 1782223490.450502 read 0x0071ea thru 0x016364 (0xf17b bytes) 000532 1782223490.492214 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbe3421ea (stale) Mirror 6: 0xbe3421ea (stale) Mirror 7: 0xbe3421ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbe3421ea (stale) Mirror 6: 0xbe3421ea (stale) Mirror 7: 0xbe3421ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc7868d08 Mirror 8: 0xc7868d08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000533 1782223491.236777 read 0x029964 thru 0x034fde (0xb67b bytes) 000539 1782223491.262729 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbe3421ea (stale) Mirror 6: 0xbe3421ea (stale) Mirror 7: 0xbe3421ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc7868d08 Mirror 8: 0xc7868d08 Mirror 9: 0xc7868d08 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000540 1782223492.431927 trunc from 0x040000 to 0x00d762 (0x3289d bytes) 000541 1782223492.634022 MIRROR_RESYNC 6 mirrors 000542 1782223494.072099 write 0x023c8a thru 0x02fd39 (0xc0b0 bytes) 000546 1782223494.164575 trunc from 0x02fd3a to 0x032919 (0x2be0 bytes) 000547 1782223494.188547 read 0x01eec5 thru 0x02a255 (0xb391 bytes) 000548 1782223494.199952 trunc from 0x03291a to 0x028474 (0xa4a5 bytes) 000552 1782223494.233814 mapwrite 0x009531 thru 0x00ae8e (0x195e bytes) 000553 1782223494.314397 write 0x0334b8 thru 0x03dc5c (0xa7a5 bytes) 000555 1782223494.316492 mapwrite 0x037688 thru 0x03ffff (0x8978 bytes) 000556 1782223494.393439 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x374b7079 (stale) Mirror 6: 0x374b7079 (stale) Mirror 8: 0x374b7079 (stale) Mirror 9: 0x374b7079 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x374b7079 (stale) Mirror 6: 0x374b7079 (stale) Mirror 8: 0x374b7079 (stale) Mirror 9: 0x374b7079 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000558 1782223494.810295 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x374b7079 (stale) Mirror 6: 0x374b7079 (stale) Mirror 8: 0x374b7079 (stale) Mirror 9: 0x374b7079 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4f46f16b Mirror 10: 0x4f46f16b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000560 1782223496.033263 mapwrite 0x03da10 thru 0x03e129 (0x71a bytes) 000561 1782223496.246713 mapwrite 0x01d506 thru 0x01d999 (0x494 bytes) 000562 1782223496.387205 MIRROR_RESYNC 6 mirrors 000563 1782223498.064945 mapread 0x006045 thru 0x0086a1 (0x265d bytes) 000564 1782223498.106399 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000565 1782223498.638180 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000566 1782223499.086754 mapread 0x02de0f thru 0x033d21 (0x5f13 bytes) 000568 1782223499.128027 write 0x000969 thru 0x005744 (0x4ddc bytes) 000571 1782223499.226600 trunc from 0x040000 to 0x023322 (0x1ccdd bytes) 000572 1782223499.262447 mapwrite 0x009f47 thru 0x019389 (0xf443 bytes) 000573 1782223499.446595 read 0x014916 thru 0x021eca (0xd5b5 bytes) 000575 1782223499.464831 mapread 0x0027a8 thru 0x00936f (0x6bc8 bytes) 000576 1782223499.465979 mapwrite 0x016a5b thru 0x021ed6 (0xb47c bytes) 000578 1782223499.562323 trunc from 0x023323 to 0x028eae (0x5b8c bytes) 000581 1782223499.583363 mapwrite 0x03229f thru 0x0324be (0x220 bytes) 000584 1782223499.664168 read 0x0211e1 thru 0x02ad91 (0x9bb1 bytes) 000586 1782223499.676588 mapwrite 0x01e4bb thru 0x0298ed (0xb433 bytes) 000587 1782223499.739292 mapwrite 0x022178 thru 0x02d872 (0xb6fb bytes) 000588 1782223499.800889 trunc from 0x0324bf to 0x0219d5 (0x10ae9 bytes) 000589 1782223499.822217 trunc from 0x0219d6 to 0x003373 (0x1e662 bytes) 000590 1782223499.843896 write 0x02f4f6 thru 0x039b75 (0xa680 bytes) 000592 1782223499.847307 read 0x024877 thru 0x029033 (0x47bd bytes) 000593 1782223499.867791 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b76): Mirror 9: 0xdcac147a (stale) Mirror 10: 0xdcac147a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b76): Mirror 9: 0xdcac147a (stale) Mirror 10: 0xdcac147a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b76: skipped 000595 1782223500.239403 write 0x027dca thru 0x032fef (0xb226 bytes) 000596 1782223500.266351 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39b76): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b76): Mirror 9: 0xdcac147a (stale) Mirror 10: 0xdcac147a (stale) CRC-32 checksum value for chunk [0, 0x39b76): Mirror 1: 0x446fd144 Mirror 11: 0x446fd144 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b76: skipped 000597 1782223501.251755 read 0x002627 thru 0x00b6ec (0x90c6 bytes) 000598 1782223501.263053 mapwrite 0x02cabb thru 0x033152 (0x6698 bytes) 000599 1782223501.487763 mapwrite 0x01f555 thru 0x023c54 (0x4700 bytes) 000600 1782223501.585072 mapread 0x011fe1 thru 0x0193fc (0x741c bytes) 000601 1782223501.598891 read 0x0399a0 thru 0x039b75 (0x1d6 bytes) 000603 1782223501.625672 mapwrite 0x03c583 thru 0x03ffff (0x3a7d bytes) 000605 1782223501.770473 read 0x0364ae thru 0x039c69 (0x37bc bytes) 000607 1782223501.784744 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf8770d63 (stale) Mirror 11: 0x4a44417f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xf8770d63 (stale) Mirror 11: 0x4a44417f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000608 1782223502.195433 mapread 0x02e7a8 thru 0x038c69 (0xa4c2 bytes) 000609 1782223502.228759 write 0x0331fe thru 0x03df2d (0xad30 bytes) 000610 1782223502.238527 read 0x03f2ac thru 0x03ffff (0xd54 bytes) 000611 1782223502.248192 MIRROR_RESYNC 3 mirrors 000612 1782223503.062697 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000614 1782223503.520400 mapwrite 0x0375ed thru 0x03b5a4 (0x3fb8 bytes) 000616 1782223503.862168 write 0x01df5a thru 0x0283a7 (0xa44e bytes) 000617 1782223503.893576 trunc from 0x040000 to 0x03522c (0xadd3 bytes) 000620 1782223503.967477 mapread 0x0281fe thru 0x03158e (0x9391 bytes) 000621 1782223503.975498 read 0x0238a7 thru 0x0278e3 (0x403d bytes) 000624 1782223503.977432 mapread 0x006acf thru 0x0142bd (0xd7ef bytes) 000626 1782223504.001024 read 0x027c0a thru 0x02b7a2 (0x3b99 bytes) 000627 1782223504.001742 read 0x026e0a thru 0x03522c (0xe423 bytes) 000628 1782223504.006633 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3522d): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3522d): Mirror 10: 0xe801175b (stale) CRC-32 checksum value for chunk [0, 0x3522d): Mirror 1: 0xb2e3bb8c Mirror 11: 0xb2e3bb8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3522d: skipped 000629 1782223505.095180 mapwrite 0x03e837 thru 0x03ffff (0x17c9 bytes) 000631 1782223505.269532 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x469c4763 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x469c4763 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000632 1782223505.518487 write 0x022488 thru 0x02c697 (0xa210 bytes) 000633 1782223505.533358 trunc from 0x040000 to 0x024d15 (0x1b2ea bytes) 000634 1782223505.549271 mapwrite 0x0014a4 thru 0x00cd3e (0xb89b bytes) 000636 1782223505.632518 write 0x022788 thru 0x03174b (0xefc4 bytes) 000637 1782223505.636980 trunc from 0x03174c to 0x019d1c (0x17a2f bytes) 000638 1782223505.663627 MIRROR_RESYNC 2 mirrors 000640 1782223506.343312 trunc from 0x019d1d to 0x03e2c7 (0x245ab bytes) 000641 1782223506.451793 read 0x027a4b thru 0x02bce3 (0x4299 bytes) 000643 1782223506.463495 read 0x03cb17 thru 0x03e2c7 (0x17b1 bytes) 000644 1782223506.496558 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000645 1782223506.722852 write 0x011f9e thru 0x01412e (0x2191 bytes) 000646 1782223506.754510 write 0x02e136 thru 0x03e0f5 (0xffc0 bytes) 000647 1782223506.768754 read 0x03c8c6 thru 0x03e2c7 (0x1a02 bytes) 000650 1782223506.771990 write 0x03f602 thru 0x03fe36 (0x835 bytes) 000652 1782223506.773730 mapread 0x030916 thru 0x03c60f (0xbcfa bytes) 000656 1782223506.777785 mapread 0x03ede2 thru 0x03fe36 (0x1055 bytes) 000659 1782223506.779764 trunc from 0x03fe37 to 0x01b7e8 (0x2464e bytes) 000660 1782223506.821864 read 0x018c7a thru 0x01b7e8 (0x2b6f bytes) 000661 1782223506.854014 trunc from 0x01b7e9 to 0x00547f (0x16369 bytes) 000663 1782223506.895598 MIRROR_EXTEND to 2 mirrors 000665 1782223507.706403 read 0x00134e thru 0x00398a (0x263d bytes) 000667 1782223507.723213 trunc from 0x005480 to 0x0072e8 (0x1e69 bytes) 000672 1782223507.898706 mapwrite 0x035e57 thru 0x03d543 (0x76ed bytes) 000674 1782223508.142167 read 0x00963f thru 0x0124e3 (0x8ea5 bytes) 000675 1782223508.162418 mapread 0x002094 thru 0x007727 (0x5694 bytes) 000678 1782223508.172428 write 0x019cdf thru 0x01ff44 (0x6266 bytes) 000680 1782223508.178880 write 0x00046f thru 0x00b911 (0xb4a3 bytes) 000681 1782223508.181899 write 0x01cd35 thru 0x02c3df (0xf6ab bytes) 000682 1782223508.190181 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000683 1782223508.445080 mapwrite 0x03e2d3 thru 0x03f593 (0x12c1 bytes) 000684 1782223508.620336 mapwrite 0x0358d4 thru 0x0366e1 (0xe0e bytes) 000687 1782223508.678102 trunc from 0x03f594 to 0x0052f9 (0x3a29a bytes) 000688 1782223508.698964 read 0x004183 thru 0x0052f9 (0x1177 bytes) 000689 1782223508.706094 trunc from 0x0052fa to 0x029838 (0x2453f bytes) 000690 1782223508.728063 trunc from 0x029839 to 0x03b20e (0x119d6 bytes) 000692 1782223508.748788 mapwrite 0x00d5fd thru 0x012756 (0x515a bytes) 000693 1782223508.815464 mapwrite 0x01342f thru 0x01ed01 (0xb8d3 bytes) 000694 1782223508.874425 write 0x03d6f6 thru 0x03ffff (0x290a bytes) 000695 1782223508.876688 trunc from 0x040000 to 0x03d6d1 (0x292e bytes) 000696 1782223508.903837 read 0x00a712 thru 0x016de3 (0xc6d2 bytes) 000698 1782223508.905097 mapread 0x01a23d thru 0x0239c6 (0x978a bytes) 000700 1782223508.919040 mapread 0x01837f thru 0x018afd (0x77f bytes) 000701 1782223508.920398 write 0x0359d0 thru 0x03f068 (0x9699 bytes) 000703 1782223508.923434 mapread 0x00ef93 thru 0x017965 (0x89d3 bytes) 000704 1782223508.925460 write 0x03b65e thru 0x03ffff (0x49a2 bytes) 000706 1782223508.927850 mapread 0x0011c1 thru 0x00785f (0x669f bytes) 000707 1782223508.929457 write 0x023ebf thru 0x032431 (0xe573 bytes) 000713 1782223508.936151 mapread 0x02bcb7 thru 0x02e7fa (0x2b44 bytes) 000715 1782223508.938147 mapwrite 0x017358 thru 0x023e89 (0xcb32 bytes) 000719 1782223509.016606 trunc from 0x040000 to 0x026dbe (0x19241 bytes) 000721 1782223509.048626 MIRROR_EXTEND to 2 mirrors 000723 1782223510.060698 mapwrite 0x03e82b thru 0x03ffff (0x17d5 bytes) 000724 1782223510.225430 mapwrite 0x00add7 thru 0x0176b1 (0xc8db bytes) 000725 1782223510.311184 mapwrite 0x027845 thru 0x02a353 (0x2b0f bytes) 000726 1782223510.361478 mapwrite 0x035f69 thru 0x03ffff (0xa097 bytes) truncating to largest ever: 0x3fa6f 000728 1782223510.435956 trunc from 0x040000 to 0x03fa6e (0x591 bytes) 000729 1782223510.470953 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fa6f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa6f): Mirror 2: 0xa647d60c (stale) CRC-32 checksum value for chunk [0, 0x3fa6f): Mirror 1: 0x2845a8f Mirror 3: 0x2845a8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa6f: skipped 000731 1782223511.504212 mapread 0x02a1c2 thru 0x02d30b (0x314a bytes) 000733 1782223511.548632 write 0x023f32 thru 0x02d9df (0x9aae bytes) 000734 1782223511.750846 MIRROR_RESYNC 3 mirrors 000735 1782223512.709678 trunc from 0x03fa6f to 0x0294f4 (0x1657a bytes) 000737 1782223512.822300 trunc from 0x0294f5 to 0x00c3d7 (0x1d11d bytes) 000742 1782223512.858060 MIRROR_RESYNC 3 mirrors 000743 1782223513.863908 mapread 0x001e5f thru 0x00982a (0x79cc bytes) 000745 1782223513.905603 read 0x009389 thru 0x00c3d7 (0x304f bytes) 000749 1782223513.913123 mapread 0x00beab thru 0x00c3d7 (0x52d bytes) 000751 1782223513.917313 mapread 0x007004 thru 0x00c3d7 (0x53d4 bytes) 000752 1782223513.918187 mapread 0x0042e8 thru 0x00c3d7 (0x80f0 bytes) 000753 1782223513.923416 trunc from 0x00c3d8 to 0x0067c5 (0x5c12 bytes) 000755 1782223514.063810 write 0x029ecf thru 0x038162 (0xe294 bytes) 000757 1782223514.070338 mapread 0x011aa3 thru 0x01efd0 (0xd52e bytes) 000758 1782223514.117470 trunc from 0x038163 to 0x0279ca (0x10798 bytes) 000760 1782223514.163465 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x279cb): Mirror 3: 0xfbf34a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x279cb): Mirror 3: 0xfbf34a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x279cb: skipped 000762 1782223514.542822 write 0x02ef76 thru 0x0314c4 (0x254f bytes) 000763 1782223514.548556 mapwrite 0x017e8e thru 0x02508e (0xd201 bytes) 000764 1782223514.767364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x314c5): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x314c5): Mirror 3: 0xfd9ceebc (stale) CRC-32 checksum value for chunk [0, 0x314c5): Mirror 1: 0x3b242df5 Mirror 4: 0x3b242df5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x314c5: skipped 000766 1782223515.737603 trunc from 0x0314c5 to 0x016ce0 (0x1a7e4 bytes) 000767 1782223515.857384 mapwrite 0x0147df thru 0x01c265 (0x7a87 bytes) 000768 1782223516.013003 mapwrite 0x00182c thru 0x0021f6 (0x9cb bytes) 000770 1782223516.062966 read 0x008706 thru 0x01507a (0xc975 bytes) 000775 1782223516.067461 write 0x01a3ad thru 0x01ffcc (0x5c20 bytes) 000776 1782223516.069340 mapwrite 0x028ce0 thru 0x033e1b (0xb13c bytes) 000778 1782223516.200189 write 0x024ff0 thru 0x027aee (0x2aff bytes) 000780 1782223516.203244 write 0x033274 thru 0x03c5ad (0x933a bytes) 000781 1782223516.206578 mapread 0x02d396 thru 0x030fbe (0x3c29 bytes) 000782 1782223516.210278 mapread 0x021c15 thru 0x02d22e (0xb61a bytes) 000783 1782223516.211722 mapwrite 0x02b4d7 thru 0x02b68b (0x1b5 bytes) 000784 1782223516.278186 mapwrite 0x038bae thru 0x03ffff (0x7452 bytes) 000786 1782223516.370256 mapwrite 0x019c65 thru 0x01a3f7 (0x793 bytes) 000788 1782223516.452814 read 0x03c581 thru 0x03ffff (0x3a7f bytes) 000789 1782223516.459501 MIRROR_RESYNC 3 mirrors 000790 1782223517.225318 mapwrite 0x022300 thru 0x02e135 (0xbe36 bytes) 000791 1782223517.409526 mapwrite 0x01e2cf thru 0x020341 (0x2073 bytes) 000792 1782223517.513907 write 0x00da02 thru 0x0182bf (0xa8be bytes) 000794 1782223517.563597 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbdc48029 (stale) Mirror 4: 0xbdc48029 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xde25e5c7 Mirror 5: 0xde25e5c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000799 1782223518.661532 read 0x03d3d6 thru 0x03ffff (0x2c2a bytes) 000802 1782223518.674003 mapwrite 0x00b10b thru 0x01258c (0x7482 bytes) 000803 1782223518.844028 mapread 0x03ba86 thru 0x03f230 (0x37ab bytes) 000804 1782223518.859463 trunc from 0x040000 to 0x00cdc5 (0x3323a bytes) 000805 1782223518.885542 read 0x003d9d thru 0x00cdc5 (0x9029 bytes) 000806 1782223518.891837 mapread 0x008cd3 thru 0x00cdc5 (0x40f3 bytes) 000807 1782223518.892710 write 0x0295bb thru 0x033ec2 (0xa908 bytes) 000809 1782223518.901736 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33ec3): Mirror 4: 0xfef59826 (stale) Mirror 5: 0xf765199 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ec3): Mirror 4: 0xfef59826 (stale) Mirror 5: 0xf765199 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ec3: skipped 000810 1782223519.342369 mapwrite 0x00bff6 thru 0x00f51c (0x3527 bytes) 000811 1782223519.411100 read 0x00caa0 thru 0x0194bf (0xca20 bytes) 000812 1782223519.419145 trunc from 0x033ec3 to 0x010310 (0x23bb2 bytes) 000814 1782223519.441534 read 0x0057eb thru 0x009e6a (0x4680 bytes) 000815 1782223519.456721 MIRROR_RESYNC 3 mirrors 000817 1782223520.286329 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000819 1782223520.621621 trunc from 0x010311 to 0x008696 (0x7c7a bytes) 000820 1782223520.793965 MIRROR_RESYNC 2 mirrors 000821 1782223521.569463 MIRROR_RESYNC 2 mirrors 000822 1782223521.922428 mapwrite 0x002e85 thru 0x012653 (0xf7cf bytes) 000824 1782223522.203820 trunc from 0x012654 to 0x024cfc (0x126a9 bytes) 000827 1782223522.249243 mapread 0x0137be thru 0x01863c (0x4e7f bytes) 000828 1782223522.277301 mapread 0x011694 thru 0x01d18d (0xbafa bytes) 000831 1782223522.303313 trunc from 0x024cfd to 0x030f55 (0xc259 bytes) 000832 1782223522.350516 read 0x027f54 thru 0x030f55 (0x9002 bytes) 000833 1782223522.373566 mapread 0x02b58f thru 0x02ca90 (0x1502 bytes) 000834 1782223522.377406 read 0x0217bc thru 0x022fa1 (0x17e6 bytes) 000835 1782223522.379320 trunc from 0x030f56 to 0x01ba69 (0x154ec bytes) 000836 1782223522.400158 write 0x02f9b6 thru 0x03bdee (0xc439 bytes) 000838 1782223522.406163 mapwrite 0x009c49 thru 0x01883a (0xebf2 bytes) 000839 1782223522.496198 trunc from 0x03bdef to 0x00e11f (0x2dccf bytes) 000840 1782223522.529687 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000842 1782223522.784305 MIRROR_EXTEND to 2 mirrors 000843 1782223523.654401 mapwrite 0x020e6b thru 0x0213fe (0x594 bytes) 000844 1782223523.945473 read 0x0015cc thru 0x00f0b1 (0xdae6 bytes) 000846 1782223523.964738 mapwrite 0x01f515 thru 0x023fbf (0x4aab bytes) 000848 1782223524.102870 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000849 1782223524.271913 write 0x03f657 thru 0x03ffff (0x9a9 bytes) 000850 1782223524.278519 mapread 0x033339 thru 0x03afda (0x7ca2 bytes) 000851 1782223524.329184 mapwrite 0x006cb9 thru 0x0161b3 (0xf4fb bytes) 000852 1782223524.405208 mapread 0x03296b thru 0x03ffff (0xd695 bytes) 000855 1782223524.413497 mapread 0x0397ac thru 0x03ffff (0x6854 bytes) 000859 1782223524.418630 write 0x038144 thru 0x03b025 (0x2ee2 bytes) 000862 1782223524.423364 mapwrite 0x01a3fa thru 0x024aea (0xa6f1 bytes) 000863 1782223524.499272 trunc from 0x040000 to 0x01961c (0x269e3 bytes) 000864 1782223524.522129 write 0x0352da thru 0x039106 (0x3e2d bytes) 000866 1782223524.525452 trunc from 0x039107 to 0x028c93 (0x10473 bytes) 000871 1782223524.554356 mapwrite 0x028c6f thru 0x03128d (0x861f bytes) 000873 1782223524.633334 trunc from 0x03128e to 0x033a9b (0x280e bytes) 000874 1782223524.654157 mapwrite 0x03a42f thru 0x03ffff (0x5bd1 bytes) 000875 1782223524.749627 trunc from 0x040000 to 0x02ab4b (0x154b4 bytes) 000876 1782223524.771515 write 0x03c8bc thru 0x03e372 (0x1ab7 bytes) 000877 1782223524.776073 trunc from 0x03e373 to 0x0160a2 (0x282d0 bytes) 000878 1782223524.811425 write 0x03f0f1 thru 0x03ffff (0xf0f bytes) 000880 1782223524.816474 mapread 0x035724 thru 0x03ffff (0xa8dc bytes) 000881 1782223524.837339 mapwrite 0x002c4e thru 0x00de04 (0xb1b7 bytes) 000882 1782223524.910850 trunc from 0x040000 to 0x011401 (0x2ebfe bytes) 000887 1782223524.945678 trunc from 0x011402 to 0x03a4d2 (0x290d1 bytes) 000888 1782223524.964066 mapread 0x003302 thru 0x003ac0 (0x7bf bytes) 000889 1782223524.965286 write 0x01aa2f thru 0x02533a (0xa90c bytes) 000890 1782223524.978819 read 0x02f1f9 thru 0x039c3b (0xaa43 bytes) 000891 1782223525.010302 MIRROR_EXTEND to 2 mirrors 000892 1782223525.793444 write 0x0383e6 thru 0x03ffff (0x7c1a bytes) 000893 1782223525.925197 mapwrite 0x015056 thru 0x01ad9e (0x5d49 bytes) 000894 1782223526.033717 write 0x018e01 thru 0x01a366 (0x1566 bytes) 000895 1782223526.041972 mapwrite 0x03e374 thru 0x03ffff (0x1c8c bytes) 000896 1782223526.118421 mapread 0x00746a thru 0x0079c5 (0x55c bytes) 000897 1782223526.151307 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16b:0x0]: Invalid argument (22) 000898 1782223526.364555 write 0x02f941 thru 0x03a7d0 (0xae90 bytes) 000901 1782223526.401967 mapread 0x0286da thru 0x03821a (0xfb41 bytes) 000903 1782223526.425451 trunc from 0x040000 to 0x0325cd (0xda32 bytes) 000907 1782223526.448377 mapwrite 0x0237f8 thru 0x024b5c (0x1365 bytes) 000909 1782223526.518205 write 0x03e28f thru 0x03ffff (0x1d71 bytes) 000910 1782223526.519473 write 0x03581f thru 0x0378ed (0x20cf bytes) 000912 1782223526.536540 mapwrite 0x033a87 thru 0x03ce1c (0x9396 bytes) 000915 1782223526.647875 mapread 0x022ffc thru 0x02bcc4 (0x8cc9 bytes) 000916 1782223526.665025 mapread 0x03a7bf thru 0x03ffff (0x5841 bytes) 000918 1782223526.671650 mapread 0x01a97f thru 0x02123f (0x68c1 bytes) 000921 1782223526.690085 mapread 0x02a466 thru 0x031de6 (0x7981 bytes) 000922 1782223526.691418 mapwrite 0x001c3c thru 0x0069de (0x4da3 bytes) 000924 1782223526.806109 read 0x03e175 thru 0x03ffff (0x1e8b bytes) 000926 1782223526.810167 trunc from 0x040000 to 0x034e97 (0xb168 bytes) 000927 1782223526.844283 read 0x026183 thru 0x027baa (0x1a28 bytes) 000928 1782223526.845130 trunc from 0x034e98 to 0x01d451 (0x17a46 bytes) 000930 1782223526.902150 read 0x00355c thru 0x005311 (0x1db6 bytes) 000931 1782223526.904102 write 0x000380 thru 0x00170b (0x138c bytes) 000934 1782223526.929504 trunc from 0x01d452 to 0x005b37 (0x1791a bytes) 000939 1782223526.960125 read 0x00408e thru 0x005b37 (0x1aaa bytes) 000940 1782223526.960903 mapread 0x0059be thru 0x005b37 (0x17a bytes) 000941 1782223526.962554 trunc from 0x005b38 to 0x0337b2 (0x2dc7b bytes) 000944 1782223527.010046 MIRROR_EXTEND to 2 mirrors 000945 1782223527.657623 write 0x025082 thru 0x02e686 (0x9605 bytes) 000946 1782223527.748321 write 0x02c5c6 thru 0x036095 (0x9ad0 bytes) 000947 1782223527.750700 mapwrite 0x004f2c thru 0x011119 (0xc1ee bytes) 000953 1782223527.923294 read 0x00bd84 thru 0x011a8c (0x5d09 bytes) 000954 1782223527.924227 read 0x02e3e1 thru 0x02febf (0x1adf bytes) 000956 1782223527.926303 trunc from 0x036096 to 0x014448 (0x21c4d bytes) 000957 1782223527.963129 read 0x00a264 thru 0x00eda9 (0x4b46 bytes) 000958 1782223527.964291 read 0x000026 thru 0x006139 (0x6114 bytes) 000959 1782223527.977575 read 0x012d56 thru 0x014448 (0x16f3 bytes) 000962 1782223527.983820 trunc from 0x014449 to 0x03e14f (0x29d07 bytes) 000964 1782223528.004186 trunc from 0x03e150 to 0x034d90 (0x93bf bytes) 000971 1782223528.053525 MIRROR_RESYNC 2 mirrors 000973 1782223528.948691 mapwrite 0x024996 thru 0x032b38 (0xe1a3 bytes) 000974 1782223529.219306 mapwrite 0x01bdc6 thru 0x02822a (0xc465 bytes) 000977 1782223529.345626 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34d91): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d91): Mirror 2: 0x97f290e9 (stale) CRC-32 checksum value for chunk [0, 0x34d91): Mirror 1: 0x3fe53c3a Mirror 3: 0x3fe53c3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d91: skipped 000981 1782223530.075675 mapwrite 0x03f1f4 thru 0x03ffff (0xe0c bytes) 000988 1782223530.336498 MIRROR_RESYNC 3 mirrors 000989 1782223531.098567 mapwrite 0x0053f6 thru 0x007b34 (0x273f bytes) 000991 1782223531.251513 MIRROR_RESYNC 3 mirrors 000992 1782223532.060159 read 0x03c0e9 thru 0x03ffff (0x3f17 bytes) 000993 1782223532.069175 write 0x02474f thru 0x028d7e (0x4630 bytes) 000994 1782223532.161554 MIRROR_RESYNC 3 mirrors 000995 1782223533.133068 read 0x03138c thru 0x037103 (0x5d78 bytes) 000996 1782223533.147927 trunc from 0x040000 to 0x012048 (0x2dfb7 bytes) 000999 1782223533.250344 read 0x00809a thru 0x009ec0 (0x1e27 bytes) All operations completed A-OK! PASS 70a (82s) == sanity-flr test 71: check mirror extend parent fid ==== 10:05:38 (1782223538) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 10:05:41 (1782223541) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.236615 s, 8.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186821 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 10:05:50 (1782223550) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6702816 .. Extending file size to 6300672 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7681120 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9111872 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8199840 .. Extending file size to 9150016 .. Extending file size to 9325760 .. Extending file size to 8299680 .. Extending file size to 8419072 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9199008 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56767 56768 56769 56772 56775 done 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:07:13 (1782223633) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7682144 .. Extending file size to 846816 .. Extending file size to 3295168 .. Extending file size to 8833312 .. Extending file size to 7721088 .. Extending file size to 8930560 .. Extending file size to 8509152 .. Extending file size to 9169280 .. Extending file size to 9213248 .. Extending file size to 9397472 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58182 58183 58185 58187 58189 failed 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) 192.168.203.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:08:34 (1782223714) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152265 s, 13.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.316908 s, 6.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7fbfd87f (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:08:59 (1782223739) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0751103 s, 27.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 10:09:06 (1782223746) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ae:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:09:14 (1782223754) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178204 s, 11.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:09:22 (1782223762) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192907 s, 10.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:09:30 (1782223770) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232697 s, 9.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:10:02 (1782223802) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940916 s, 11.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:10:58 (1782223858) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220199 s, 9.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:12:00 (1782223920) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg337-server mds-ost sync done. Creating new pool test_204f oleg337-server: Pool lustre.test_204f created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.444923 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg337-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:13:23 (1782224003) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:13:30 (1782224010) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:13:37 (1782224017) oleg337-server: Pool lustre.test_206 created Adding targets to pool oleg337-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg337-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg337-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg337-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:14:03 (1782224043) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516578 s, 20.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060489 s, 17.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x5eb61727 Mirror 3: 0x5eb61727 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:14:13 (1782224053) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:14:16 (1782224056) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:14:19 (1782224059) Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0970149 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x15b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2110000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2110000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2110000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c90000 Read this (4 bytes): BBBB 192.168.203.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg337-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:14:55 (1782224095) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.018309 s, 447 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:15:03 (1782224103) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801567 s, 13.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d3 lmm_fid: [0x200000404:0x1d3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d3 lmm_fid: [0x200000404:0x1d3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834387 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-23 10:15:06.000000000 -0400 Modify: 2026-06-23 10:15:06.000000000 -0400 Change: 2026-06-23 10:15:08.000000000 -0400 Birth: 2026-06-23 10:15:06.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:15:13 (1782224113) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:15:17 (1782224117) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.8129 s, 50.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0538913 s, 76.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 23 10:15 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 23 10:15 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:15:25 (1782224125) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:15:33 (1782224133) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1840 sec =========== 10:15:43 (1782224143) === sanity-flr: start cleanup 10:15:45 (1782224145) === === sanity-flr: finish cleanup 10:15:51 (1782224151) ===