-----============= acceptance-small: sanity-flr ============----- Wed Jul 8 09:22:45 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg245-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:22:55 (1783516975) === oleg245-client.virtnet: executing check_config_client /mnt/lustre oleg245-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg245-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90e7c7c19800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90e7c7c19800.idle_timeout=debug disable quota as required oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:23:13 (1783516993) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7589] == sanity-flr test 0a: lfs mirror create with -N option == 09:23:14 (1783516994) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:23:23 (1783517003) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:23:24 (1783517004) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:23:26 (1783517006) PASS 0d (13s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:23:39 (1783517019) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:23:41 (1783517021) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:23:43 (1783517023) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff90e7c7c19800.stats=clear osc.lustre-OST0001-osc-ffff90e7c7c19800.stats=clear PASS 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:23:47 (1783517027) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:23:52 (1783517032) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:23:56 (1783517036) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:23:58 (1783517038) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:24:03 (1783517043) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 09:24:14 (1783517054) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:24:21 (1783517061) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:24:24 (1783517064) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:24:31 (1783517071) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:24:42 (1783517082) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:24:48 (1783517088) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.346 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110897 s, 9.5 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:24:58 (1783517098) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0810515 s, 25.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:25:05 (1783517105) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:25:14 (1783517114) Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg245-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (110s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:27:04 (1783517224) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.147618 s, 35.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (22s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:27:26 (1783517246) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (38s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:28:04 (1783517284) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.40844 s, 21.8 MB/s -rw-r--r-- 1 root root 52428800 Jul 8 09:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.483s user 0m0.001s sys 0m0.141s Stopping /mnt/lustre-ost1 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e7c7c19800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.570s user 0m0.001s sys 0m0.150s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg245-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg245-server oleg245-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.532s user 0m0.001s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg245-server: oleg245-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg245-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e7c7c19800.ost_server_uuid in FULL state after 0 sec PASS 33b (42s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:28:46 (1783517326) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:28:48 (1783517328) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:28:49 (1783517329) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:28:51 (1783517331) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 09:28:54 (1783517334) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.250923 s, 50.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:29:00 (1783517340) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116551 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149393 s, 70.2 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:29:03 (1783517343) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:29:07 (1783517347) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00070092 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00182808 s, 560 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0014109 s, 726 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000700926 s, 1.5 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 09:29:11 (1783517351) create test files with size 15738091 .. b06a1345d89c1969c86543d0f09d85b8 - 8dcca3f8da9140916ec66b723b88ec02 - 12635c955ce6954dfb11474679489ed2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 09:29:19 (1783517359) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:29:25 (1783517365) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:29:27 (1783517367) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430638 s, 24.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111571 s, 94.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 09:29:30 (1783517370) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0415947 s, 50.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.103221 s, 40.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0497803 s, 42.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 09:29:35 (1783517375) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:29:36 (1783517376) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:29:38 (1783517378) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147579 s, 71.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 09:29:41 (1783517381) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:29:42 (1783517382) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:29:46 (1783517386) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.100126 s, 105 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:30:03 (1783517403) llite.lustre-ffff90e7d0cfc800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0755694 s, 139 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg245-client.virtnet /mnt/lustre (opts:) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff90e7c2f0f000.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:30:07 (1783517407) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:30:18 (1783517418) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (11s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:30:29 (1783517429) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (0s) == sanity-flr test 48: Verify snapshot mirror ============ 09:30:29 (1783517429) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.07329 s, 42.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0614137 s, 51.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xcbd66922 (nosync,stale) Mirror 2: 0xcbd66922 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8b3db358e2b76154ed9ecfb3e82b6528 - mirror 1 checksum: da71bcf51470fcbb288408cf2cd98bd0 - mirror 2 checksum: 8b3db358e2b76154ed9ecfb3e82b6528 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-08T09:30:29-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-08T09:30:29-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8b3db358e2b76154ed9ecfb3e82b6528 - mirror 1 checksum: da71bcf51470fcbb288408cf2cd98bd0 - mirror 2 checksum: da71bcf51470fcbb288408cf2cd98bd0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-08T09:30:29-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-08T09:30:29-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:30:33 (1783517433) Failing mds1 on oleg245-server Stopping /mnt/lustre-mds1 (opts:) on oleg245-server 09:30:35 (1783517435) shut down facet: mds1 facet_host: oleg245-server facet_failover_host: oleg245-server Failover mds1 to oleg245-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 09:30:49 (1783517449) targets are mounted 09:30:49 (1783517449) facet_failover done oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103118 s, 102 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:30:55 (1783517455) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:30:56 (1783517456) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:30:57 (1783517457) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00251444 s, 6.5 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:30:59 (1783517459) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:31:00 (1783517460) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:31:01 (1783517461) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:31:07 (1783517467) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (18s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:31:25 (1783517485) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:31:43 (1783517503) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:32:00 (1783517520) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 09:32:02 (1783517522) mirror create pid 53680 mirror split pid 53682 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53680 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53682 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 09:33:04 (1783517584) Seed set to 7585 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1783517585.316267 mapwrite 0x0305c0 thru 0x03838e (0x7dcf bytes) truncating to largest ever: 0x213ac 000002 1783517585.339261 trunc from 0x03838f to 0x0213ab (0x16fe3 bytes) 000009 1783517585.345854 read 0x004be5 thru 0x007812 (0x2c2e bytes) 000012 1783517585.350544 MIRROR_EXTEND to 2 mirrors 000014 1783517585.508714 mapwrite 0x025bbb thru 0x026c52 (0x1098 bytes) 000015 1783517585.587557 read 0x01ce68 thru 0x026c52 (0x9deb bytes) 000016 1783517585.592734 read 0x005ba6 thru 0x00f18c (0x95e7 bytes) 000017 1783517585.604085 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26c53): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c53): Mirror 2: 0x4af9a234 (stale) CRC-32 checksum value for chunk [0, 0x26c53): Mirror 1: 0x2c0784ef Mirror 3: 0x2c0784ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c53: skipped 000018 1783517585.818495 mapread 0x00af23 thru 0x00e092 (0x3170 bytes) 000020 1783517585.826428 mapread 0x013ac7 thru 0x01d7c9 (0x9d03 bytes) 000021 1783517585.832658 MIRROR_RESYNC 3 mirrors 000022 1783517585.999527 MIRROR_EXTEND to 4 mirrors 000023 1783517586.165486 trunc from 0x026c53 to 0x001868 (0x253ea bytes) 000026 1783517586.196261 mapread 0x0015d6 thru 0x001868 (0x293 bytes) truncating to largest ever: 0x23c76 000031 1783517586.200513 trunc from 0x001869 to 0x023c75 (0x2240d bytes) 000032 1783517586.209313 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23c76): Mirror 2: 0x8d4c96e6 (stale) Mirror 4: 0x8d4c96e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23c76): Mirror 2: 0x8d4c96e6 (stale) Mirror 4: 0x8d4c96e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23c76: skipped 000033 1783517586.294112 write 0x008546 thru 0x01745a (0xef15 bytes) 000035 1783517586.298195 mapwrite 0x005729 thru 0x00d500 (0x7dd8 bytes) 000036 1783517586.316532 trunc from 0x023c76 to 0x01fa1d (0x4258 bytes) 000039 1783517586.322310 mapwrite 0x01fe7a thru 0x02bca1 (0xbe28 bytes) 000040 1783517586.345770 mapread 0x00c3f2 thru 0x0186ff (0xc30e bytes) 000041 1783517586.348784 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bca2): Mirror 4: 0xcce1a131 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bca2): Mirror 4: 0xcce1a131 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bca2: skipped 000043 1783517586.421472 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bca2): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bca2): Mirror 4: 0xcce1a131 (stale) CRC-32 checksum value for chunk [0, 0x2bca2): Mirror 1: 0xd6942f6a Mirror 5: 0xd6942f6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bca2: skipped 000044 1783517586.632484 mapwrite 0x03170b thru 0x0335bb (0x1eb1 bytes) 000047 1783517586.669391 trunc from 0x0335bc to 0x0052b4 (0x2e307 bytes) 000048 1783517586.674862 trunc from 0x0052b5 to 0x015809 (0x10555 bytes) 000052 1783517586.680252 trunc from 0x01580a to 0x01410e (0x16fb bytes) 000053 1783517586.684502 mapwrite 0x00e9c1 thru 0x01473f (0x5d7f bytes) 000054 1783517586.705384 write 0x02b635 thru 0x032de4 (0x77b0 bytes) truncating to largest ever: 0x25065 000055 1783517586.706344 trunc from 0x032de5 to 0x025064 (0xdd80 bytes) 000056 1783517586.712376 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25065): Mirror 5: 0x93cc69d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25065): Mirror 5: 0x93cc69d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25065: skipped 000057 1783517586.791707 write 0x02679d thru 0x02be03 (0x5667 bytes) 000058 1783517586.794366 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) 000060 1783517586.828505 trunc from 0x02be04 to 0x01a63a (0x117c9 bytes) 000062 1783517586.833055 mapread 0x001a6d thru 0x00f2d5 (0xd869 bytes) 000063 1783517586.837971 write 0x02c670 thru 0x02da36 (0x13c7 bytes) 000064 1783517586.838651 trunc from 0x02da37 to 0x003b32 (0x29f04 bytes) truncating to largest ever: 0x3f070 000068 1783517586.844653 trunc from 0x003b33 to 0x03f06f (0x3b53d bytes) 000069 1783517586.848745 write 0x016080 thru 0x018c2f (0x2bb0 bytes) 000070 1783517586.851609 read 0x03d35d thru 0x03f06f (0x1d13 bytes) 000073 1783517586.854288 read 0x00bb6c thru 0x01018a (0x461f bytes) 000076 1783517586.858378 mapwrite 0x02090c thru 0x02dabe (0xd1b3 bytes) 000077 1783517586.872954 write 0x001e08 thru 0x0091c2 (0x73bb bytes) 000081 1783517586.875406 read 0x025b60 thru 0x02f7f1 (0x9c92 bytes) 000084 1783517586.877544 write 0x00953f thru 0x011021 (0x7ae3 bytes) 000085 1783517586.878709 trunc from 0x03f070 to 0x004c75 (0x3a3fa bytes) 000086 1783517586.884196 read 0x002564 thru 0x004c75 (0x2712 bytes) 000087 1783517586.884896 read 0x00445d thru 0x004c75 (0x819 bytes) 000088 1783517586.885495 mapwrite 0x017657 thru 0x018fd8 (0x1982 bytes) 000090 1783517586.904972 trunc from 0x018fd9 to 0x012f13 (0x60c5 bytes) 000091 1783517586.908759 mapwrite 0x035ebe thru 0x03ffff (0xa142 bytes) 000092 1783517586.927834 mapread 0x009aa3 thru 0x01934c (0xf8aa bytes) 000093 1783517586.935325 trunc from 0x040000 to 0x0175da (0x28a25 bytes) 000097 1783517586.940778 trunc from 0x0175db to 0x005ceb (0x118ef bytes) 000100 1783517586.945725 mapwrite 0x03209a thru 0x039abe (0x7a25 bytes) 000101 1783517586.970652 mapread 0x022b73 thru 0x02638b (0x3819 bytes) 000102 1783517586.977338 read 0x00af88 thru 0x016b5c (0xbbd5 bytes) 000104 1783517586.980713 mapwrite 0x004e0b thru 0x013e5d (0xf053 bytes) 000105 1783517586.994963 mapwrite 0x00a69d thru 0x01838c (0xdcf0 bytes) 000106 1783517587.010651 mapread 0x00e190 thru 0x017e3b (0x9cac bytes) 000109 1783517587.012510 mapread 0x0119af thru 0x020112 (0xe764 bytes) 000111 1783517587.013760 mapread 0x00c288 thru 0x00f0f1 (0x2e6a bytes) 000112 1783517587.014421 mapwrite 0x0131b7 thru 0x018a38 (0x5882 bytes) 000113 1783517587.027291 write 0x009c17 thru 0x012e68 (0x9252 bytes) 000116 1783517587.028905 mapwrite 0x00d278 thru 0x00f87c (0x2605 bytes) 000118 1783517587.041332 mapwrite 0x023dbc thru 0x029ab5 (0x5cfa bytes) 000119 1783517587.053855 read 0x004ab4 thru 0x014a25 (0xff72 bytes) 000122 1783517587.055619 trunc from 0x039abf to 0x011cbb (0x27e03 bytes) 000127 1783517587.062554 mapread 0x00a376 thru 0x011cbb (0x7946 bytes) 000129 1783517587.065309 MIRROR_EXTEND to 2 mirrors 000130 1783517587.203167 MIRROR_RESYNC 2 mirrors 000131 1783517587.298858 write 0x027bec thru 0x02bd4c (0x4161 bytes) 000132 1783517587.316691 write 0x027caa thru 0x033664 (0xb9bb bytes) 000135 1783517587.318736 mapread 0x0207a1 thru 0x0280a8 (0x7908 bytes) 000136 1783517587.324388 mapread 0x003aa0 thru 0x00ee81 (0xb3e2 bytes) 000138 1783517587.329944 MIRROR_RESYNC 2 mirrors 000139 1783517587.485326 write 0x0220f2 thru 0x030cc2 (0xebd1 bytes) 000141 1783517587.504129 trunc from 0x033665 to 0x0213cd (0x12297 bytes) 000144 1783517587.513169 write 0x0131ca thru 0x01931c (0x6153 bytes) 000145 1783517587.515841 mapread 0x0032c5 thru 0x00f4bf (0xc1fb bytes) 000146 1783517587.520368 trunc from 0x0213ce to 0x028c3a (0x786d bytes) 000147 1783517587.525052 MIRROR_RESYNC 2 mirrors 000148 1783517587.681917 trunc from 0x028c3b to 0x0156fc (0x1353e bytes) 000153 1783517587.704366 mapwrite 0x003ce3 thru 0x00c850 (0x8b6e bytes) 000154 1783517587.723421 mapread 0x00b9a1 thru 0x0156fc (0x9d5c bytes) 000155 1783517587.726628 read 0x0043e3 thru 0x0056b8 (0x12d6 bytes) 000156 1783517587.727249 mapwrite 0x02b31b thru 0x033fb8 (0x8c9e bytes) 000158 1783517587.749287 read 0x0261c9 thru 0x033db3 (0xdbeb bytes) 000160 1783517587.750039 mapwrite 0x0200f6 thru 0x0229be (0x28c9 bytes) 000161 1783517587.764262 read 0x0253ed thru 0x033fb8 (0xebcc bytes) 000162 1783517587.764970 trunc from 0x033fb9 to 0x003cc7 (0x302f1 bytes) 000163 1783517587.769934 mapwrite 0x016f3c thru 0x01e881 (0x7946 bytes) 000164 1783517587.789046 mapread 0x0191a3 thru 0x01e881 (0x56df bytes) 000167 1783517587.791178 trunc from 0x01e882 to 0x0378a0 (0x1901f bytes) 000168 1783517587.795852 mapread 0x0175bd thru 0x01ccf1 (0x5735 bytes) 000169 1783517587.796851 write 0x007fea thru 0x00cd6a (0x4d81 bytes) 000170 1783517587.797903 write 0x00d2c6 thru 0x013dc6 (0x6b01 bytes) 000171 1783517587.798942 trunc from 0x0378a1 to 0x0019aa (0x35ef6 bytes) 000173 1783517587.804086 mapread 0x00045f thru 0x0019aa (0x154c bytes) 000175 1783517587.805541 write 0x01cd8c thru 0x028874 (0xbae9 bytes) 000177 1783517587.808966 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28875): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28875): Mirror 2: 0xf0a13ae1 (stale) CRC-32 checksum value for chunk [0, 0x28875): Mirror 1: 0x12f67ade Mirror 3: 0x12f67ade lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28875: skipped 000178 1783517588.006435 write 0x0223de thru 0x0232ff (0xf22 bytes) 000180 1783517588.024752 mapread 0x021fe5 thru 0x028874 (0x6890 bytes) 000181 1783517588.029429 trunc from 0x028875 to 0x03d7de (0x14f6a bytes) 000183 1783517588.039956 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d7df): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7df): Mirror 2: 0x12c499dc (stale) Mirror 3: 0x7d468258 (stale) CRC-32 checksum value for chunk [0, 0x3d7df): Mirror 1: 0xa5fdb00a Mirror 4: 0xa5fdb00a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7df: skipped 000184 1783517588.252275 read 0x007190 thru 0x007e35 (0xca6 bytes) 000186 1783517588.261005 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d7df): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7df): Mirror 2: 0x12c499dc (stale) Mirror 3: 0x7d468258 (stale) CRC-32 checksum value for chunk [0, 0x3d7df): Mirror 1: 0xa5fdb00a Mirror 4: 0xa5fdb00a Mirror 5: 0xa5fdb00a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7df: skipped 000187 1783517588.480182 write 0x00b6cd thru 0x00e46b (0x2d9f bytes) 000189 1783517588.498569 mapread 0x03cdc4 thru 0x03d7de (0xa1b bytes) 000191 1783517588.503168 MIRROR_RESYNC 5 mirrors 000192 1783517588.717600 trunc from 0x03d7df to 0x029ff8 (0x137e6 bytes) 000196 1783517588.741156 mapwrite 0x03eba2 thru 0x03ffff (0x145e bytes) 000201 1783517588.762944 MIRROR_RESYNC 5 mirrors 000202 1783517588.977171 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) 000204 1783517589.062317 write 0x036423 thru 0x03ffff (0x9bdd bytes) 000205 1783517589.081929 write 0x03d8cb thru 0x03ffff (0x2735 bytes) 000206 1783517589.084558 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x63b48c23 (stale) Mirror 5: 0x63b48c23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x63b48c23 (stale) Mirror 5: 0x63b48c23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000207 1783517589.169685 mapwrite 0x01ae1b thru 0x023fe3 (0x91c9 bytes) 000208 1783517589.188170 read 0x0088b6 thru 0x00b019 (0x2764 bytes) 000209 1783517589.190524 read 0x037570 thru 0x03ffff (0x8a90 bytes) 000210 1783517589.193399 mapwrite 0x02b73f thru 0x035941 (0xa203 bytes) 000211 1783517589.206069 trunc from 0x040000 to 0x000ce6 (0x3f319 bytes) 000213 1783517589.217789 read 0x000149 thru 0x000ce6 (0xb9e bytes) 000215 1783517589.219139 write 0x028fb7 thru 0x03078e (0x77d8 bytes) 000216 1783517589.220267 write 0x0084aa thru 0x01465a (0xc1b1 bytes) 000218 1783517589.224875 write 0x020304 thru 0x02ee67 (0xeb64 bytes) 000220 1783517589.226847 mapread 0x0016cb thru 0x010f6b (0xf8a1 bytes) 000221 1783517589.228123 trunc from 0x03078f to 0x02f17e (0x1610 bytes) 000222 1783517589.233414 mapread 0x02e934 thru 0x02f17e (0x84b bytes) 000225 1783517589.235480 mapread 0x0147ba thru 0x02101c (0xc863 bytes) 000226 1783517589.236427 mapread 0x0288b0 thru 0x028932 (0x83 bytes) 000228 1783517589.237377 read 0x013a4a thru 0x016690 (0x2c47 bytes) 000229 1783517589.237915 mapwrite 0x00e733 thru 0x01d3cb (0xec99 bytes) 000231 1783517589.256812 mapread 0x00ef86 thru 0x019321 (0xa39c bytes) 000232 1783517589.258025 mapread 0x00b11e thru 0x01382a (0x870d bytes) 000233 1783517589.259114 read 0x007bf3 thru 0x00ab93 (0x2fa1 bytes) 000236 1783517589.260933 mapwrite 0x01021f thru 0x01ad29 (0xab0b bytes) 000237 1783517589.274851 mapwrite 0x00b3fd thru 0x013338 (0x7f3c bytes) 000239 1783517589.288977 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f17f): Mirror 2: 0xbb93635c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f17f): Mirror 2: 0xbb93635c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f17f: skipped 000240 1783517589.363847 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f17f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f17f): Mirror 2: 0xbb93635c (stale) CRC-32 checksum value for chunk [0, 0x2f17f): Mirror 1: 0xf79c5cd1 Mirror 3: 0xf79c5cd1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f17f: skipped 000245 1783517589.568137 MIRROR_RESYNC 3 mirrors 000246 1783517589.767373 mapread 0x0181c1 thru 0x019ec9 (0x1d09 bytes) 000247 1783517589.771776 mapread 0x002024 thru 0x00dc0d (0xbbea bytes) 000248 1783517589.778770 mapwrite 0x036d55 thru 0x03739d (0x649 bytes) 000249 1783517589.841035 mapwrite 0x0096b7 thru 0x018919 (0xf263 bytes) 000250 1783517589.863318 trunc from 0x03739e to 0x03a8f9 (0x355c bytes) 000252 1783517589.870971 mapread 0x038ece thru 0x03a8f9 (0x1a2c bytes) 000253 1783517589.873783 write 0x034822 thru 0x03eb79 (0xa358 bytes) 000254 1783517589.876735 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb7a): Mirror 3: 0xbf1b3e15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb7a): Mirror 3: 0xbf1b3e15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb7a: skipped 000255 1783517589.969080 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3eb7a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb7a): Mirror 3: 0xbf1b3e15 (stale) CRC-32 checksum value for chunk [0, 0x3eb7a): Mirror 1: 0x903c27fd Mirror 4: 0x903c27fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb7a: skipped 000256 1783517590.204403 mapwrite 0x027f33 thru 0x0342fa (0xc3c8 bytes) 000257 1783517590.259098 write 0x024d4b thru 0x027d1e (0x2fd4 bytes) 000258 1783517590.259822 read 0x01dc1a thru 0x021dda (0x41c1 bytes) 000260 1783517590.262400 read 0x02efb4 thru 0x03992a (0xa977 bytes) 000261 1783517590.263088 trunc from 0x03eb7a to 0x00805b (0x36b1e bytes) 000267 1783517590.271826 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x805c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x805c): Mirror 3: 0xa12e422e (stale) Mirror 4: 0xa12e422e (stale) CRC-32 checksum value for chunk [0, 0x805c): Mirror 1: 0xa12e422e Mirror 5: 0xa12e422e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x805c: skipped 000268 1783517590.455337 mapwrite 0x01f26e thru 0x02c8f8 (0xd68b bytes) 000269 1783517590.497517 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c8f9): Mirror 4: 0x3fd7b9b0 (stale) Mirror 5: 0x73548a42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c8f9): Mirror 4: 0x3fd7b9b0 (stale) Mirror 5: 0x73548a42 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c8f9: skipped 000271 1783517590.579881 read 0x02bc5f thru 0x02c8f8 (0xc9a bytes) 000272 1783517590.581697 mapread 0x025669 thru 0x02c8f8 (0x7290 bytes) 000273 1783517590.586370 write 0x01bc88 thru 0x0262b5 (0xa62e bytes) 000274 1783517590.588374 read 0x00721e thru 0x007e87 (0xc6a bytes) 000278 1783517590.592380 mapread 0x01980b thru 0x01c240 (0x2a36 bytes) 000280 1783517590.593494 trunc from 0x02c8f9 to 0x007b97 (0x24d61 bytes) 000281 1783517590.598897 mapread 0x00217a thru 0x004ad4 (0x295b bytes) 000282 1783517590.599516 mapwrite 0x009f0e thru 0x01099d (0x6a90 bytes) 000283 1783517590.626929 trunc from 0x01099e to 0x031ee4 (0x21547 bytes) 000284 1783517590.631060 read 0x031533 thru 0x031ee4 (0x9b2 bytes) 000286 1783517590.634783 mapwrite 0x037944 thru 0x03ffff (0x86bc bytes) 000288 1783517590.654789 write 0x011bc6 thru 0x013717 (0x1b52 bytes) 000289 1783517590.655729 trunc from 0x040000 to 0x019c36 (0x263c9 bytes) 000292 1783517590.662843 MIRROR_RESYNC 3 mirrors 000293 1783517590.833446 mapwrite 0x00e30c thru 0x0148ee (0x65e3 bytes) 000294 1783517590.881257 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19c37): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19c37): Mirror 4: 0xbb20a5db (stale) Mirror 5: 0xbb20a5db (stale) CRC-32 checksum value for chunk [0, 0x19c37): Mirror 1: 0xfefbbae0 Mirror 6: 0xfefbbae0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19c37: skipped 000295 1783517591.074119 trunc from 0x019c37 to 0x00c224 (0xda12 bytes) 000297 1783517591.096416 trunc from 0x00c225 to 0x00c35b (0x137 bytes) 000298 1783517591.101524 trunc from 0x00c35c to 0x029eab (0x1db50 bytes) 000299 1783517591.105869 read 0x01691c thru 0x018fea (0x26cf bytes) 000301 1783517591.108236 mapwrite 0x03ccf1 thru 0x03ffff (0x330f bytes) 000302 1783517591.126865 trunc from 0x040000 to 0x007b93 (0x3846c bytes) 000303 1783517591.131117 write 0x02774b thru 0x0298e8 (0x219e bytes) 000305 1783517591.132198 write 0x019bcf thru 0x01a282 (0x6b4 bytes) 000306 1783517591.134756 mapwrite 0x00372f thru 0x0108e1 (0xd1b3 bytes) 000307 1783517591.150336 write 0x01f1b3 thru 0x02de88 (0xecd6 bytes) 000309 1783517591.153558 mapwrite 0x02b163 thru 0x0372eb (0xc189 bytes) 000312 1783517591.175411 trunc from 0x0372ec to 0x0264f2 (0x10df9 bytes) 000314 1783517591.179916 mapwrite 0x025d7a thru 0x02cb50 (0x6dd7 bytes) 000316 1783517591.197275 mapwrite 0x01e64d thru 0x01ee0b (0x7bf bytes) 000318 1783517591.210619 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cb51): Mirror 4: 0x60c20706 (stale) Mirror 6: 0xf31468fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb51): Mirror 4: 0x60c20706 (stale) Mirror 6: 0xf31468fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb51: skipped 000320 1783517591.289896 write 0x00f2a2 thru 0x01a615 (0xb374 bytes) 000323 1783517591.294782 read 0x019e60 thru 0x01c917 (0x2ab8 bytes) 000325 1783517591.297655 mapwrite 0x009258 thru 0x014178 (0xaf21 bytes) 000326 1783517591.315103 write 0x001f2c thru 0x00842e (0x6503 bytes) 000327 1783517591.317544 trunc from 0x02cb51 to 0x03ec3e (0x120ee bytes) 000328 1783517591.322304 mapread 0x01f927 thru 0x0211f9 (0x18d3 bytes) 000332 1783517591.327156 write 0x03188d thru 0x036583 (0x4cf7 bytes) 000334 1783517591.330960 write 0x025f5f thru 0x033dda (0xde7c bytes) 000335 1783517591.332123 read 0x00ddd9 thru 0x0173b2 (0x95da bytes) 000336 1783517591.332545 trunc from 0x03ec3f to 0x030eb7 (0xdd87 bytes) 000337 1783517591.336931 trunc from 0x030eb8 to 0x025f5d (0xaf5a bytes) 000339 1783517591.341749 mapwrite 0x012e25 thru 0x01c1bd (0x9399 bytes) 000341 1783517591.355671 read 0x0120e5 thru 0x01d1a1 (0xb0bd bytes) 000342 1783517591.357757 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25f5e): Mirror 4: 0xca2cb73d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25f5e): Mirror 4: 0xca2cb73d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25f5e: skipped 000343 1783517591.429189 mapwrite 0x019c04 thru 0x0220a2 (0x849f bytes) 000345 1783517591.449438 read 0x013d17 thru 0x01fb42 (0xbe2c bytes) 000346 1783517591.452090 write 0x037217 thru 0x0396ad (0x2497 bytes) 000349 1783517591.454328 trunc from 0x0396ae to 0x030da7 (0x8906 bytes) 000350 1783517591.460232 MIRROR_RESYNC 2 mirrors 000351 1783517591.616051 read 0x00c15f thru 0x017cde (0xbb80 bytes) 000352 1783517591.619928 MIRROR_RESYNC 2 mirrors 000353 1783517591.723682 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) 000355 1783517591.759874 trunc from 0x030da8 to 0x021056 (0xfd51 bytes) 000356 1783517591.769762 write 0x03191d thru 0x03462a (0x2d0e bytes) 000357 1783517591.770491 trunc from 0x03462b to 0x03104e (0x35dc bytes) 000360 1783517591.776392 MIRROR_EXTEND to 2 mirrors 000364 1783517591.930934 read 0x00b216 thru 0x00ebd9 (0x39c4 bytes) 000365 1783517591.932968 trunc from 0x03104f to 0x01cbe7 (0x14467 bytes) 000366 1783517591.961405 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) 000368 1783517592.002573 mapwrite 0x008bca thru 0x0111a6 (0x85dd bytes) 000370 1783517592.022409 mapread 0x0012f2 thru 0x005fa3 (0x4cb2 bytes) 000371 1783517592.025278 trunc from 0x01cbe8 to 0x00c77d (0x1046a bytes) 000374 1783517592.030425 mapread 0x00078b thru 0x001dcb (0x1641 bytes) 000375 1783517592.032733 mapread 0x00622b thru 0x009dd0 (0x3ba6 bytes) 000376 1783517592.033534 mapread 0x00a1f7 thru 0x00c77d (0x2587 bytes) 000377 1783517592.034080 mapwrite 0x030eee thru 0x03ffff (0xf112 bytes) 000378 1783517592.057910 read 0x019f89 thru 0x0206f1 (0x6769 bytes) 000379 1783517592.062046 MIRROR_EXTEND to 2 mirrors 000380 1783517592.221556 MIRROR_EXTEND to 3 mirrors 000381 1783517592.374416 write 0x027849 thru 0x02e802 (0x6fba bytes) 000382 1783517592.396812 MIRROR_RESYNC 3 mirrors 000384 1783517592.576922 trunc from 0x040000 to 0x03657a (0x9a85 bytes) 000386 1783517592.602078 MIRROR_RESYNC 3 mirrors 000388 1783517592.766663 mapwrite 0x01cd76 thru 0x029587 (0xc812 bytes) 000389 1783517592.825183 mapwrite 0x034c8b thru 0x03d4b1 (0x8827 bytes) 000391 1783517592.863466 read 0x037a42 thru 0x0399fd (0x1fbc bytes) 000393 1783517592.864829 trunc from 0x03d4b2 to 0x02ff82 (0xd52f bytes) 000394 1783517592.873646 mapwrite 0x034ad9 thru 0x03dcc8 (0x91f0 bytes) 000395 1783517592.896375 MIRROR_RESYNC 3 mirrors 000396 1783517593.064021 trunc from 0x03dcc9 to 0x0326c9 (0xb5ff bytes) 000400 1783517593.086914 mapwrite 0x035700 thru 0x03ffff (0xa900 bytes) 000403 1783517593.110720 mapread 0x034477 thru 0x03815b (0x3ce5 bytes) 000404 1783517593.111505 mapread 0x02e677 thru 0x031e9d (0x3827 bytes) 000405 1783517593.114550 write 0x029438 thru 0x02a527 (0x10f0 bytes) 000407 1783517593.115626 write 0x00239a thru 0x0053b8 (0x301f bytes) 000408 1783517593.118751 mapread 0x03edfa thru 0x03ffff (0x1206 bytes) 000409 1783517593.119763 mapwrite 0x0062bf thru 0x015518 (0xf25a bytes) 000410 1783517593.141033 write 0x005ea3 thru 0x00c439 (0x6597 bytes) 000411 1783517593.143685 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8bbf7655 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8bbf7655 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000412 1783517593.216271 read 0x01b109 thru 0x01e81b (0x3713 bytes) 000413 1783517593.218578 mapwrite 0x0243b4 thru 0x02f478 (0xb0c5 bytes) 000415 1783517593.239688 mapwrite 0x0293f0 thru 0x02d79a (0x43ab bytes) 000417 1783517593.252083 mapwrite 0x00aa12 thru 0x013ecc (0x94bb bytes) 000418 1783517593.270424 mapread 0x012ecc thru 0x014473 (0x15a8 bytes) 000419 1783517593.271455 trunc from 0x040000 to 0x00f159 (0x30ea6 bytes) 000421 1783517593.277823 mapread 0x0034aa thru 0x00a987 (0x74de bytes) 000422 1783517593.279002 trunc from 0x00f15a to 0x021ae9 (0x12990 bytes) 000424 1783517593.284449 trunc from 0x021aea to 0x039b92 (0x180a9 bytes) 000426 1783517593.289902 read 0x0042d4 thru 0x0101b5 (0xbee2 bytes) 000427 1783517593.294775 mapread 0x0070fe thru 0x007546 (0x449 bytes) 000430 1783517593.296284 write 0x000a11 thru 0x004b96 (0x4186 bytes) 000431 1783517593.297330 write 0x011e5b thru 0x02119e (0xf344 bytes) 000432 1783517593.298292 write 0x03d296 thru 0x03ffff (0x2d6a bytes) 000434 1783517593.299638 read 0x039b8c thru 0x03ffff (0x6474 bytes) 000435 1783517593.302193 write 0x01d60f thru 0x02588a (0x827c bytes) 000436 1783517593.303204 trunc from 0x040000 to 0x00919b (0x36e64 bytes) 000442 1783517593.312231 write 0x00cd34 thru 0x00ec25 (0x1ef2 bytes) 000443 1783517593.313416 write 0x03b6ae thru 0x03cba1 (0x14f4 bytes) 000444 1783517593.314412 trunc from 0x03cba2 to 0x01074b (0x2c456 bytes) 000446 1783517593.320531 write 0x0391d1 thru 0x03db90 (0x49c0 bytes) 000447 1783517593.321380 trunc from 0x03db91 to 0x024f6d (0x18c23 bytes) 000449 1783517593.326669 write 0x023d45 thru 0x0336eb (0xf9a7 bytes) 000451 1783517593.329951 mapwrite 0x00fc18 thru 0x00fc96 (0x7f bytes) 000452 1783517593.347184 mapread 0x030288 thru 0x0336eb (0x3464 bytes) 000453 1783517593.348168 mapread 0x00f8ae thru 0x01eafe (0xf251 bytes) 000454 1783517593.369456 trunc from 0x0336ec to 0x02649a (0xd251 bytes) 000455 1783517593.374368 mapread 0x0254f7 thru 0x02649a (0xfa4 bytes) 000456 1783517593.375049 write 0x01ed80 thru 0x0211ed (0x246e bytes) 000457 1783517593.377259 mapwrite 0x00eb07 thru 0x010969 (0x1e63 bytes) 000459 1783517593.390589 trunc from 0x02649b to 0x00926a (0x1d230 bytes) 000461 1783517593.397364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x926b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x926b): Mirror 3: 0x281b78eb (stale) CRC-32 checksum value for chunk [0, 0x926b): Mirror 1: 0x972e922d Mirror 4: 0x972e922d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x926b: skipped 000462 1783517593.576254 mapwrite 0x01a1c2 thru 0x01d341 (0x3180 bytes) 000464 1783517593.622677 mapwrite 0x02160f thru 0x02bb4b (0xa53d bytes) 000465 1783517593.646019 trunc from 0x02bb4c to 0x014f24 (0x16c27 bytes) 000466 1783517593.652129 trunc from 0x014f25 to 0x003932 (0x115f2 bytes) 000468 1783517593.657090 read 0x00065a thru 0x003932 (0x32d9 bytes) 000469 1783517593.659166 trunc from 0x003933 to 0x012fd9 (0xf6a7 bytes) 000471 1783517593.664911 MIRROR_RESYNC 3 mirrors 000472 1783517593.848861 MIRROR_EXTEND to 4 mirrors 000474 1783517594.007644 mapwrite 0x006710 thru 0x0103af (0x9ca0 bytes) 000476 1783517594.062666 read 0x0129d3 thru 0x012fd9 (0x607 bytes) 000477 1783517594.064638 trunc from 0x012fda to 0x02f218 (0x1c23f bytes) 000479 1783517594.075989 MIRROR_RESYNC 4 mirrors 000480 1783517594.284153 write 0x03a13e thru 0x03ffff (0x5ec2 bytes) 000481 1783517594.303724 read 0x01e5e7 thru 0x026cff (0x8719 bytes) 000484 1783517594.308018 trunc from 0x040000 to 0x001a6b (0x3e594 bytes) 000485 1783517594.326612 trunc from 0x001a6c to 0x01f01b (0x1d5b0 bytes) 000486 1783517594.333689 MIRROR_RESYNC 4 mirrors 000488 1783517594.547949 mapwrite 0x01445d thru 0x023897 (0xf43b bytes) 000489 1783517594.589168 mapread 0x005089 thru 0x011b94 (0xcb0c bytes) 000490 1783517594.594224 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23898): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23898): Mirror 3: 0x7787dd8f (stale) Mirror 4: 0x7787dd8f (stale) Mirror 5: 0x7787dd8f (stale) CRC-32 checksum value for chunk [0, 0x23898): Mirror 1: 0x3a4460d9 Mirror 6: 0x3a4460d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23898: skipped 000491 1783517594.799390 mapread 0x022665 thru 0x023897 (0x1233 bytes) 000492 1783517594.807056 write 0x0278a7 thru 0x02afd1 (0x372b bytes) 000496 1783517594.827947 read 0x0191ee thru 0x029125 (0xff38 bytes) 000497 1783517594.830809 read 0x012687 thru 0x01de7d (0xb7f7 bytes) 000501 1783517594.835437 mapread 0x004043 thru 0x009785 (0x5743 bytes) 000503 1783517594.836689 mapwrite 0x007d0a thru 0x012034 (0xa32b bytes) 000504 1783517594.851373 read 0x026928 thru 0x02afd1 (0x46aa bytes) 000506 1783517594.852257 mapread 0x017ff6 thru 0x024517 (0xc522 bytes) 000507 1783517594.853197 write 0x03c0fa thru 0x03c9f1 (0x8f8 bytes) 000508 1783517594.853919 write 0x003ed5 thru 0x007a7c (0x3ba8 bytes) 000511 1783517594.855307 write 0x02683e thru 0x0333cc (0xcb8f bytes) 000513 1783517594.858403 write 0x00e190 thru 0x014c15 (0x6a86 bytes) 000514 1783517594.859426 mapread 0x01ed7c thru 0x026a7c (0x7d01 bytes) 000515 1783517594.860236 write 0x0172d0 thru 0x01fbba (0x88eb bytes) 000516 1783517594.860938 mapwrite 0x030b96 thru 0x03bd0b (0xb176 bytes) 000517 1783517594.880462 write 0x02bda1 thru 0x037a4e (0xbcae bytes) 000519 1783517594.882376 read 0x008da1 thru 0x013a1e (0xac7e bytes) 000523 1783517594.884748 mapread 0x01c1c8 thru 0x024e9b (0x8cd4 bytes) 000526 1783517594.888454 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 4: 0x21696f19 (stale) Mirror 5: 0x21696f19 (stale) Mirror 6: 0x95123cc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 4: 0x21696f19 (stale) Mirror 5: 0x21696f19 (stale) Mirror 6: 0x95123cc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9f2: skipped 000527 1783517595.004854 mapwrite 0x0338da thru 0x03ffff (0xc726 bytes) 000531 1783517595.034217 read 0x0166ad thru 0x020d05 (0xa659 bytes) 000532 1783517595.036933 trunc from 0x040000 to 0x00b3a4 (0x34c5b bytes) 000533 1783517595.041457 mapread 0x006638 thru 0x00b3a4 (0x4d6d bytes) 000534 1783517595.044986 trunc from 0x00b3a5 to 0x006318 (0x508c bytes) 000536 1783517595.050197 read 0x00309c thru 0x00338c (0x2f1 bytes) 000537 1783517595.052259 MIRROR_RESYNC 4 mirrors 000538 1783517595.256352 mapread 0x002cb4 thru 0x006318 (0x3665 bytes) 000539 1783517595.261293 mapread 0x004ea8 thru 0x006318 (0x1471 bytes) 000541 1783517595.262200 write 0x0039eb thru 0x010093 (0xc6a9 bytes) 000542 1783517595.280712 read 0x0041c6 thru 0x00fec8 (0xbd03 bytes) 000543 1783517595.281157 write 0x003404 thru 0x00c7e7 (0x93e4 bytes) 000546 1783517595.282544 read 0x003ff3 thru 0x009bab (0x5bb9 bytes) 000547 1783517595.282919 mapread 0x00e14b thru 0x010093 (0x1f49 bytes) 000549 1783517595.283846 trunc from 0x010094 to 0x0175a2 (0x750f bytes) 000550 1783517595.293474 mapread 0x014b3e thru 0x0175a2 (0x2a65 bytes) 000551 1783517595.296545 trunc from 0x0175a3 to 0x02c439 (0x14e97 bytes) 000556 1783517595.302014 write 0x0114dc thru 0x01fe08 (0xe92d bytes) 000557 1783517595.304262 trunc from 0x02c43a to 0x008629 (0x23e10 bytes) 000558 1783517595.308228 read 0x007957 thru 0x008629 (0xcd3 bytes) 000562 1783517595.311330 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x862a): Mirror 4: 0x39879441 (stale) Mirror 6: 0x39879441 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x862a): Mirror 4: 0x39879441 (stale) Mirror 6: 0x39879441 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x862a: skipped 000563 1783517595.400479 trunc from 0x00862a to 0x02771a (0x1f0f1 bytes) 000564 1783517595.406732 mapread 0x00a9e2 thru 0x01966e (0xec8d bytes) 000565 1783517595.417771 mapread 0x016bee thru 0x0258cd (0xece0 bytes) 000566 1783517595.422258 write 0x02d91e thru 0x03101d (0x3700 bytes) 000567 1783517595.423472 write 0x0077d0 thru 0x007999 (0x1ca bytes) 000568 1783517595.424655 write 0x01926b thru 0x01ebd1 (0x5967 bytes) 000570 1783517595.426307 mapread 0x01a7b4 thru 0x029e42 (0xf68f bytes) 000571 1783517595.430099 mapread 0x0082de thru 0x009e1e (0x1b41 bytes) 000576 1783517595.432501 mapread 0x011699 thru 0x01a158 (0x8ac0 bytes) 000577 1783517595.433350 mapwrite 0x03b16a thru 0x03dbcd (0x2a64 bytes) 000579 1783517595.466957 read 0x03c963 thru 0x03dbcd (0x126b bytes) 000580 1783517595.467614 mapread 0x02cd84 thru 0x031863 (0x4ae0 bytes) 000581 1783517595.468638 mapwrite 0x00ca25 thru 0x01341d (0x69f9 bytes) 000582 1783517595.494166 trunc from 0x03dbce to 0x02db23 (0x100aa bytes) 000583 1783517595.505415 trunc from 0x02db24 to 0x036917 (0x8df4 bytes) 000584 1783517595.515220 write 0x02d721 thru 0x02ecc4 (0x15a4 bytes) 000588 1783517595.522491 mapread 0x023b76 thru 0x02e251 (0xa6dc bytes) 000589 1783517595.523306 trunc from 0x036918 to 0x006125 (0x307f2 bytes) 000590 1783517595.528952 read 0x0049a0 thru 0x006125 (0x1786 bytes) 000592 1783517595.532649 mapread 0x002c04 thru 0x0037f6 (0xbf3 bytes) 000595 1783517595.534367 write 0x008b71 thru 0x013901 (0xad91 bytes) 000596 1783517595.535722 mapread 0x00e52d thru 0x00f6a6 (0x117a bytes) 000597 1783517595.538484 MIRROR_RESYNC 3 mirrors 000598 1783517595.705881 read 0x012a3b thru 0x013901 (0xec7 bytes) 000600 1783517595.708018 write 0x00537a thru 0x00869f (0x3326 bytes) 000605 1783517595.727069 write 0x02dafd thru 0x036738 (0x8c3c bytes) 000607 1783517595.729683 MIRROR_RESYNC 3 mirrors 000608 1783517595.913675 mapwrite 0x0012d5 thru 0x00d17d (0xbea9 bytes) 000609 1783517595.963398 mapwrite 0x02eb70 thru 0x0303d6 (0x1867 bytes) 000610 1783517595.975619 write 0x01aedb thru 0x028984 (0xdaaa bytes) 000612 1783517595.978406 write 0x02f691 thru 0x036a40 (0x73b0 bytes) 000613 1783517595.979100 write 0x00c594 thru 0x012caf (0x671c bytes) 000614 1783517595.981145 write 0x02aa4a thru 0x02f8c4 (0x4e7b bytes) 000615 1783517595.981812 write 0x00c95c thru 0x010157 (0x37fc bytes) 000616 1783517595.982388 write 0x011785 thru 0x01a95b (0x91d7 bytes) 000618 1783517595.983503 mapwrite 0x02533a thru 0x02b3de (0x60a5 bytes) 000619 1783517595.996265 read 0x0096ec thru 0x00bca8 (0x25bd bytes) 000620 1783517595.996738 write 0x02b2c0 thru 0x03ad68 (0xfaa9 bytes) 000621 1783517595.998005 mapwrite 0x0326a3 thru 0x03ffff (0xd95d bytes) 000622 1783517596.023901 write 0x006bdb thru 0x014c2d (0xe053 bytes) 000623 1783517596.024810 mapread 0x02f7bb thru 0x03f66c (0xfeb2 bytes) 000624 1783517596.025888 trunc from 0x040000 to 0x0307e1 (0xf81e bytes) 000625 1783517596.030688 write 0x00109f thru 0x00c6cc (0xb62e bytes) 000626 1783517596.031557 trunc from 0x0307e2 to 0x00d5d6 (0x2320b bytes) 000629 1783517596.036291 write 0x035650 thru 0x036186 (0xb37 bytes) 000630 1783517596.037271 read 0x01d94e thru 0x01e7ac (0xe5f bytes) 000636 1783517596.041725 mapwrite 0x03b716 thru 0x03ffff (0x48ea bytes) 000637 1783517596.062497 write 0x005971 thru 0x006f93 (0x1623 bytes) 000638 1783517596.063489 trunc from 0x040000 to 0x00e59d (0x31a62 bytes) 000639 1783517596.068653 trunc from 0x00e59e to 0x02392a (0x1538d bytes) 000640 1783517596.073144 mapwrite 0x0254d5 thru 0x02f8cd (0xa3f9 bytes) 000641 1783517596.107131 mapwrite 0x0092e3 thru 0x011bb4 (0x88d2 bytes) 000643 1783517596.127520 trunc from 0x02f8ce to 0x023a09 (0xbec4 bytes) 000645 1783517596.133784 read 0x01cecd thru 0x023a09 (0x6b3d bytes) 000646 1783517596.137735 mapwrite 0x02f52c thru 0x03595f (0x6434 bytes) 000648 1783517596.160176 MIRROR_RESYNC 3 mirrors 000650 1783517596.334557 mapwrite 0x00e101 thru 0x019ef9 (0xbdf9 bytes) 000653 1783517596.397041 MIRROR_RESYNC 3 mirrors 000654 1783517596.579877 mapread 0x00e79d thru 0x01348c (0x4cf0 bytes) 000655 1783517596.585751 mapwrite 0x0105d4 thru 0x019eba (0x98e7 bytes) 000656 1783517596.631034 read 0x0025ce thru 0x00f611 (0xd044 bytes) 000658 1783517596.634512 trunc from 0x035960 to 0x00f481 (0x264de bytes) 000660 1783517596.646896 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf482): Mirror 4: 0x6467d1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf482): Mirror 4: 0x6467d1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf482: skipped 000661 1783517596.723644 write 0x001346 thru 0x00ea8b (0xd746 bytes) 000662 1783517596.728425 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xf482): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf482): Mirror 4: 0x6467d1c (stale) CRC-32 checksum value for chunk [0, 0xf482): Mirror 1: 0xbf85c16e Mirror 5: 0xbf85c16e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf482: skipped 000665 1783517596.924181 mapread 0x007ff8 thru 0x00f481 (0x748a bytes) 000666 1783517596.930498 mapread 0x002d2a thru 0x00b45e (0x8735 bytes) 000667 1783517596.933153 trunc from 0x00f482 to 0x023278 (0x13df7 bytes) 000668 1783517596.958274 write 0x027355 thru 0x0295ec (0x2298 bytes) 000669 1783517596.959102 write 0x02eb31 thru 0x03e743 (0xfc13 bytes) 000671 1783517596.960569 write 0x009000 thru 0x00d4a2 (0x44a3 bytes) 000672 1783517596.963203 mapwrite 0x01a5ad thru 0x023cd0 (0x9724 bytes) 000674 1783517596.984169 write 0x03610a thru 0x039ba7 (0x3a9e bytes) 000676 1783517596.985413 write 0x0071c5 thru 0x011276 (0xa0b2 bytes) 000678 1783517596.990241 write 0x033001 thru 0x03ffff (0xcfff bytes) 000679 1783517596.991586 mapwrite 0x016435 thru 0x01e717 (0x82e3 bytes) 000680 1783517597.004392 read 0x020237 thru 0x02ee9e (0xec68 bytes) 000681 1783517597.006300 mapwrite 0x024f44 thru 0x02a51f (0x55dc bytes) 000683 1783517597.019504 trunc from 0x040000 to 0x0215f9 (0x1ea06 bytes) 000684 1783517597.026298 mapread 0x012831 thru 0x012bde (0x3ae bytes) 000687 1783517597.028530 write 0x0383ce thru 0x039e53 (0x1a86 bytes) 000688 1783517597.029691 mapwrite 0x023efb thru 0x02898f (0x4a95 bytes) 000691 1783517597.046350 write 0x00d2c5 thru 0x013e37 (0x6b73 bytes) 000693 1783517597.047953 read 0x018b89 thru 0x019d3f (0x11b7 bytes) 000695 1783517597.048730 trunc from 0x039e54 to 0x03b1cd (0x137a bytes) 000697 1783517597.054587 mapwrite 0x008c65 thru 0x010c55 (0x7ff1 bytes) 000698 1783517597.068814 trunc from 0x03b1ce to 0x01d5ef (0x1dbde bytes) 000700 1783517597.074404 mapread 0x014529 thru 0x01d5ef (0x90c7 bytes) 000702 1783517597.076009 mapread 0x0147ec thru 0x01d5ef (0x8e04 bytes) 000704 1783517597.077434 mapwrite 0x0101f2 thru 0x01fbd7 (0xf9e6 bytes) 000705 1783517597.106218 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1fbd8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fbd8): Mirror 4: 0x27699bad (stale) Mirror 5: 0x72b11aa (stale) CRC-32 checksum value for chunk [0, 0x1fbd8): Mirror 1: 0x60aaab71 Mirror 6: 0x60aaab71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fbd8: skipped 000706 1783517597.310086 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fbd8): Mirror 5: 0x72b11aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fbd8): Mirror 5: 0x72b11aa (stale) CRC-32 checksum value for chunk [0, 0x1fbd8): Mirror 1: 0x60aaab71 Mirror 6: 0x60aaab71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fbd8: skipped 000707 1783517597.468101 mapwrite 0x014f4a thru 0x01d0db (0x8192 bytes) 000708 1783517597.527328 MIRROR_RESYNC 3 mirrors 000712 1783517597.718406 trunc from 0x01fbd8 to 0x00c05b (0x13b7c bytes) 000713 1783517597.740168 write 0x01438f thru 0x018b6f (0x47e1 bytes) 000714 1783517597.740979 mapwrite 0x019c97 thru 0x027e3f (0xe1a9 bytes) 000715 1783517597.768057 MIRROR_RESYNC 3 mirrors 000717 1783517597.946513 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) 000722 1783517598.040770 write 0x02451f thru 0x0340ed (0xfbcf bytes) 000725 1783517598.065553 MIRROR_RESYNC 2 mirrors 000727 1783517598.241255 trunc from 0x0340ee to 0x00fac2 (0x2462b bytes) 000728 1783517598.266003 read 0x00e81f thru 0x00fac2 (0x12a4 bytes) 000729 1783517598.268410 read 0x0091f4 thru 0x00fac2 (0x68cf bytes) 000730 1783517598.271416 write 0x034633 thru 0x03ffff (0xb9cd bytes) 000732 1783517598.272929 mapwrite 0x019756 thru 0x0261e4 (0xca8f bytes) 000734 1783517598.294711 mapread 0x02bbb2 thru 0x030bff (0x504e bytes) 000736 1783517598.296356 write 0x00f42f thru 0x01abe6 (0xb7b8 bytes) 000737 1783517598.299884 MIRROR_RESYNC 2 mirrors 000739 1783517598.471326 read 0x03cba0 thru 0x03ffff (0x3460 bytes) 000742 1783517598.474087 read 0x01c0e7 thru 0x025f6a (0x9e84 bytes) 000743 1783517598.478159 read 0x006e20 thru 0x008837 (0x1a18 bytes) 000744 1783517598.478429 read 0x03a834 thru 0x03fbd9 (0x53a6 bytes) 000748 1783517598.479386 mapread 0x01e11d thru 0x02706f (0x8f53 bytes) 000749 1783517598.480111 write 0x009b1f thru 0x00ba04 (0x1ee6 bytes) 000751 1783517598.503900 mapread 0x021d5f thru 0x022168 (0x40a bytes) 000754 1783517598.505959 mapread 0x03e195 thru 0x03ffff (0x1e6b bytes) 000755 1783517598.506632 mapread 0x0150f5 thru 0x01b8a1 (0x67ad bytes) 000756 1783517598.507644 read 0x012176 thru 0x01bacd (0x9958 bytes) 000757 1783517598.508232 write 0x0081f5 thru 0x01362f (0xb43b bytes) 000759 1783517598.509584 read 0x036530 thru 0x03fce8 (0x97b9 bytes) 000760 1783517598.510258 mapwrite 0x0389d7 thru 0x03d0cc (0x46f6 bytes) 000761 1783517598.526784 read 0x02d1f5 thru 0x02dfbd (0xdc9 bytes) 000762 1783517598.527535 read 0x00ea14 thru 0x011264 (0x2851 bytes) 000763 1783517598.529991 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) 000768 1783517598.569580 read 0x014885 thru 0x015e4f (0x15cb bytes) 000769 1783517598.571964 read 0x03cc80 thru 0x03ffff (0x3380 bytes) 000771 1783517598.576902 write 0x02aa75 thru 0x031560 (0x6aec bytes) 000774 1783517598.579701 mapread 0x01d5b1 thru 0x0267fb (0x924b bytes) 000776 1783517598.581537 read 0x00fc47 thru 0x01563c (0x59f6 bytes) 000777 1783517598.582229 trunc from 0x040000 to 0x015e05 (0x2a1fa bytes) 000778 1783517598.591473 write 0x0234dd thru 0x02f397 (0xbebb bytes) 000781 1783517598.594166 mapread 0x022579 thru 0x02f397 (0xce1f bytes) 000782 1783517598.597213 read 0x0138e9 thru 0x014491 (0xba9 bytes) 000785 1783517598.598290 trunc from 0x02f398 to 0x017365 (0x18032 bytes) 000790 1783517598.604357 trunc from 0x017366 to 0x02fdd9 (0x18a74 bytes) 000793 1783517598.609631 write 0x03faf2 thru 0x03ffff (0x50e bytes) 000794 1783517598.610696 read 0x009767 thru 0x015c9b (0xc535 bytes) 000799 1783517598.613688 trunc from 0x040000 to 0x005512 (0x3aaed bytes) 000801 1783517598.619500 trunc from 0x005513 to 0x006206 (0xcf4 bytes) 000804 1783517598.625175 write 0x02d5d8 thru 0x0319fd (0x4426 bytes) 000805 1783517598.625982 trunc from 0x0319fe to 0x03d1ad (0xb7b0 bytes) 000808 1783517598.632885 read 0x004e70 thru 0x005796 (0x927 bytes) 000810 1783517598.634039 write 0x00f2f1 thru 0x017f64 (0x8c74 bytes) 000812 1783517598.639078 read 0x00f2e3 thru 0x01daad (0xe7cb bytes) 000813 1783517598.639565 read 0x0351c3 thru 0x03d1ad (0x7feb bytes) 000814 1783517598.640050 read 0x02d8b1 thru 0x034889 (0x6fd9 bytes) 000815 1783517598.640419 write 0x034693 thru 0x03a797 (0x6105 bytes) 000817 1783517598.641389 mapread 0x006c5b thru 0x014d94 (0xe13a bytes) 000819 1783517598.642474 trunc from 0x03d1ae to 0x00d5cb (0x2fbe2 bytes) 000820 1783517598.647292 trunc from 0x00d5cc to 0x003cc9 (0x9902 bytes) 000822 1783517598.651787 mapwrite 0x00bfe8 thru 0x014b08 (0x8b21 bytes) 000823 1783517598.672844 mapread 0x00d404 thru 0x0149e5 (0x75e2 bytes) 000824 1783517598.673843 read 0x00dc18 thru 0x012b22 (0x4f0b bytes) 000825 1783517598.674209 write 0x033ded thru 0x03ec19 (0xae2d bytes) 000826 1783517598.675478 write 0x003ffb thru 0x008a7f (0x4a85 bytes) 000827 1783517598.676291 mapread 0x006f4e thru 0x00bb37 (0x4bea bytes) 000828 1783517598.677017 trunc from 0x03ec1a to 0x0079f3 (0x37226 bytes) 000829 1783517598.683119 trunc from 0x0079f4 to 0x03530a (0x2d917 bytes) 000830 1783517598.688278 mapwrite 0x00fe9d thru 0x0173f1 (0x7555 bytes) 000835 1783517598.712728 mapwrite 0x02e137 thru 0x02f9cd (0x1897 bytes) 000837 1783517598.730757 trunc from 0x03530b to 0x00c27e (0x2908c bytes) 000838 1783517598.735891 mapwrite 0x031ff8 thru 0x037bdf (0x5be8 bytes) 000844 1783517598.763404 trunc from 0x037be0 to 0x02ffb3 (0x7c2c bytes) 000851 1783517598.770378 mapwrite 0x0394ee thru 0x03ffff (0x6b12 bytes) 000853 1783517598.790145 mapread 0x008755 thru 0x012064 (0x9910 bytes) 000860 1783517598.801842 mapread 0x016b7f thru 0x01d5b5 (0x6a37 bytes) 000861 1783517598.813067 write 0x001778 thru 0x006ec6 (0x574f bytes) 000862 1783517598.813998 mapwrite 0x03a848 thru 0x03ffff (0x57b8 bytes) 000865 1783517598.828148 write 0x03fb22 thru 0x03ffff (0x4de bytes) 000869 1783517598.829831 mapread 0x03224a thru 0x03a8b9 (0x8670 bytes) 000870 1783517598.842538 write 0x03c012 thru 0x03ffff (0x3fee bytes) 000874 1783517598.844355 mapread 0x00a6d6 thru 0x01120a (0x6b35 bytes) 000875 1783517598.845164 mapwrite 0x001f4c thru 0x00ad36 (0x8deb bytes) 000876 1783517598.859352 mapread 0x002829 thru 0x00bd9b (0x9573 bytes) 000877 1783517598.860511 mapread 0x014645 thru 0x01b584 (0x6f40 bytes) 000878 1783517598.867333 read 0x019b34 thru 0x028d06 (0xf1d3 bytes) 000879 1783517598.870914 trunc from 0x040000 to 0x01910a (0x26ef5 bytes) 000880 1783517598.875688 read 0x00ed38 thru 0x0146d7 (0x59a0 bytes) 000881 1783517598.876256 mapread 0x002979 thru 0x004b18 (0x21a0 bytes) 000882 1783517598.878357 MIRROR_EXTEND to 2 mirrors 000883 1783517599.042350 trunc from 0x01910b to 0x018c22 (0x4e8 bytes) 000886 1783517599.072171 trunc from 0x018c23 to 0x00ab73 (0xe0af bytes) 000889 1783517599.082020 read 0x004e83 thru 0x00ab73 (0x5cf1 bytes) 000892 1783517599.086150 trunc from 0x00ab74 to 0x03e14a (0x335d7 bytes) 000894 1783517599.094314 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) 000896 1783517599.139426 read 0x01213c thru 0x020300 (0xe1c5 bytes) 000897 1783517599.142698 mapwrite 0x002e89 thru 0x0054c7 (0x263f bytes) 000898 1783517599.162709 trunc from 0x03e14b to 0x00e5e1 (0x2fb69 bytes) 000899 1783517599.169388 read 0x009900 thru 0x00e5e1 (0x4ce2 bytes) 000905 1783517599.171856 mapwrite 0x037491 thru 0x03b8fe (0x446e bytes) 000907 1783517599.195755 trunc from 0x03b8ff to 0x01c301 (0x1f5fd bytes) 000909 1783517599.201632 mapread 0x005224 thru 0x014528 (0xf305 bytes) 000910 1783517599.202891 write 0x03f027 thru 0x03ffff (0xfd9 bytes) 000912 1783517599.204514 write 0x02c842 thru 0x02ffd8 (0x3797 bytes) 000913 1783517599.210015 read 0x018835 thru 0x023953 (0xb11f bytes) 000914 1783517599.215375 write 0x007702 thru 0x015300 (0xdbff bytes) 000919 1783517599.218533 read 0x00526f thru 0x00f121 (0x9eb3 bytes) 000923 1783517599.220771 read 0x017bc9 thru 0x01db28 (0x5f60 bytes) 000924 1783517599.221175 write 0x01c911 thru 0x0241d2 (0x78c2 bytes) 000925 1783517599.221901 read 0x015ed2 thru 0x023b46 (0xdc75 bytes) 000927 1783517599.222927 read 0x021704 thru 0x02c00e (0xa90b bytes) 000929 1783517599.223646 mapread 0x00d00b thru 0x0157e9 (0x87df bytes) 000930 1783517599.224455 write 0x02f696 thru 0x039ba7 (0xa512 bytes) 000931 1783517599.225420 trunc from 0x040000 to 0x03036e (0xfc91 bytes) 000932 1783517599.233576 mapread 0x011b65 thru 0x016b51 (0x4fed bytes) 000934 1783517599.235263 write 0x02c57f thru 0x0370fd (0xab7f bytes) 000935 1783517599.236718 trunc from 0x0370fe to 0x02c0a6 (0xb057 bytes) 000940 1783517599.244827 read 0x01355c thru 0x01a411 (0x6eb6 bytes) 000941 1783517599.245519 mapwrite 0x03f4fc thru 0x03ffff (0xb04 bytes) 000942 1783517599.280315 trunc from 0x040000 to 0x001618 (0x3e9e7 bytes) 000943 1783517599.286208 read 0x0012ad thru 0x001618 (0x36c bytes) 000944 1783517599.286776 trunc from 0x001619 to 0x0366bf (0x350a7 bytes) 000945 1783517599.292920 MIRROR_EXTEND to 2 mirrors 000946 1783517599.446446 read 0x00ecb3 thru 0x016804 (0x7b52 bytes) 000947 1783517599.449567 mapread 0x01ac6f thru 0x0279bb (0xcd4d bytes) 000949 1783517599.458516 read 0x02d63a thru 0x032002 (0x49c9 bytes) 000950 1783517599.465423 write 0x00a39b thru 0x00e249 (0x3eaf bytes) 000951 1783517599.489479 mapwrite 0x018f34 thru 0x01bb60 (0x2c2d bytes) 000953 1783517599.513344 mapread 0x0111ef thru 0x01d2d9 (0xc0eb bytes) 000954 1783517599.528344 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x366c0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366c0): Mirror 2: 0x5a3c5283 (stale) CRC-32 checksum value for chunk [0, 0x366c0): Mirror 1: 0x290525cb Mirror 3: 0x290525cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366c0: skipped 000957 1783517599.736919 read 0x0231a5 thru 0x02d463 (0xa2bf bytes) 000959 1783517599.742316 mapwrite 0x00d5d5 thru 0x018f28 (0xb954 bytes) 000961 1783517599.800217 trunc from 0x0366c0 to 0x00bf81 (0x2a73e bytes) 000965 1783517599.812544 trunc from 0x00bf82 to 0x02a152 (0x1e1d1 bytes) 000967 1783517599.818341 read 0x01edda thru 0x0236d2 (0x48f9 bytes) 000968 1783517599.821637 trunc from 0x02a153 to 0x01cb65 (0xd5ed bytes) 000969 1783517599.826211 trunc from 0x01cb66 to 0x039f99 (0x1d434 bytes) 000970 1783517599.832668 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x709:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39f9a): Mirror 2: 0x56a992c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f9a): Mirror 2: 0x56a992c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f9a: skipped 000972 1783517599.926147 read 0x02306c thru 0x02f0ac (0xc041 bytes) 000973 1783517599.931034 MIRROR_RESYNC 2 mirrors 000974 1783517600.123659 mapwrite 0x01d054 thru 0x02678a (0x9737 bytes) 000975 1783517600.181201 write 0x00767f thru 0x01628b (0xec0d bytes) 000976 1783517600.184303 mapwrite 0x0083ac thru 0x00862d (0x282 bytes) 000977 1783517600.195237 write 0x016fab thru 0x02083b (0x9891 bytes) 000978 1783517600.196042 mapwrite 0x009ee4 thru 0x00a067 (0x184 bytes) 000979 1783517600.206574 write 0x00f167 thru 0x013fce (0x4e68 bytes) 000980 1783517600.207293 trunc from 0x039f9a to 0x019500 (0x20a99 bytes) 000982 1783517600.213632 mapwrite 0x018aba thru 0x0276c5 (0xec0c bytes) 000983 1783517600.238745 read 0x0172d0 thru 0x0183a8 (0x10d9 bytes) 000984 1783517600.239215 mapread 0x0135e7 thru 0x016402 (0x2e1c bytes) 000987 1783517600.240636 mapread 0x0124c9 thru 0x017ed8 (0x5a10 bytes) 000988 1783517600.241369 read 0x00f159 thru 0x0124ed (0x3395 bytes) 000989 1783517600.241760 write 0x01d24e thru 0x022104 (0x4eb7 bytes) 000990 1783517600.242539 mapwrite 0x00daac thru 0x0113ef (0x3944 bytes) 000994 1783517600.260325 MIRROR_RESYNC 2 mirrors 000995 1783517600.442247 write 0x03e49d thru 0x03ffff (0x1b63 bytes) 000997 1783517600.469126 mapread 0x025a79 thru 0x02b4e9 (0x5a71 bytes) 000999 1783517600.476260 mapread 0x00e597 thru 0x01b528 (0xcf92 bytes) 001000 1783517600.485371 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x193aee96 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1bcd0b62 Mirror 3: 0x1bcd0b62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped All operations completed A-OK! PASS 70a (18s) == sanity-flr test 71: check mirror extend parent fid ==== 09:33:22 (1783517602) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 09:33:23 (1783517603) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa81:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa7d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0641104 s, 32.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa81:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa82:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa7d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434948 s, 24.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa81:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa82:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa7e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa7d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa81:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa82:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa7e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa7d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa83:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (2s) == sanity-flr test 200a: stress test ===================== 09:33:25 (1783517605) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8598080 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2699008 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8920896 .. Extending file size to 4204832 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4544544 .. done Extending file size to 7093472 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7977536 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8449664 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8709600 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8999392 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8777088 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9230240 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9085440 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 60395 60396 60398 60400 60404 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:34:30 (1783517670) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1087264 .. Extending file size to 4212768 .. failed Extending file size to 8474304 .. Extending file size to 5718240 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8155040 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8436480 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8932544 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9095040 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9155808 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9163648 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63953 63954 63956 63958 63960 done 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) 192.168.202.145@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:35:35 (1783517735) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0376714 s, 55.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0661825 s, 31.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc589a8d1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:35:41 (1783517741) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0254922 s, 82.3 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 09:35:43 (1783517743) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x748:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa94:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa94:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa95:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0xa98:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:35:45 (1783517745) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.034164 s, 61.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa96:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa99:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:35:47 (1783517747) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0435919 s, 48.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa98:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa97:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa98:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9c:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:35:49 (1783517749) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0353429 s, 59.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa99:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa99:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9e:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:36:13 (1783517773) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313457 s, 33.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaa0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaa0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa9f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9b:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:36:57 (1783517817) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.043746 s, 47.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaa1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaa1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:37:44 (1783517864) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg245-server mds-ost sync done. Creating new pool test_204f oleg245-server: Pool lustre.test_204f created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0429808 s, 48.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaa2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa9e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaa2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg245-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:38:41 (1783517921) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:38:43 (1783517923) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:38:46 (1783517926) oleg245-server: Pool lustre.test_206 created Adding targets to pool oleg245-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg245-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg245-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg245-server: Pool lustre.test_206 destroyed PASS 206 (14s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:39:00 (1783517940) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228877 s, 45.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106666 s, 98.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd2b4b42b Mirror 3: 0xd2b4b42b PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:39:03 (1783517943) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:39:04 (1783517944) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:39:05 (1783517945) Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.024163 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xee0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xee0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xee0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x13f0000 Read this (4 bytes): BBBB 192.168.202.145@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg245-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:39:21 (1783517961) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00461271 s, 1.8 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:39:23 (1783517963) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149502 s, 70.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x76d lmm_fid: [0x200000404:0x76d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x76d lmm_fid: [0x200000404:0x76d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xaac:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835821 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-08 09:39:24.000000000 -0400 Modify: 2026-07-08 09:39:24.000000000 -0400 Change: 2026-07-08 09:39:24.000000000 -0400 Birth: 2026-07-08 09:39:23.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:39:26 (1783517966) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:39:27 (1783517967) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.236252 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0418113 s, 98.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 8 09:39 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 8 09:39 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:39:32 (1783517972) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:39:36 (1783517976) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (4s) == sanity-flr test complete, duration 1015 sec =========== 09:39:40 (1783517980) === sanity-flr: start cleanup 09:39:41 (1783517981) === === sanity-flr: finish cleanup 09:39:45 (1783517985) ===