-----============= acceptance-small: sanity-flr ============----- Sun Aug 23 00:41:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg307-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 00:42:13 (1787460133) === oleg307-client.virtnet: executing check_config_client /mnt/lustre oleg307-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg307-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff976f931e7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff976f931e7800.idle_timeout=debug disable quota as required oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 00:42:51 (1787460171) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7619] == sanity-flr test 0a: lfs mirror create with -N option == 00:42:54 (1787460174) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:43:12 (1787460192) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (11s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:43:23 (1787460203) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:43:27 (1787460207) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:43:57 (1787460237) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:44:02 (1787460242) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 00:44:05 (1787460245) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff976f931e7800.stats=clear osc.lustre-OST0001-osc-ffff976f931e7800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:44:14 (1787460254) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:44:27 (1787460267) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:44:35 (1787460275) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 00:44:39 (1787460279) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:44:47 (1787460287) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 00:44:57 (1787460297) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:45:05 (1787460305) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:45:09 (1787460309) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:45:16 (1787460316) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:45:27 (1787460327) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:45:35 (1787460335) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.79758 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194797 s, 5.4 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:45:47 (1787460347) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.379638 s, 44.2 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 00:45:56 (1787460356) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:46:08 (1787460368) Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg307-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (147s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:48:35 (1787460515) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.251547 s, 16.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:49:21 (1787460561) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:50:48 (1787460648) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.30522 s, 8.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 23 00:50 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.408s user 0m0.001s sys 0m0.200s Stopping /mnt/lustre-ost1 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976f931e7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m2.193s user 0m0.002s sys 0m0.244s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg307-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg307-server oleg307-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.604s user 0m0.000s sys 0m0.158s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg307-server: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg307-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg307-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff976f931e7800.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:52:32 (1787460752) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:52:36 (1787460756) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:52:39 (1787460759) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 00:52:43 (1787460763) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 00:52:51 (1787460771) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.777211 s, 16.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:53:06 (1787460786) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047082 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664618 s, 15.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:53:14 (1787460794) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:53:23 (1787460803) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010764 s, 951 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00303615 s, 337 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113192 s, 905 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00167096 s, 613 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 00:53:34 (1787460814) create test files with size 15740351 .. a4108d2bb6c3e3063b3b568e3559e91b - 1d5e37dbd1ad296ae64934c1c60fc791 - 3424138d447f9f786ff6b806cc51b513 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 00:54:01 (1787460841) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:54:20 (1787460860) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:54:27 (1787460867) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153046 s, 6.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500793 s, 20.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 00:54:35 (1787460875) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165499 s, 12.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.35577 s, 11.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.294059 s, 7.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 00:54:52 (1787460892) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 00:54:55 (1787460895) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:54:58 (1787460898) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409812 s, 25.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 00:55:06 (1787460906) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:55:10 (1787460910) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:55:22 (1787460922) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.327374 s, 32.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:55:36 (1787460936) llite.lustre-ffff976f996f6000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.50601 s, 20.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg307-client.virtnet /mnt/lustre (opts:) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff976f915f8000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:55:52 (1787460952) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:56:37 (1787460997) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:57:18 (1787461038) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 00:57:21 (1787461041) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.151643 s, 20.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.220426 s, 14.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd19f3936 (nosync,stale) Mirror 2: 0xd19f3936 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3475c985ad3bed2c0d96131da5e5db30 - mirror 1 checksum: da419de92e8d019c58facbe204f01650 - mirror 2 checksum: 3475c985ad3bed2c0d96131da5e5db30 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-23T00:57:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-23T00:57:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3475c985ad3bed2c0d96131da5e5db30 - mirror 1 checksum: da419de92e8d019c58facbe204f01650 - mirror 2 checksum: da419de92e8d019c58facbe204f01650 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-23T00:57:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-23T00:57:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:57:35 (1787461055) Failing mds1 on oleg307-server Stopping /mnt/lustre-mds1 (opts:) on oleg307-server 00:57:42 (1787461062) shut down facet: mds1 facet_host: oleg307-server facet_failover_host: oleg307-server Failover mds1 to oleg307-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 00:58:06 (1787461086) targets are mounted 00:58:06 (1787461086) facet_failover done oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263709 s, 39.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:58:24 (1787461104) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:58:28 (1787461108) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:58:31 (1787461111) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0109133 s, 1.5 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:58:40 (1787461120) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:58:43 (1787461123) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00942169 s, 435 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0571667 s, 71.7 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:58:52 (1787461132) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:59:18 (1787461158) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:59:49 (1787461189) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:00:14 (1787461214) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:00:39 (1787461239) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 01:00:46 (1787461246) mirror create pid 54338 mirror split pid 54339 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54338 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54339 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 01:01:54 (1787461314) Seed set to 1316 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1787461316.939122 mapwrite 0x020cef thru 0x022922 (0x1c34 bytes) 000005 1787461317.027012 mapread 0x00c028 thru 0x013187 (0x7160 bytes) 000006 1787461317.041411 mapwrite 0x033550 thru 0x03ed9c (0xb84d bytes) 000007 1787461317.211324 mapread 0x012200 thru 0x015402 (0x3203 bytes) 000012 1787461317.226165 mapwrite 0x0257e7 thru 0x0310fb (0xb915 bytes) 000014 1787461317.294324 MIRROR_EXTEND to 2 mirrors 000017 1787461318.173937 mapread 0x0269ce thru 0x02807d (0x16b0 bytes) 000018 1787461318.213823 mapread 0x0068ab thru 0x0155c4 (0xed1a bytes) 000019 1787461318.232748 mapread 0x03c56d thru 0x03ed9c (0x2830 bytes) 000020 1787461318.246660 read 0x03c949 thru 0x03ed9c (0x2454 bytes) truncating to largest ever: 0x315d7 000021 1787461318.247641 trunc from 0x03ed9d to 0x0315d6 (0xd7c6 bytes) truncating to largest ever: 0x38bfd 000022 1787461318.398487 trunc from 0x0315d7 to 0x038bfc (0x7626 bytes) 000025 1787461318.426548 read 0x00175c thru 0x004dfe (0x36a3 bytes) truncating to largest ever: 0x3e8cf 000026 1787461318.448916 trunc from 0x038bfd to 0x03e8ce (0x5cd2 bytes) 000028 1787461318.475050 read 0x024520 thru 0x0324d5 (0xdfb6 bytes) 000029 1787461318.476221 mapwrite 0x0215e2 thru 0x023e5d (0x287c bytes) 000031 1787461318.518497 read 0x032692 thru 0x03e439 (0xbda8 bytes) 000035 1787461318.529312 read 0x005077 thru 0x006f3a (0x1ec4 bytes) 000036 1787461318.534768 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000038 1787461318.724073 trunc from 0x03e8cf to 0x00ed67 (0x2fb67 bytes) 000039 1787461318.746397 read 0x004c96 thru 0x00866d (0x39d8 bytes) 000041 1787461318.762582 trunc from 0x00ed68 to 0x00f4ed (0x786 bytes) 000042 1787461318.788927 mapread 0x00ed2a thru 0x00f4ed (0x7c4 bytes) 000044 1787461318.811598 mapread 0x001a46 thru 0x002f49 (0x1504 bytes) 000049 1787461318.830044 mapread 0x00a452 thru 0x00d6ce (0x327d bytes) 000055 1787461318.838745 mapwrite 0x01d90e thru 0x01f79d (0x1e90 bytes) 000058 1787461318.946693 read 0x01a838 thru 0x01c399 (0x1b62 bytes) 000059 1787461318.948737 write 0x001846 thru 0x003557 (0x1d12 bytes) 000060 1787461318.958427 read 0x011d81 thru 0x01b729 (0x99a9 bytes) 000061 1787461318.986034 write 0x03b020 thru 0x03ffff (0x4fe0 bytes) 000062 1787461318.989456 trunc from 0x040000 to 0x02f34d (0x10cb2 bytes) 000066 1787461319.047015 trunc from 0x02f34e to 0x00b21c (0x24131 bytes) 000069 1787461319.084707 mapread 0x002ae5 thru 0x0067fd (0x3d19 bytes) 000072 1787461319.107995 mapread 0x007a0f thru 0x00b21c (0x380e bytes) 000074 1787461319.111491 mapwrite 0x01103f thru 0x01f53c (0xe4fe bytes) 000077 1787461319.341704 mapread 0x00f09d thru 0x01165b (0x25bf bytes) 000078 1787461319.361094 trunc from 0x01f53d to 0x007fac (0x17590 bytes) 000085 1787461319.385195 write 0x02c3d2 thru 0x02f018 (0x2c47 bytes) 000086 1787461319.386339 write 0x020e96 thru 0x029387 (0x84f2 bytes) 000088 1787461319.396742 mapread 0x014957 thru 0x01eb5d (0xa207 bytes) 000090 1787461319.482658 mapread 0x00b103 thru 0x00b6d9 (0x5d7 bytes) 000092 1787461319.489392 write 0x01c399 thru 0x02441c (0x8084 bytes) 000095 1787461319.492414 write 0x03b09b thru 0x03ffff (0x4f65 bytes) 000096 1787461319.494738 trunc from 0x040000 to 0x02ca5a (0x135a5 bytes) 000098 1787461319.522153 read 0x02c9b7 thru 0x02ca5a (0xa4 bytes) 000099 1787461319.522920 mapread 0x016738 thru 0x0251ea (0xeab3 bytes) 000101 1787461319.527227 mapread 0x01c9ce thru 0x029de3 (0xd416 bytes) 000103 1787461319.531106 write 0x01471d thru 0x01977e (0x5062 bytes) 000104 1787461319.532660 mapwrite 0x01e430 thru 0x028efb (0xaacc bytes) 000107 1787461319.624368 trunc from 0x02ca5b to 0x011f62 (0x1aaf8 bytes) 000111 1787461319.668654 write 0x02972b thru 0x035869 (0xc13f bytes) 000112 1787461319.700507 MIRROR_EXTEND to 2 mirrors 000116 1787461320.836193 read 0x022785 thru 0x02db58 (0xb3d4 bytes) 000117 1787461320.843751 trunc from 0x03586a to 0x02e016 (0x7853 bytes) 000118 1787461320.980422 mapread 0x01172a thru 0x019b87 (0x845e bytes) 000119 1787461321.030211 mapwrite 0x029cdc thru 0x032b66 (0x8e8b bytes) 000120 1787461321.235472 read 0x00b79f thru 0x00e671 (0x2ed3 bytes) 000123 1787461321.264900 mapwrite 0x029a3b thru 0x0353a8 (0xb96e bytes) 000125 1787461321.458563 mapwrite 0x0076df thru 0x016d4b (0xf66d bytes) 000128 1787461321.585451 mapread 0x0106f9 thru 0x018706 (0x800e bytes) 000129 1787461321.586377 read 0x032dfc thru 0x0353a8 (0x25ad bytes) 000130 1787461321.586902 mapwrite 0x00df5b thru 0x014e89 (0x6f2f bytes) 000131 1787461321.695930 mapread 0x0193b6 thru 0x01ad92 (0x19dd bytes) 000132 1787461321.697044 mapwrite 0x021a1c thru 0x028238 (0x681d bytes) 000133 1787461321.796926 mapwrite 0x01691d thru 0x0246c9 (0xddad bytes) 000134 1787461321.878990 mapread 0x019672 thru 0x01eb67 (0x54f6 bytes) 000137 1787461321.887174 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000138 1787461322.043134 MIRROR_EXTEND to 2 mirrors 000139 1787461322.867615 write 0x01d2eb thru 0x0217ae (0x44c4 bytes) 000141 1787461322.970972 mapread 0x01ff3d thru 0x0227a8 (0x286c bytes) 000143 1787461322.977974 read 0x002d24 thru 0x006c7e (0x3f5b bytes) 000144 1787461322.986756 trunc from 0x0353a9 to 0x02101e (0x1438a bytes) 000146 1787461323.030439 MIRROR_RESYNC 2 mirrors 000148 1787461323.780545 mapwrite 0x031d92 thru 0x03d0e7 (0xb356 bytes) 000149 1787461323.987040 MIRROR_RESYNC 2 mirrors 000151 1787461325.339640 mapwrite 0x00d695 thru 0x00e021 (0x98d bytes) 000152 1787461325.576250 read 0x02fed9 thru 0x036237 (0x635f bytes) 000153 1787461325.607160 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d0e8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0e8): Mirror 2: 0x4c742120 (stale) CRC-32 checksum value for chunk [0, 0x3d0e8): Mirror 1: 0xcd45d967 Mirror 3: 0xcd45d967 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0e8: skipped 000154 1787461327.047079 mapread 0x0125e7 thru 0x017ecb (0x58e5 bytes) 000156 1787461327.075099 mapread 0x022d1c thru 0x02f4f1 (0xc7d6 bytes) 000157 1787461327.098200 read 0x00d865 thru 0x0141ac (0x6948 bytes) 000159 1787461327.101526 mapread 0x01967f thru 0x01f76b (0x60ed bytes) 000160 1787461327.102412 mapwrite 0x0098fe thru 0x01045d (0x6b60 bytes) 000165 1787461327.291588 mapwrite 0x020be4 thru 0x0231ee (0x260b bytes) 000166 1787461327.374318 read 0x00ecf9 thru 0x017c83 (0x8f8b bytes) 000167 1787461327.396570 mapwrite 0x008c72 thru 0x015568 (0xc8f7 bytes) 000169 1787461327.507181 read 0x012d8d thru 0x015d2d (0x2fa1 bytes) 000170 1787461327.507845 mapwrite 0x03bbf7 thru 0x03ffff (0x4409 bytes) 000171 1787461327.663095 trunc from 0x040000 to 0x01dcc8 (0x22337 bytes) 000173 1787461327.722339 write 0x0077dd thru 0x01609e (0xe8c2 bytes) 000174 1787461327.725824 mapwrite 0x00ea27 thru 0x0185c1 (0x9b9b bytes) 000175 1787461327.845605 mapread 0x00c687 thru 0x01bc90 (0xf60a bytes) 000177 1787461327.848987 mapwrite 0x00ec2c thru 0x01c487 (0xd85c bytes) 000184 1787461327.942201 read 0x0087e5 thru 0x012d22 (0xa53e bytes) 000185 1787461327.959146 MIRROR_RESYNC 3 mirrors 000187 1787461329.466196 trunc from 0x01dcc9 to 0x00ff4c (0xdd7c bytes) 000188 1787461329.635274 trunc from 0x00ff4d to 0x0247c6 (0x1487a bytes) 000190 1787461329.681601 MIRROR_RESYNC 3 mirrors 000191 1787461330.964494 mapwrite 0x03ace6 thru 0x03ffff (0x531a bytes) 000192 1787461331.269578 write 0x0351cb thru 0x03ffff (0xae35 bytes) 000193 1787461331.273037 mapread 0x01eb94 thru 0x0277ae (0x8c1b bytes) 000194 1787461331.304589 mapwrite 0x0183d8 thru 0x0260d2 (0xdcfb bytes) 000195 1787461331.471571 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x79ed5937 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x79ed5937 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000196 1787461331.975519 trunc from 0x040000 to 0x004591 (0x3ba6e bytes) 000197 1787461332.015590 mapwrite 0x03748a thru 0x03accd (0x3844 bytes) 000198 1787461332.162025 mapread 0x01caa5 thru 0x02128c (0x47e8 bytes) 000199 1787461332.185316 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000200 1787461332.389622 mapread 0x034432 thru 0x035aa0 (0x166f bytes) 000202 1787461332.423570 mapread 0x016161 thru 0x021224 (0xb0c4 bytes) 000205 1787461332.465903 mapread 0x003647 thru 0x007f0b (0x48c5 bytes) 000208 1787461332.489891 read 0x03a9e9 thru 0x03accd (0x2e5 bytes) 000212 1787461332.522369 mapwrite 0x0272d5 thru 0x02e8e4 (0x7610 bytes) 000214 1787461332.606416 trunc from 0x03acce to 0x00c450 (0x2e87d bytes) 000217 1787461332.633921 trunc from 0x00c451 to 0x01b2d9 (0xee89 bytes) 000218 1787461332.655226 mapwrite 0x03df06 thru 0x03eadb (0xbd6 bytes) 000219 1787461332.715238 MIRROR_EXTEND to 2 mirrors 000220 1787461334.172692 mapwrite 0x03d018 thru 0x03fd5c (0x2d45 bytes) 000223 1787461334.518847 write 0x001e74 thru 0x00dff4 (0xc181 bytes) 000225 1787461334.537739 read 0x0003fc thru 0x009d92 (0x9997 bytes) 000229 1787461334.547103 write 0x002f60 thru 0x00bb97 (0x8c38 bytes) 000231 1787461334.549454 write 0x007646 thru 0x009f24 (0x28df bytes) 000232 1787461334.556092 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 2: 0xa870c51d (stale) CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 1: 0xfa3b7564 Mirror 3: 0xfa3b7564 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd5d: skipped 000233 1787461336.103638 mapread 0x0047eb thru 0x008b6e (0x4384 bytes) 000239 1787461336.142990 mapwrite 0x027a55 thru 0x02d83a (0x5de6 bytes) 000240 1787461336.372295 read 0x002fbd thru 0x0086be (0x5702 bytes) 000241 1787461336.389673 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 2: 0xa870c51d (stale) Mirror 3: 0xfa3b7564 (stale) CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 1: 0x1e5b5faa Mirror 4: 0x1e5b5faa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd5d: skipped 000242 1787461337.947897 mapwrite 0x034e47 thru 0x03ffff (0xb1b9 bytes) 000244 1787461338.276703 mapread 0x027a90 thru 0x02b75d (0x3cce bytes) 000245 1787461338.286557 write 0x0335ec thru 0x03b771 (0x8186 bytes) 000248 1787461338.298640 trunc from 0x040000 to 0x0317d9 (0xe826 bytes) 000249 1787461338.355481 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x317da): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317da): Mirror 2: 0x94a5a78b (stale) Mirror 3: 0x7278e572 (stale) Mirror 4: 0xde3eaf93 (stale) CRC-32 checksum value for chunk [0, 0x317da): Mirror 1: 0xde3eaf93 Mirror 5: 0xde3eaf93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317da: skipped 000250 1787461340.199765 trunc from 0x0317da to 0x0148c0 (0x1cf19 bytes) 000251 1787461340.352348 trunc from 0x0148c1 to 0x035437 (0x20b77 bytes) 000252 1787461340.402106 mapwrite 0x011ea0 thru 0x0171ee (0x534f bytes) 000253 1787461340.493639 write 0x02edda thru 0x03bb40 (0xcd67 bytes) 000254 1787461340.504958 trunc from 0x03bb41 to 0x01b8e4 (0x2025c bytes) 000258 1787461340.531567 write 0x01f3d0 thru 0x029560 (0xa191 bytes) 000259 1787461340.537479 write 0x03d062 thru 0x03e93a (0x18d9 bytes) 000260 1787461340.539422 mapread 0x00d6da thru 0x01b3a4 (0xdccb bytes) 000261 1787461340.542777 mapread 0x0289ce thru 0x02f83c (0x6e6f bytes) 000263 1787461340.554249 mapwrite 0x018f4d thru 0x028bf2 (0xfca6 bytes) 000264 1787461340.617713 mapwrite 0x0197d5 thru 0x023a59 (0xa285 bytes) 000269 1787461340.681750 mapread 0x0114d1 thru 0x015b26 (0x4656 bytes) 000270 1787461340.687690 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e93b): Mirror 3: 0xc1fe5161 (stale) Mirror 4: 0xbae9e680 (stale) Mirror 5: 0x9ebb5b88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e93b): Mirror 3: 0xc1fe5161 (stale) Mirror 4: 0xbae9e680 (stale) Mirror 5: 0x9ebb5b88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e93b: skipped 000271 1787461341.196218 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e93b): Mirror 3: 0xc1fe5161 (stale) Mirror 4: 0xbae9e680 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e93b): Mirror 3: 0xc1fe5161 (stale) Mirror 4: 0xbae9e680 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e93b: skipped 000273 1787461341.575980 mapwrite 0x002887 thru 0x01211b (0xf895 bytes) 000274 1787461341.652822 mapread 0x03e46b thru 0x03e93a (0x4d0 bytes) 000275 1787461341.662738 mapread 0x0218f0 thru 0x029800 (0x7f11 bytes) 000277 1787461341.671823 write 0x038761 thru 0x03ffff (0x789f bytes) 000279 1787461341.673453 mapread 0x03a4cb thru 0x03ffff (0x5b35 bytes) 000281 1787461341.681945 mapwrite 0x011c40 thru 0x018547 (0x6908 bytes) 000282 1787461341.777101 write 0x036ea1 thru 0x03ffff (0x915f bytes) 000285 1787461341.798389 trunc from 0x040000 to 0x035af2 (0xa50d bytes) 000286 1787461341.833204 trunc from 0x035af3 to 0x0175b6 (0x1e53c bytes) 000287 1787461341.856464 mapread 0x00d6e5 thru 0x011daf (0x46cb bytes) 000288 1787461341.867738 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x175b7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x175b7): Mirror 3: 0x684fee9e (stale) Mirror 4: 0x684fee9e (stale) CRC-32 checksum value for chunk [0, 0x175b7): Mirror 1: 0x543238dc Mirror 5: 0x543238dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x175b7: skipped 000289 1787461342.951002 write 0x0373ce thru 0x03f399 (0x7fcc bytes) 000290 1787461343.062941 mapread 0x03375f thru 0x03cbcc (0x946e bytes) 000293 1787461343.081822 write 0x03e54f thru 0x03ffff (0x1ab1 bytes) 000295 1787461343.083143 trunc from 0x040000 to 0x018ea1 (0x2715e bytes) 000296 1787461343.113270 trunc from 0x018ea2 to 0x01d8d4 (0x4a33 bytes) 000298 1787461343.141891 read 0x009f6d thru 0x00b422 (0x14b6 bytes) 000299 1787461343.153241 trunc from 0x01d8d5 to 0x0022bb (0x1b619 bytes) 000300 1787461343.182887 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22bc): Mirror 4: 0xa53347bd (stale) Mirror 5: 0xa53347bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22bc): Mirror 4: 0xa53347bd (stale) Mirror 5: 0xa53347bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22bc: skipped 000301 1787461343.611104 mapwrite 0x013fc3 thru 0x01970d (0x574b bytes) 000302 1787461343.794255 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1970e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1970e): Mirror 4: 0xe18d61ac (stale) Mirror 5: 0x70fce523 (stale) CRC-32 checksum value for chunk [0, 0x1970e): Mirror 1: 0xfa06ae70 Mirror 6: 0xfa06ae70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1970e: skipped 000303 1787461344.870684 mapwrite 0x003288 thru 0x00c2c1 (0x903a bytes) 000304 1787461345.112648 trunc from 0x01970e to 0x011794 (0x7f79 bytes) 000305 1787461345.148460 read 0x00d79c thru 0x011794 (0x3ff9 bytes) 000307 1787461345.157265 trunc from 0x011795 to 0x011625 (0x16f bytes) 000309 1787461345.179200 write 0x021f20 thru 0x0262c3 (0x43a4 bytes) 000310 1787461345.180407 trunc from 0x0262c4 to 0x039c90 (0x139cd bytes) 000311 1787461345.201764 mapwrite 0x033ea8 thru 0x0348b3 (0xa0c bytes) 000312 1787461345.274766 mapwrite 0x00fbe4 thru 0x015511 (0x592e bytes) 000319 1787461345.374314 write 0x034bd4 thru 0x03ffff (0xb42c bytes) 000320 1787461345.376982 trunc from 0x040000 to 0x01f5d5 (0x20a2a bytes) 000324 1787461345.411900 trunc from 0x01f5d6 to 0x01090f (0xecc6 bytes) 000326 1787461345.443362 write 0x00f30a thru 0x0119d8 (0x26cf bytes) 000328 1787461345.445636 write 0x00d512 thru 0x01ab2e (0xd61d bytes) 000329 1787461345.448375 mapwrite 0x00f073 thru 0x018994 (0x9922 bytes) 000331 1787461345.510226 mapread 0x017ffb thru 0x01ab2e (0x2b34 bytes) 000332 1787461345.511296 write 0x028549 thru 0x02fab6 (0x756e bytes) 000334 1787461345.526227 mapread 0x006140 thru 0x0116c7 (0xb588 bytes) 000336 1787461345.529725 trunc from 0x02fab7 to 0x0070c7 (0x289ef bytes) 000337 1787461345.564403 write 0x018a6d thru 0x027cfa (0xf28e bytes) 000341 1787461345.573650 write 0x02b44f thru 0x038931 (0xd4e3 bytes) 000343 1787461345.584220 mapwrite 0x00c220 thru 0x018b23 (0xc904 bytes) 000344 1787461345.703095 write 0x03f4c5 thru 0x03ffff (0xb3b bytes) 000345 1787461345.709775 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x59d327c0 (stale) Mirror 6: 0xb96d9386 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x59d327c0 (stale) Mirror 6: 0xb96d9386 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000347 1787461346.037733 mapwrite 0x001a9e thru 0x002cd3 (0x1236 bytes) 000349 1787461346.109707 write 0x02a0bc thru 0x02b3db (0x1320 bytes) 000350 1787461346.120821 trunc from 0x040000 to 0x008dc7 (0x37238 bytes) 000356 1787461346.152756 mapwrite 0x036593 thru 0x03c43a (0x5ea8 bytes) 000358 1787461346.337528 write 0x01b079 thru 0x027c7d (0xcc05 bytes) 000359 1787461346.354789 trunc from 0x03c43b to 0x03725a (0x51e0 bytes) 000361 1787461346.388249 trunc from 0x03725b to 0x034a16 (0x2844 bytes) 000362 1787461346.433248 mapwrite 0x03bd06 thru 0x03ffff (0x42fa bytes) 000364 1787461346.572538 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x59d327c0 (stale) Mirror 6: 0xb96d9386 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x17fe0e17 Mirror 7: 0x17fe0e17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000369 1787461347.888488 write 0x0246fd thru 0x02edf2 (0xa6f6 bytes) 000370 1787461347.981225 read 0x03d9fc thru 0x03ffff (0x2604 bytes) 000372 1787461347.989289 mapwrite 0x027f95 thru 0x02d47d (0x54e9 bytes) 000374 1787461348.059721 mapread 0x00ce64 thru 0x017bdc (0xad79 bytes) 000375 1787461348.090664 trunc from 0x040000 to 0x016454 (0x29bab bytes) 000377 1787461348.117053 trunc from 0x016455 to 0x00de9b (0x85b9 bytes) 000378 1787461348.136840 mapwrite 0x03f706 thru 0x03ffff (0x8fa bytes) 000381 1787461348.242985 mapread 0x01a4c6 thru 0x026c6e (0xc7a9 bytes) 000384 1787461348.286301 mapread 0x00dc13 thru 0x01b785 (0xdb73 bytes) 000386 1787461348.300109 mapread 0x00c009 thru 0x00fed1 (0x3ec9 bytes) 000387 1787461348.302527 mapwrite 0x00d26b thru 0x013275 (0x600b bytes) 000388 1787461348.375682 write 0x01af6f thru 0x01fef9 (0x4f8b bytes) 000392 1787461348.379226 write 0x0049a9 thru 0x00cb10 (0x8168 bytes) 000403 1787461348.391183 mapread 0x03227d thru 0x03af22 (0x8ca6 bytes) 000404 1787461348.398522 mapwrite 0x009a2e thru 0x013200 (0x97d3 bytes) 000405 1787461348.449981 read 0x002a75 thru 0x003df7 (0x1383 bytes) 000406 1787461348.455962 write 0x032bc5 thru 0x03ffff (0xd43b bytes) 000408 1787461348.458122 write 0x0222d6 thru 0x02dd30 (0xba5b bytes) 000409 1787461348.462132 trunc from 0x040000 to 0x01b344 (0x24cbb bytes) 000411 1787461348.493266 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b345): Mirror 5: 0x37ff8ba7 (stale) Mirror 6: 0x728e53fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b345): Mirror 5: 0x37ff8ba7 (stale) Mirror 6: 0x728e53fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b345: skipped 000413 1787461348.861968 MIRROR_RESYNC 3 mirrors 000414 1787461350.245942 mapread 0x01340a thru 0x013597 (0x18e bytes) 000415 1787461350.256862 MIRROR_RESYNC 3 mirrors 000416 1787461350.657987 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000418 1787461350.988742 MIRROR_RESYNC 2 mirrors 000420 1787461351.427306 MIRROR_RESYNC 2 mirrors 000423 1787461351.759220 mapwrite 0x01e750 thru 0x02923a (0xaaeb bytes) 000424 1787461351.952987 mapwrite 0x0186b2 thru 0x01fb85 (0x74d4 bytes) 000426 1787461352.022469 trunc from 0x02923b to 0x017516 (0x11d24 bytes) 000429 1787461352.055579 read 0x00b4b7 thru 0x011dbf (0x6909 bytes) 000430 1787461352.063254 trunc from 0x017517 to 0x017e10 (0x8fa bytes) 000432 1787461352.090335 trunc from 0x017e11 to 0x02f141 (0x17331 bytes) 000435 1787461352.134757 MIRROR_RESYNC 2 mirrors 000436 1787461352.804003 read 0x00aecd thru 0x013f5f (0x9093 bytes) 000438 1787461352.821579 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000440 1787461352.969901 trunc from 0x02f142 to 0x012cc1 (0x1c480 bytes) 000441 1787461353.009029 read 0x00f05e thru 0x012cc1 (0x3c64 bytes) 000442 1787461353.014159 read 0x0082ce thru 0x00c51b (0x424e bytes) 000443 1787461353.021001 write 0x01c97f thru 0x022c9e (0x6320 bytes) 000446 1787461353.024053 trunc from 0x022c9f to 0x01cab2 (0x61ec bytes) 000447 1787461353.049174 read 0x0058e5 thru 0x011480 (0xbb9c bytes) 000451 1787461353.055477 mapwrite 0x038e5c thru 0x03ffff (0x71a4 bytes) 000453 1787461353.148348 trunc from 0x040000 to 0x01a61b (0x259e4 bytes) 000454 1787461353.169750 trunc from 0x01a61c to 0x00ae57 (0xf7c4 bytes) 000456 1787461353.190920 mapread 0x007817 thru 0x00ae57 (0x3641 bytes) 000461 1787461353.200101 MIRROR_EXTEND to 2 mirrors 000463 1787461353.854631 MIRROR_EXTEND to 3 mirrors 000465 1787461354.599767 read 0x00a162 thru 0x00ae57 (0xcf6 bytes) 000466 1787461354.608105 mapwrite 0x007309 thru 0x008dd9 (0x1ad1 bytes) 000467 1787461354.791274 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xae58): Mirror 3: 0x531f04f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xae58): Mirror 3: 0x531f04f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xae58: skipped 000468 1787461355.200036 read 0x00622a thru 0x00ae57 (0x4c2e bytes) 000470 1787461355.218618 mapread 0x006ed9 thru 0x00ae57 (0x3f7f bytes) 000471 1787461355.219576 write 0x00762a thru 0x01123b (0x9c12 bytes) 000472 1787461355.221421 mapread 0x0021be thru 0x007c26 (0x5a69 bytes) 000473 1787461355.236469 write 0x02864f thru 0x029442 (0xdf4 bytes) 000474 1787461355.237484 mapwrite 0x03f7e6 thru 0x03ffff (0x81a bytes) 000475 1787461355.361659 trunc from 0x040000 to 0x03c45f (0x3ba0 bytes) 000477 1787461355.401167 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000479 1787461355.576749 mapwrite 0x00afd7 thru 0x013731 (0x875b bytes) 000482 1787461355.677462 mapwrite 0x007797 thru 0x0126c0 (0xaf2a bytes) 000484 1787461355.737232 mapwrite 0x02652f thru 0x02b32c (0x4dfe bytes) 000485 1787461355.815371 write 0x02e114 thru 0x03c3e6 (0xe2d3 bytes) 000486 1787461355.823536 trunc from 0x03c460 to 0x026f5a (0x15505 bytes) 000489 1787461355.860603 read 0x01ca30 thru 0x026f5a (0xa52b bytes) 000491 1787461355.874189 MIRROR_EXTEND to 2 mirrors 000493 1787461356.755996 mapread 0x00500e thru 0x00782d (0x2820 bytes) 000495 1787461356.800634 mapread 0x01187c thru 0x01a5c4 (0x8d49 bytes) 000496 1787461356.827472 write 0x011f97 thru 0x01e620 (0xc68a bytes) 000497 1787461356.934788 write 0x019667 thru 0x026a5c (0xd3f6 bytes) 000499 1787461356.941477 write 0x029bf2 thru 0x037bd0 (0xdfdf bytes) 000504 1787461356.946411 mapwrite 0x01264c thru 0x017ea8 (0x585d bytes) 000506 1787461357.029967 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 2: 0xd98ec410 (stale) CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 1: 0x2b4faa6a Mirror 3: 0x2b4faa6a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bd1: skipped 000507 1787461357.918527 mapread 0x01f2e1 thru 0x02b507 (0xc227 bytes) 000508 1787461357.948752 MIRROR_RESYNC 3 mirrors 000509 1787461358.955920 mapread 0x002b6e thru 0x00f85f (0xccf2 bytes) 000510 1787461359.009502 MIRROR_RESYNC 3 mirrors 000511 1787461359.545225 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000515 1787461359.959524 mapread 0x00341a thru 0x005371 (0x1f58 bytes) 000516 1787461359.988538 write 0x00c1dc thru 0x010599 (0x43be bytes) 000518 1787461360.102455 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 3: 0x2b4faa6a (stale) CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 1: 0x68ce5007 Mirror 4: 0x68ce5007 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bd1: skipped 000519 1787461361.007992 mapwrite 0x01486d thru 0x01d5be (0x8d52 bytes) 000521 1787461361.217161 mapwrite 0x01763f thru 0x01b0d1 (0x3a93 bytes) 000522 1787461361.277875 mapwrite 0x027d29 thru 0x037356 (0xf62e bytes) 000524 1787461361.342034 mapread 0x00a263 thru 0x016f57 (0xccf5 bytes) 000526 1787461361.351344 read 0x012433 thru 0x01a96d (0x853b bytes) 000527 1787461361.352461 write 0x0201e0 thru 0x027e19 (0x7c3a bytes) 000528 1787461361.353698 mapread 0x00dd55 thru 0x012159 (0x4405 bytes) 000529 1787461361.354935 write 0x00f902 thru 0x010832 (0xf31 bytes) 000530 1787461361.356119 mapread 0x01dfda thru 0x021b8f (0x3bb6 bytes) 000531 1787461361.362418 mapread 0x01863c thru 0x018d5f (0x724 bytes) 000534 1787461361.364201 read 0x01b0f6 thru 0x027d40 (0xcc4b bytes) 000535 1787461361.365495 mapwrite 0x00edfd thru 0x01cba9 (0xddad bytes) 000536 1787461361.424050 read 0x018b78 thru 0x01ccd2 (0x415b bytes) 000537 1787461361.425299 write 0x00ad9d thru 0x01291c (0x7b80 bytes) 000538 1787461361.427761 mapwrite 0x00c520 thru 0x0148b1 (0x8392 bytes) 000539 1787461361.470914 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 4: 0x68ce5007 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bd1): Mirror 4: 0x68ce5007 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bd1: skipped 000540 1787461361.791913 mapwrite 0x03835c thru 0x03cf3e (0x4be3 bytes) 000541 1787461361.917504 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000543 1787461362.123208 trunc from 0x03cf3f to 0x03713c (0x5e02 bytes) 000545 1787461362.151416 mapwrite 0x03c141 thru 0x03d967 (0x1827 bytes) 000547 1787461362.261712 write 0x02c345 thru 0x0399fe (0xd6ba bytes) 000548 1787461362.280756 read 0x003f7a thru 0x00e8f2 (0xa979 bytes) 000551 1787461362.294053 mapread 0x028f93 thru 0x030f36 (0x7fa4 bytes) 000552 1787461362.307532 read 0x021613 thru 0x02c3b0 (0xad9e bytes) 000553 1787461362.309081 trunc from 0x03d968 to 0x010547 (0x2d420 bytes) 000556 1787461362.339339 trunc from 0x010548 to 0x039a8f (0x29548 bytes) 000558 1787461362.376602 mapread 0x00968c thru 0x0139f2 (0xa367 bytes) 000561 1787461362.393488 mapread 0x015753 thru 0x01f17a (0x9a28 bytes) 000562 1787461362.397042 read 0x0368c7 thru 0x039a8f (0x31c9 bytes) 000566 1787461362.399413 read 0x008499 thru 0x0130c7 (0xac2f bytes) 000567 1787461362.401199 trunc from 0x039a90 to 0x001c18 (0x37e77 bytes) 000568 1787461362.441444 mapwrite 0x005a38 thru 0x008ad3 (0x309c bytes) 000571 1787461362.510761 write 0x00d937 thru 0x0100a4 (0x276e bytes) 000572 1787461362.511828 write 0x0182e0 thru 0x025206 (0xcf27 bytes) 000573 1787461362.514361 mapread 0x0209c7 thru 0x025206 (0x4840 bytes) 000574 1787461362.515517 trunc from 0x025207 to 0x022743 (0x2ac3 bytes) 000579 1787461362.544189 trunc from 0x022744 to 0x002e7b (0x1f8c8 bytes) 000580 1787461362.564754 mapwrite 0x0083e7 thru 0x013a9f (0xb6b9 bytes) 000581 1787461362.664445 mapwrite 0x0283b6 thru 0x036f0f (0xeb5a bytes) 000583 1787461362.768855 read 0x02511e thru 0x02875a (0x363d bytes) 000584 1787461362.769466 mapwrite 0x0396f6 thru 0x03d8cd (0x41d8 bytes) 000585 1787461362.857777 mapread 0x0194b1 thru 0x027afa (0xe64a bytes) 000586 1787461362.859471 write 0x01a3c8 thru 0x0203bf (0x5ff8 bytes) 000587 1787461362.861530 trunc from 0x03d8ce to 0x00aeef (0x329de bytes) 000591 1787461362.889506 write 0x01faf3 thru 0x02776e (0x7c7c bytes) 000592 1787461362.890979 read 0x0134b9 thru 0x0157de (0x2326 bytes) 000593 1787461362.902592 trunc from 0x02776f to 0x01101b (0x16753 bytes) 000594 1787461362.933763 write 0x008e56 thru 0x00fc1f (0x6dca bytes) 000595 1787461362.935454 trunc from 0x01101c to 0x00056c (0x10aaf bytes) 000597 1787461362.964333 mapread 0x00051d thru 0x00056c (0x50 bytes) 000598 1787461362.966420 mapwrite 0x02d253 thru 0x037077 (0x9e25 bytes) 000604 1787461363.096933 trunc from 0x037078 to 0x00d023 (0x2a054 bytes) 000605 1787461363.117200 read 0x00a94c thru 0x00d023 (0x26d8 bytes) 000606 1787461363.137752 MIRROR_EXTEND to 2 mirrors 000610 1787461364.050915 read 0x006877 thru 0x0095ab (0x2d35 bytes) 000611 1787461364.058496 mapwrite 0x00098f thru 0x0064b7 (0x5b29 bytes) 000613 1787461364.320552 MIRROR_RESYNC 2 mirrors 000614 1787461365.187535 mapread 0x00814a thru 0x00d023 (0x4eda bytes) 000615 1787461365.211058 mapread 0x003a27 thru 0x00d023 (0x95fd bytes) 000617 1787461365.212268 mapread 0x00b64c thru 0x00d023 (0x19d8 bytes) 000619 1787461365.213988 trunc from 0x00d024 to 0x018f46 (0xbf23 bytes) 000620 1787461365.316570 mapread 0x0013dd thru 0x003750 (0x2374 bytes) 000621 1787461365.336754 trunc from 0x018f47 to 0x000509 (0x18a3d bytes) 000622 1787461365.362287 write 0x01e6ec thru 0x02c83d (0xe152 bytes) 000624 1787461365.364556 trunc from 0x02c83e to 0x0314e5 (0x4ca8 bytes) 000625 1787461365.382075 mapwrite 0x0183c5 thru 0x01ee9a (0x6ad6 bytes) 000626 1787461365.431897 trunc from 0x0314e6 to 0x0064a4 (0x2b041 bytes) 000628 1787461365.459575 MIRROR_RESYNC 2 mirrors 000632 1787461366.416578 trunc from 0x0064a5 to 0x025d22 (0x1f87e bytes) 000633 1787461366.533257 write 0x006363 thru 0x007866 (0x1504 bytes) 000635 1787461366.555604 read 0x008c64 thru 0x01517c (0xc519 bytes) 000638 1787461366.569082 trunc from 0x025d23 to 0x00c968 (0x193ba bytes) 000639 1787461366.606517 write 0x008fca thru 0x01464b (0xb682 bytes) 000640 1787461366.612451 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1464c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1464c): Mirror 2: 0xb9db6f49 (stale) CRC-32 checksum value for chunk [0, 0x1464c): Mirror 1: 0x8fd4019e Mirror 3: 0x8fd4019e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1464c: skipped 000641 1787461367.525278 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1464c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1464c): Mirror 2: 0xb9db6f49 (stale) CRC-32 checksum value for chunk [0, 0x1464c): Mirror 1: 0x8fd4019e Mirror 3: 0x8fd4019e Mirror 4: 0x8fd4019e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1464c: skipped 000642 1787461368.508163 mapread 0x00f6ea thru 0x01464b (0x4f62 bytes) 000644 1787461368.547332 write 0x03a118 thru 0x03b496 (0x137f bytes) 000645 1787461368.734649 read 0x008f55 thru 0x017328 (0xe3d4 bytes) 000646 1787461368.753705 mapwrite 0x00d397 thru 0x01029b (0x2f05 bytes) 000647 1787461368.849420 trunc from 0x03b497 to 0x019ff3 (0x214a3 bytes) 000648 1787461368.906590 mapwrite 0x0102ee thru 0x018d67 (0x8a7a bytes) 000649 1787461368.979718 write 0x03a3f9 thru 0x03ffff (0x5c07 bytes) 000651 1787461368.991197 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc08f2ab1 (stale) Mirror 4: 0xc08f2ab1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc08f2ab1 (stale) Mirror 4: 0xc08f2ab1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000652 1787461369.313789 read 0x02832d thru 0x035917 (0xd5eb bytes) 000653 1787461369.324228 read 0x0275a7 thru 0x037579 (0xffd3 bytes) 000655 1787461369.346161 trunc from 0x040000 to 0x02803f (0x17fc0 bytes) 000656 1787461369.370963 mapread 0x008bfe thru 0x008e97 (0x29a bytes) 000659 1787461369.373431 trunc from 0x028040 to 0x01f191 (0x8eae bytes) 000660 1787461369.390606 write 0x039358 thru 0x03dbec (0x4895 bytes) 000663 1787461369.394116 mapread 0x027bd5 thru 0x02961d (0x1a49 bytes) 000664 1787461369.422734 read 0x023172 thru 0x0314c4 (0xe353 bytes) 000665 1787461369.450041 trunc from 0x03dbed to 0x034d6a (0x8e82 bytes) 000667 1787461369.476409 mapwrite 0x019168 thru 0x01ef63 (0x5dfc bytes) 000668 1787461369.545257 trunc from 0x034d6b to 0x031c71 (0x30f9 bytes) 000670 1787461369.564082 write 0x00becc thru 0x00fc2e (0x3d63 bytes) 000672 1787461369.566187 write 0x009725 thru 0x018bb3 (0xf48f bytes) 000676 1787461369.576708 mapwrite 0x01cca7 thru 0x02426b (0x75c5 bytes) 000677 1787461369.644705 MIRROR_RESYNC 3 mirrors 000678 1787461370.663684 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000679 1787461371.116933 MIRROR_RESYNC 2 mirrors 000681 1787461371.633787 mapread 0x0108ca thru 0x01c3ad (0xbae4 bytes) 000682 1787461371.675420 mapread 0x0197b0 thru 0x0290fa (0xf94b bytes) 000683 1787461371.686203 mapread 0x003c90 thru 0x00bc57 (0x7fc8 bytes) 000684 1787461371.695243 mapwrite 0x02892b thru 0x02d8e6 (0x4fbc bytes) 000688 1787461371.898765 read 0x01413e thru 0x01afe0 (0x6ea3 bytes) 000691 1787461371.916087 write 0x03e910 thru 0x03ffff (0x16f0 bytes) 000696 1787461371.943629 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000697 1787461372.181758 mapwrite 0x03a4af thru 0x03ffff (0x5b51 bytes) 000698 1787461372.349917 read 0x0339ed thru 0x038f56 (0x556a bytes) 000699 1787461372.356653 mapwrite 0x0328d5 thru 0x03b974 (0x90a0 bytes) 000700 1787461372.415270 MIRROR_EXTEND to 2 mirrors 000702 1787461373.046086 mapread 0x03e94a thru 0x03ffff (0x16b6 bytes) 000704 1787461373.082889 MIRROR_RESYNC 2 mirrors 000706 1787461373.581573 trunc from 0x040000 to 0x00e8f6 (0x31709 bytes) 000709 1787461373.723006 trunc from 0x00e8f7 to 0x002840 (0xc0b6 bytes) 000710 1787461373.758960 trunc from 0x002841 to 0x01a51a (0x17cda bytes) 000712 1787461373.804215 read 0x0146fe thru 0x01a51a (0x5e1d bytes) 000713 1787461373.833696 read 0x00e5da thru 0x014e64 (0x688b bytes) 000714 1787461373.856684 mapread 0x00eab7 thru 0x0169fb (0x7f45 bytes) 000715 1787461373.857524 write 0x00bb7a thru 0x013d47 (0x81ce bytes) 000717 1787461373.859073 read 0x015e5f thru 0x01917a (0x331c bytes) 000718 1787461373.861538 trunc from 0x01a51b to 0x024454 (0x9f3a bytes) 000720 1787461373.876657 mapwrite 0x009e7a thru 0x014e7f (0xb006 bytes) 000721 1787461373.960573 read 0x010c38 thru 0x01f433 (0xe7fc bytes) 000722 1787461373.973135 trunc from 0x024455 to 0x004ad9 (0x1f97b bytes) 000723 1787461373.992712 mapwrite 0x0336f4 thru 0x033732 (0x3f bytes) 000725 1787461374.063497 mapwrite 0x02607b thru 0x02ffa1 (0x9f27 bytes) 000727 1787461374.139537 read 0x01139f thru 0x014c9a (0x38fc bytes) 000728 1787461374.155501 mapwrite 0x01c47f thru 0x01ea43 (0x25c5 bytes) 000730 1787461374.203312 MIRROR_RESYNC 2 mirrors 000731 1787461375.087020 read 0x020362 thru 0x0246ac (0x434b bytes) 000732 1787461375.101262 trunc from 0x033733 to 0x02eb4e (0x4be4 bytes) 000733 1787461375.318363 read 0x006a64 thru 0x009a76 (0x3013 bytes) 000734 1787461375.345279 trunc from 0x02eb4f to 0x00d3f3 (0x2175b bytes) 000735 1787461375.385918 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000737 1787461375.542246 write 0x032fc9 thru 0x0364e1 (0x3519 bytes) 000738 1787461375.543458 mapread 0x01459d thru 0x01a04a (0x5aae bytes) 000741 1787461375.563048 mapwrite 0x003019 thru 0x00b0df (0x80c7 bytes) 000747 1787461375.623657 write 0x0214ca thru 0x02e59c (0xd0d3 bytes) 000748 1787461375.632650 mapread 0x0235b3 thru 0x03188c (0xe2da bytes) 000750 1787461375.638776 trunc from 0x0364e2 to 0x02f786 (0x6d5b bytes) 000752 1787461375.653512 trunc from 0x02f787 to 0x01b97b (0x13e0b bytes) 000755 1787461375.672737 write 0x02fafa thru 0x03150b (0x1a12 bytes) 000756 1787461375.675108 write 0x03ae59 thru 0x03ffff (0x51a7 bytes) 000757 1787461375.677292 trunc from 0x040000 to 0x00b256 (0x34da9 bytes) 000759 1787461375.693348 mapwrite 0x0027e0 thru 0x011fcb (0xf7ec bytes) 000761 1787461375.777516 MIRROR_EXTEND to 2 mirrors 000762 1787461376.649174 write 0x01a075 thru 0x01b123 (0x10af bytes) 000763 1787461376.772450 mapread 0x010946 thru 0x01b123 (0xa7de bytes) 000765 1787461376.796498 trunc from 0x01b124 to 0x00b2fe (0xfe25 bytes) 000766 1787461376.820724 read 0x0085e9 thru 0x00b2fe (0x2d16 bytes) 000767 1787461376.832624 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xb2ff): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb2ff): Mirror 2: 0xe2ecc78a (stale) CRC-32 checksum value for chunk [0, 0xb2ff): Mirror 1: 0xe2ecc78a Mirror 3: 0xe2ecc78a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb2ff: skipped 000769 1787461377.626023 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xb2ff): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb2ff): Mirror 2: 0xe2ecc78a (stale) CRC-32 checksum value for chunk [0, 0xb2ff): Mirror 1: 0xe2ecc78a Mirror 3: 0xe2ecc78a Mirror 4: 0xe2ecc78a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb2ff: skipped 000770 1787461378.917535 MIRROR_RESYNC 4 mirrors 000771 1787461379.882147 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000772 1787461380.318759 read 0x009f28 thru 0x00b2fe (0x13d7 bytes) 000773 1787461380.324420 mapread 0x0064fe thru 0x00b2fe (0x4e01 bytes) 000775 1787461380.346471 trunc from 0x00b2ff to 0x00a4fb (0xe03 bytes) 000777 1787461380.458223 mapwrite 0x0228dd thru 0x02537b (0x2a9f bytes) 000778 1787461380.558422 write 0x018d1d thru 0x0199c1 (0xca5 bytes) 000780 1787461380.568936 mapread 0x006ea1 thru 0x011cf4 (0xae54 bytes) 000781 1787461380.585874 read 0x00ee12 thru 0x0143cb (0x55ba bytes) 000783 1787461380.592403 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2537c): Mirror 2: 0xe6d0e483 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2537c): Mirror 2: 0xe6d0e483 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2537c: skipped 000784 1787461380.960601 mapread 0x0027e7 thru 0x009996 (0x71b0 bytes) 000785 1787461381.001236 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000786 1787461381.175606 mapwrite 0x004d5e thru 0x00b86c (0x6b0f bytes) 000787 1787461381.317395 trunc from 0x02537c to 0x013d8a (0x115f1 bytes) 000790 1787461381.361540 write 0x02eb92 thru 0x030315 (0x1784 bytes) 000793 1787461381.363401 trunc from 0x030316 to 0x00a298 (0x2607d bytes) 000794 1787461381.393752 write 0x011eba thru 0x01d5b2 (0xb6f9 bytes) 000795 1787461381.396068 mapwrite 0x03e310 thru 0x03f657 (0x1348 bytes) 000798 1787461381.575374 trunc from 0x03f658 to 0x03d9c8 (0x1c8f bytes) 000799 1787461381.605206 mapwrite 0x007ea0 thru 0x00add9 (0x2f3a bytes) 000801 1787461381.707555 mapwrite 0x010ddc thru 0x0160e7 (0x530c bytes) 000803 1787461381.837221 mapread 0x00dd51 thru 0x00eeb3 (0x1163 bytes) 000804 1787461381.838578 read 0x0099cf thru 0x00bdb1 (0x23e3 bytes) 000808 1787461381.842413 mapread 0x0075c6 thru 0x00ea0b (0x7446 bytes) 000809 1787461381.844053 mapread 0x007970 thru 0x015149 (0xd7da bytes) 000811 1787461381.845974 trunc from 0x03d9c9 to 0x01f363 (0x1e665 bytes) 000813 1787461381.865066 mapwrite 0x03a15c thru 0x03ffff (0x5ea4 bytes) 000814 1787461381.963047 trunc from 0x040000 to 0x000d1b (0x3f2e4 bytes) 000815 1787461381.986880 mapwrite 0x011e44 thru 0x0219bd (0xfb7a bytes) 000816 1787461382.125170 mapwrite 0x03b8d8 thru 0x03ffff (0x4728 bytes) 000817 1787461382.254955 mapread 0x0114fb thru 0x01c5f3 (0xb0f9 bytes) 000818 1787461382.263143 mapwrite 0x026a64 thru 0x0346dd (0xdc7a bytes) 000820 1787461382.436058 write 0x02cc54 thru 0x039bd0 (0xcf7d bytes) 000821 1787461382.437510 trunc from 0x040000 to 0x00375e (0x3c8a1 bytes) 000823 1787461382.460524 write 0x000dab thru 0x00355c (0x27b2 bytes) 000824 1787461382.478538 mapread 0x003472 thru 0x00375e (0x2ed bytes) 000827 1787461382.482729 trunc from 0x00375f to 0x01c98f (0x19231 bytes) 000829 1787461382.517833 mapwrite 0x005593 thru 0x011ea4 (0xc912 bytes) 000830 1787461382.621762 trunc from 0x01c990 to 0x037d84 (0x1b3f5 bytes) 000831 1787461382.654282 MIRROR_EXTEND to 2 mirrors 000832 1787461383.745324 write 0x024c79 thru 0x02e286 (0x960e bytes) 000833 1787461383.832426 read 0x0306aa thru 0x037d84 (0x76db bytes) 000834 1787461383.845255 write 0x029da3 thru 0x0333f0 (0x964e bytes) 000835 1787461383.846669 mapwrite 0x03fe5f thru 0x03ffff (0x1a1 bytes) 000836 1787461383.946608 MIRROR_RESYNC 2 mirrors 000837 1787461384.980332 read 0x025492 thru 0x0285f5 (0x3164 bytes) 000838 1787461384.985401 mapwrite 0x01182e thru 0x019692 (0x7e65 bytes) 000839 1787461385.212309 mapwrite 0x013b0f thru 0x01a769 (0x6c5b bytes) 000841 1787461385.313859 read 0x0234b1 thru 0x0284e0 (0x5030 bytes) 000842 1787461385.339531 trunc from 0x040000 to 0x004aae (0x3b551 bytes) 000843 1787461385.392916 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000844 1787461385.626769 mapwrite 0x020400 thru 0x02fe03 (0xfa04 bytes) 000845 1787461385.794676 mapread 0x004bec thru 0x00f7de (0xabf3 bytes) 000846 1787461385.824667 read 0x00f518 thru 0x01c5eb (0xd0d4 bytes) 000849 1787461385.843462 read 0x016537 thru 0x025130 (0xebfa bytes) 000850 1787461385.844374 mapwrite 0x01a700 thru 0x0204bc (0x5dbd bytes) 000852 1787461385.938585 mapread 0x009868 thru 0x016c98 (0xd431 bytes) 000857 1787461385.948998 write 0x03fffa thru 0x03ffff (0x6 bytes) 000858 1787461385.951032 mapread 0x01335f thru 0x01aace (0x7770 bytes) 000860 1787461385.958306 read 0x03be84 thru 0x03ffff (0x417c bytes) 000862 1787461385.987765 write 0x009df4 thru 0x00eaec (0x4cf9 bytes) 000865 1787461386.000725 write 0x01832f thru 0x0221df (0x9eb1 bytes) 000866 1787461386.003980 read 0x028a04 thru 0x02bac4 (0x30c1 bytes) 000867 1787461386.005251 mapwrite 0x00ad13 thru 0x0126f2 (0x79e0 bytes) 000869 1787461386.156324 trunc from 0x040000 to 0x0343f5 (0xbc0a bytes) 000871 1787461386.177040 trunc from 0x0343f6 to 0x00a1df (0x2a216 bytes) 000872 1787461386.199444 write 0x006eeb thru 0x01413b (0xd251 bytes) 000876 1787461386.213739 mapread 0x0074c7 thru 0x008869 (0x13a3 bytes) 000879 1787461386.216474 mapwrite 0x0114f4 thru 0x01fd95 (0xe8a2 bytes) 000881 1787461386.314189 mapread 0x01f092 thru 0x01fd95 (0xd04 bytes) 000882 1787461386.316307 read 0x007da6 thru 0x01455e (0xc7b9 bytes) 000883 1787461386.317138 mapwrite 0x004246 thru 0x005594 (0x134f bytes) 000885 1787461386.362882 write 0x016506 thru 0x0252b3 (0xedae bytes) 000886 1787461386.366226 write 0x03040b thru 0x037410 (0x7006 bytes) 000889 1787461386.370641 write 0x0099ad thru 0x0153f8 (0xba4c bytes) 000890 1787461386.372498 read 0x0348b7 thru 0x037410 (0x2b5a bytes) 000891 1787461386.374025 read 0x01e602 thru 0x02d9bd (0xf3bc bytes) 000893 1787461386.384327 read 0x02e6e0 thru 0x02ecbe (0x5df bytes) 000896 1787461386.387038 trunc from 0x037411 to 0x01cbf0 (0x1a820 bytes) 000897 1787461386.416836 trunc from 0x01cbf1 to 0x026735 (0x9b45 bytes) 000898 1787461386.455735 read 0x022b4c thru 0x026735 (0x3bea bytes) 000900 1787461386.482170 mapread 0x014d2e thru 0x015bdd (0xeb0 bytes) 000901 1787461386.498945 MIRROR_EXTEND to 2 mirrors 000902 1787461387.648455 mapread 0x0124cb thru 0x013efb (0x1a31 bytes) 000903 1787461387.665664 write 0x03f7a5 thru 0x03ffff (0x85b bytes) 000906 1787461387.782414 trunc from 0x040000 to 0x002838 (0x3d7c7 bytes) 000907 1787461387.843141 write 0x012837 thru 0x01c41a (0x9be4 bytes) 000908 1787461387.852447 trunc from 0x01c41b to 0x035a2f (0x19615 bytes) 000909 1787461387.882635 trunc from 0x035a30 to 0x017155 (0x1e8da bytes) 000912 1787461387.940375 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000913 1787461388.149770 mapread 0x015b62 thru 0x017155 (0x15f4 bytes) 000914 1787461388.179353 mapread 0x004dc2 thru 0x00f68d (0xa8cc bytes) 000920 1787461388.196387 write 0x02fe52 thru 0x03190a (0x1ab9 bytes) 000921 1787461388.198561 mapread 0x00dfd6 thru 0x00ea0d (0xa38 bytes) 000925 1787461388.211612 MIRROR_EXTEND to 2 mirrors 000926 1787461389.329495 write 0x018cbc thru 0x01ec28 (0x5f6d bytes) 000929 1787461389.430140 mapwrite 0x01754a thru 0x021e4f (0xa906 bytes) 000930 1787461389.509200 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000935 1787461389.731964 mapread 0x030d8f thru 0x03190a (0xb7c bytes) 000937 1787461389.753248 write 0x03807a thru 0x03d201 (0x5188 bytes) 000938 1787461389.755579 write 0x02ff5b thru 0x030d12 (0xdb8 bytes) 000939 1787461389.761674 read 0x02ac14 thru 0x0351dc (0xa5c9 bytes) 000941 1787461389.789901 write 0x01ee79 thru 0x02501c (0x61a4 bytes) 000943 1787461389.809644 write 0x01e18e thru 0x02d810 (0xf683 bytes) 000944 1787461389.816206 read 0x039db5 thru 0x03d201 (0x344d bytes) 000947 1787461389.824497 read 0x00c0bd thru 0x016588 (0xa4cc bytes) 000948 1787461389.855501 read 0x006e43 thru 0x00ce81 (0x603f bytes) 000951 1787461389.866289 read 0x03bfc5 thru 0x03d201 (0x123d bytes) 000952 1787461389.867003 mapwrite 0x018794 thru 0x024180 (0xb9ed bytes) 000954 1787461389.958702 write 0x005138 thru 0x00e106 (0x8fcf bytes) 000956 1787461389.961609 read 0x027528 thru 0x02d009 (0x5ae2 bytes) 000957 1787461389.962463 trunc from 0x03d202 to 0x036664 (0x6b9d bytes) 000959 1787461389.998930 mapread 0x025891 thru 0x03130b (0xba7b bytes) 000961 1787461390.001717 mapwrite 0x01662f thru 0x016647 (0x19 bytes) 000962 1787461390.052034 write 0x038521 thru 0x03ffff (0x7adf bytes) 000965 1787461390.058344 read 0x008c85 thru 0x011ff9 (0x9375 bytes) 000967 1787461390.059670 trunc from 0x040000 to 0x02c657 (0x139a8 bytes) 000970 1787461390.098021 MIRROR_EXTEND to 2 mirrors 000972 1787461390.944702 read 0x01a60f thru 0x024ef0 (0xa8e2 bytes) 000973 1787461390.956415 trunc from 0x02c658 to 0x02daed (0x1496 bytes) 000974 1787461391.104580 write 0x015a08 thru 0x02283c (0xce35 bytes) 000975 1787461391.149518 trunc from 0x02daee to 0x0266f2 (0x73fb bytes) 000976 1787461391.199950 write 0x036dee thru 0x03fa8e (0x8ca1 bytes) 000978 1787461391.211684 trunc from 0x03fa8f to 0x027c03 (0x17e8b bytes) 000979 1787461391.256378 mapread 0x00cfc5 thru 0x00d09e (0xda bytes) 000980 1787461391.282518 MIRROR_RESYNC 2 mirrors 000983 1787461391.916073 mapread 0x023da8 thru 0x027c03 (0x3e5c bytes) 000987 1787461391.956045 mapread 0x005e89 thru 0x0148fa (0xea72 bytes) 000988 1787461391.985302 write 0x00280a thru 0x003603 (0xdfa bytes) 000991 1787461392.093212 read 0x00a2f9 thru 0x00ede1 (0x4ae9 bytes) 000992 1787461392.111507 write 0x020d13 thru 0x0250e3 (0x43d1 bytes) 000993 1787461392.126217 read 0x010c69 thru 0x01dd09 (0xd0a1 bytes) 000995 1787461392.137391 write 0x032345 thru 0x03abd3 (0x888f bytes) 000997 1787461392.141045 mapwrite 0x01c5b8 thru 0x01e777 (0x21c0 bytes) 000998 1787461392.200372 MIRROR_RESYNC 2 mirrors All operations completed A-OK! PASS 70a (84s) == sanity-flr test 71: check mirror extend parent fid ==== 01:03:18 (1787461398) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 01:03:21 (1787461401) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.210446 s, 10.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219116 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x292:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x292:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 01:03:30 (1787461410) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5212992 .. Extending file size to 2313056 .. Extending file size to 5102688 .. Extending file size to 9136224 .. Extending file size to 6077216 .. Extending file size to 8520832 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8938496 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9237088 .. Extending file size to 9167904 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57265 57266 57269 57271 57272 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) 192.168.203.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:04:50 (1787461490) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7138592 .. Extending file size to 2086464 .. failed Extending file size to 8542592 .. Extending file size to 7698272 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8676448 .. Extending file size to 8579456 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8624064 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9255648 .. Extending file size to 8702656 .. Waiting 58658 58659 58660 58663 58666 done 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) 192.168.203.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:06:12 (1787461572) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18841 s, 11.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.407965 s, 5.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x13c893a7 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:06:32 (1787461592) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0744029 s, 28.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 01:06:39 (1787461599) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:06:47 (1787461607) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272325 s, 7.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:06:55 (1787461615) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194062 s, 10.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:07:03 (1787461623) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218888 s, 9.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:07:36 (1787461656) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114848 s, 9.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:08:36 (1787461716) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255511 s, 8.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:09:35 (1787461775) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg307-server mds-ost sync done. Creating new pool test_204f oleg307-server: Pool lustre.test_204f created Adding targets to pool oleg307-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg307-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242759 s, 8.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg307-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg307-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg307-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:10:53 (1787461853) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:11:01 (1787461861) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:11:10 (1787461870) oleg307-server: Pool lustre.test_206 created Adding targets to pool oleg307-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg307-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg307-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg307-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg307-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:11:35 (1787461895) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662272 s, 15.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491249 s, 21.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8b8e0fd4 Mirror 3: 0x8b8e0fd4 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:11:48 (1787461908) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:11:52 (1787461912) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:11:55 (1787461915) Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.19112 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1420000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2430000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2430000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2430000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b90000 Read this (4 bytes): BBBB 192.168.203.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:12:37 (1787461957) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0220444 s, 372 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:12:45 (1787461965) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10207 s, 10.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e9 lmm_fid: [0x200000404:0x1e9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e9 lmm_fid: [0x200000404:0x1e9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834409 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-23 01:12:47.000000000 -0400 Modify: 2026-08-23 01:12:47.000000000 -0400 Change: 2026-08-23 01:12:48.000000000 -0400 Birth: 2026-08-23 01:12:46.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:12:54 (1787461974) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:12:58 (1787461978) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.703209 s, 58.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0498673 s, 82.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 23 01:13 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 23 01:13 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:13:07 (1787461987) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:13:14 (1787461994) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1887 sec =========== 01:13:23 (1787462003) === sanity-flr: start cleanup 01:13:25 (1787462005) === === sanity-flr: finish cleanup 01:13:32 (1787462012) ===