-----============= acceptance-small: sanity-flr ============----- Mon Aug 10 22:21:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg243-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:22:15 (1786414935) === oleg243-client.virtnet: executing check_config_client /mnt/lustre oleg243-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg243-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d916074f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d916074f000.idle_timeout=debug disable quota as required oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:22:55 (1786414975) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanity-flr test 0a: lfs mirror create with -N option == 22:22:58 (1786414978) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:23:16 (1786414996) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:23:20 (1786415000) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:23:24 (1786415004) PASS 0d (34s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:23:58 (1786415038) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:24:02 (1786415042) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 22:24:07 (1786415047) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9d916074f000.stats=clear osc.lustre-OST0001-osc-ffff9d916074f000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:24:16 (1786415056) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:24:29 (1786415069) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:24:39 (1786415079) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:24:43 (1786415083) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:24:52 (1786415092) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 22:25:02 (1786415102) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:25:10 (1786415110) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:25:13 (1786415113) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:25:22 (1786415122) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:25:33 (1786415133) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:25:41 (1786415141) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.50363 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227831 s, 4.6 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:25:53 (1786415153) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.120968 s, 52.0 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:26:02 (1786415162) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:26:13 (1786415173) Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg243-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (141s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:28:34 (1786415314) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.812971 s, 12.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:29:25 (1786415365) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (96s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:31:01 (1786415461) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.53168 s, 11.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 10 22:31 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.618s user 0m0.001s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d916074f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.620s user 0m0.001s sys 0m0.184s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg243-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg243-server oleg243-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.544s user 0m0.002s sys 0m0.209s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg243-server: oleg243-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg243-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg243-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d916074f000.ost_server_uuid in FULL state after 0 sec PASS 33b (112s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:32:53 (1786415573) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:32:58 (1786415578) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:33:02 (1786415582) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:33:06 (1786415586) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 22:33:14 (1786415594) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.65788 s, 19.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:33:30 (1786415610) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354833 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388619 s, 27.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:33:38 (1786415618) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:33:47 (1786415627) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00166559 s, 615 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00217745 s, 470 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105862 s, 967 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011407 s, 898 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 22:34:00 (1786415640) create test files with size 15731726 .. 9377aed1ddcc55bab93d76cc6faea140 - 243deebd693fca65ff2623e537656001 - e4c2a350468e58b3b765a5ed4a7bd317 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 22:34:27 (1786415667) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:34:42 (1786415682) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:34:49 (1786415689) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1476 s, 7.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251511 s, 41.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 22:34:59 (1786415699) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134962 s, 15.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.321713 s, 13.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186696 s, 11.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 22:35:17 (1786415717) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 22:35:21 (1786415721) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:35:26 (1786415726) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322815 s, 32.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 44a: lfs mirror split check =========== 22:35:37 (1786415737) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:35:41 (1786415741) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:35:53 (1786415753) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.346906 s, 30.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:36:08 (1786415768) llite.lustre-ffff9d914a7e0000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.33816 s, 31.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg243-client.virtnet /mnt/lustre (opts:) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9d9151e24800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:36:24 (1786415784) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:37:11 (1786415831) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (45s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:37:56 (1786415876) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 22:38:00 (1786415880) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.459636 s, 6.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.216926 s, 14.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4ad62549 (nosync,stale) Mirror 2: 0x4ad62549 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3806b7afbb9047bf716337a9872cfbda - mirror 1 checksum: 7ea95f2181904aece26b22e51735d612 - mirror 2 checksum: 3806b7afbb9047bf716337a9872cfbda - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T22:38:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-10T22:38:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3806b7afbb9047bf716337a9872cfbda - mirror 1 checksum: 7ea95f2181904aece26b22e51735d612 - mirror 2 checksum: 7ea95f2181904aece26b22e51735d612 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T22:38:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-10T22:38:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:38:15 (1786415895) Failing mds1 on oleg243-server Stopping /mnt/lustre-mds1 (opts:) on oleg243-server 22:38:24 (1786415904) shut down facet: mds1 facet_host: oleg243-server facet_failover_host: oleg243-server Failover mds1 to oleg243-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 22:38:58 (1786415938) targets are mounted 22:38:58 (1786415938) facet_failover done oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030865 s, 34.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:39:21 (1786415961) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:39:25 (1786415965) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:39:28 (1786415968) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00590296 s, 2.8 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:39:38 (1786415978) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:39:42 (1786415982) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00868936 s, 471 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0851647 s, 48.1 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:39:52 (1786415992) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:40:20 (1786416020) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:40:51 (1786416051) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:41:16 (1786416076) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:41:44 (1786416104) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (11s) == sanity-flr test 70: mirror create and split race ====== 22:41:55 (1786416115) mirror create pid 54341 mirror split pid 54343 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6991: 54343 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6991: 54343 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54341 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54343 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 22:43:03 (1786416183) Seed set to 6185 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000008 1786416185.803013 write 0x03efda thru 0x03f477 (0x49e bytes) 000009 1786416185.821568 read 0x004f5e thru 0x014c71 (0xfd14 bytes) truncating to largest ever: 0x9543 000010 1786416185.835650 trunc from 0x03f478 to 0x009542 (0x35f35 bytes) 000011 1786416185.864965 write 0x01b56d thru 0x027ae6 (0xc57a bytes) 000012 1786416185.868692 mapread 0x014178 thru 0x016a35 (0x28be bytes) 000013 1786416185.892516 write 0x036b45 thru 0x03f743 (0x8bff bytes) 000014 1786416185.908391 mapwrite 0x019109 thru 0x019ea3 (0xd9b bytes) truncating to largest ever: 0x2e6e9 000016 1786416185.966860 trunc from 0x03f744 to 0x02e6e8 (0x1105b bytes) 000017 1786416185.991679 mapwrite 0x0347e4 thru 0x03ffff (0xb81c bytes) 000019 1786416186.090206 read 0x020b7b thru 0x0248b1 (0x3d37 bytes) 000020 1786416186.090824 mapwrite 0x00c592 thru 0x0123c8 (0x5e37 bytes) truncating to largest ever: 0x3aeaa 000021 1786416186.163349 trunc from 0x040000 to 0x03aea9 (0x5156 bytes) 000022 1786416186.193018 write 0x018a95 thru 0x024700 (0xbc6c bytes) 000023 1786416186.195572 trunc from 0x03aeaa to 0x00f8fd (0x2b5ac bytes) 000024 1786416186.219747 mapread 0x0027d7 thru 0x00411d (0x1947 bytes) 000025 1786416186.224116 write 0x02f4da thru 0x03a164 (0xac8b bytes) 000026 1786416186.226293 mapread 0x000614 thru 0x00b7dd (0xb1ca bytes) 000028 1786416186.227824 write 0x025d2f thru 0x025d3e (0x10 bytes) 000033 1786416186.241627 write 0x03aab7 thru 0x03ffff (0x5549 bytes) 000034 1786416186.243624 write 0x00f4a7 thru 0x019305 (0x9e5f bytes) 000036 1786416186.250872 write 0x004251 thru 0x00fc67 (0xba17 bytes) 000037 1786416186.252896 read 0x005852 thru 0x00c3dc (0x6b8b bytes) 000039 1786416186.255125 read 0x018382 thru 0x020687 (0x8306 bytes) 000042 1786416186.265015 read 0x02aa20 thru 0x036f3a (0xc51b bytes) 000045 1786416186.270663 read 0x0192dd thru 0x023e94 (0xabb8 bytes) 000047 1786416186.271898 mapwrite 0x036a90 thru 0x0394f6 (0x2a67 bytes) 000048 1786416186.319854 trunc from 0x040000 to 0x03060a (0xf9f5 bytes) 000051 1786416186.356855 write 0x015ebf thru 0x016e08 (0xf4a bytes) 000052 1786416186.358337 mapwrite 0x020976 thru 0x0227fa (0x1e85 bytes) 000053 1786416186.412648 read 0x0301fd thru 0x030520 (0x324 bytes) 000054 1786416186.417047 MIRROR_EXTEND to 2 mirrors 000055 1786416187.021315 read 0x00fa6a thru 0x01506a (0x5601 bytes) 000056 1786416187.051376 trunc from 0x03060b to 0x0012b1 (0x2f359 bytes) 000059 1786416187.157889 mapread 0x001024 thru 0x0012b1 (0x28e bytes) 000061 1786416187.170790 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000063 1786416187.316014 trunc from 0x0012b2 to 0x03678f (0x354de bytes) 000066 1786416187.335393 trunc from 0x036790 to 0x00907c (0x2d713 bytes) 000067 1786416187.359863 write 0x039ce1 thru 0x03ffff (0x631f bytes) 000069 1786416187.362142 mapread 0x006751 thru 0x00c45e (0x5d0e bytes) 000070 1786416187.384886 trunc from 0x040000 to 0x030193 (0xfe6c bytes) 000072 1786416187.404245 trunc from 0x030194 to 0x0171f5 (0x18f9e bytes) 000073 1786416187.428450 trunc from 0x0171f6 to 0x026a42 (0xf84d bytes) 000074 1786416187.444916 mapwrite 0x001dd0 thru 0x005077 (0x32a8 bytes) 000075 1786416187.497790 mapwrite 0x033cd0 thru 0x03bb1c (0x7e4d bytes) 000076 1786416187.608872 MIRROR_EXTEND to 2 mirrors 000077 1786416188.844928 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000080 1786416189.097622 mapwrite 0x0014a6 thru 0x002845 (0x13a0 bytes) 000081 1786416189.272304 trunc from 0x03bb1d to 0x037dec (0x3d30 bytes) 000084 1786416189.315184 mapread 0x006705 thru 0x00aa6e (0x436a bytes) 000085 1786416189.344333 write 0x01ab7d thru 0x01b051 (0x4d5 bytes) 000087 1786416189.379763 trunc from 0x037ded to 0x019b1e (0x1e2ce bytes) 000088 1786416189.414065 mapwrite 0x039eec thru 0x03ffff (0x6114 bytes) 000089 1786416189.524317 trunc from 0x040000 to 0x002401 (0x3dbfe bytes) 000091 1786416189.569530 mapwrite 0x00d3f3 thru 0x010208 (0x2e16 bytes) 000097 1786416189.707689 trunc from 0x010209 to 0x018c32 (0x8a2a bytes) 000098 1786416189.735311 mapwrite 0x01a521 thru 0x02672c (0xc20c bytes) 000100 1786416189.866526 trunc from 0x02672d to 0x0174bf (0xf26d bytes) 000102 1786416189.887003 mapwrite 0x01c6a8 thru 0x01f8d7 (0x3230 bytes) 000104 1786416189.959183 mapwrite 0x0038fa thru 0x005d80 (0x2487 bytes) 000106 1786416190.004024 read 0x00b2a0 thru 0x018907 (0xd668 bytes) 000109 1786416190.011642 trunc from 0x01f8d8 to 0x036f4d (0x17676 bytes) 000111 1786416190.035999 trunc from 0x036f4e to 0x013ebb (0x23092 bytes) 000112 1786416190.056407 mapread 0x0120b7 thru 0x013ebb (0x1e05 bytes) 000114 1786416190.061173 read 0x00aeb5 thru 0x013ebb (0x9007 bytes) 000118 1786416190.065924 read 0x00c135 thru 0x013ebb (0x7d87 bytes) 000121 1786416190.067974 write 0x015e76 thru 0x0247d3 (0xe95e bytes) 000122 1786416190.073040 mapwrite 0x03cf49 thru 0x03ffff (0x30b7 bytes) 000124 1786416190.152321 trunc from 0x040000 to 0x036eba (0x9145 bytes) 000126 1786416190.173712 read 0x02ad4f thru 0x036eba (0xc16c bytes) 000127 1786416190.174910 mapread 0x0292af thru 0x033351 (0xa0a3 bytes) 000128 1786416190.176550 write 0x006c6a thru 0x007812 (0xba9 bytes) 000129 1786416190.178342 write 0x01cd24 thru 0x01d74c (0xa29 bytes) 000130 1786416190.184099 MIRROR_EXTEND to 2 mirrors 000131 1786416191.005879 read 0x015539 thru 0x0247d6 (0xf29e bytes) 000132 1786416191.040594 write 0x02fbaa thru 0x03aacb (0xaf22 bytes) 000133 1786416191.198209 mapwrite 0x01d9e7 thru 0x0253b7 (0x79d1 bytes) 000134 1786416191.325669 mapwrite 0x011596 thru 0x01d60d (0xc078 bytes) 000137 1786416191.439863 read 0x016987 thru 0x024dfa (0xe474 bytes) 000139 1786416191.441224 mapwrite 0x035686 thru 0x03f547 (0x9ec2 bytes) 000142 1786416191.648402 trunc from 0x03f548 to 0x034070 (0xb4d7 bytes) 000145 1786416191.671037 trunc from 0x034071 to 0x001ee8 (0x32188 bytes) truncating to largest ever: 0x3d858 000146 1786416191.706962 trunc from 0x001ee9 to 0x03d857 (0x3b96f bytes) 000148 1786416191.727919 mapread 0x00f5d1 thru 0x00fcc4 (0x6f4 bytes) 000149 1786416191.742781 mapwrite 0x016799 thru 0x0209f2 (0xa25a bytes) 000150 1786416191.972901 mapread 0x015765 thru 0x01cfaa (0x7846 bytes) 000151 1786416191.985012 read 0x027d16 thru 0x027d97 (0x82 bytes) 000152 1786416192.017491 read 0x01542a thru 0x019af4 (0x46cb bytes) 000153 1786416192.018316 read 0x00f675 thru 0x01a7cc (0xb158 bytes) 000155 1786416192.021252 mapread 0x010117 thru 0x015b47 (0x5a31 bytes) 000156 1786416192.037758 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d858): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d858): Mirror 2: 0x266e8bb1 (stale) CRC-32 checksum value for chunk [0, 0x3d858): Mirror 1: 0x272f9636 Mirror 3: 0x272f9636 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d858: skipped 000159 1786416193.387324 mapwrite 0x02bd3a thru 0x034dd0 (0x9097 bytes) 000161 1786416193.523219 mapwrite 0x038430 thru 0x03ffff (0x7bd0 bytes) 000162 1786416193.573945 write 0x028635 thru 0x033ab1 (0xb47d bytes) 000164 1786416193.575638 trunc from 0x040000 to 0x02ac36 (0x153c9 bytes) 000165 1786416193.588990 MIRROR_RESYNC 3 mirrors 000168 1786416194.731116 write 0x03679a thru 0x03d904 (0x716b bytes) 000169 1786416194.889861 write 0x03585d thru 0x0359cf (0x173 bytes) 000170 1786416194.902069 mapread 0x0146b4 thru 0x01c5a1 (0x7eee bytes) 000172 1786416194.955280 read 0x012e88 thru 0x0174b9 (0x4632 bytes) 000174 1786416194.959955 mapwrite 0x027f87 thru 0x02ec92 (0x6d0c bytes) 000177 1786416195.076621 trunc from 0x03d905 to 0x008f10 (0x349f4 bytes) 000178 1786416195.129991 mapread 0x006c65 thru 0x008f10 (0x22ac bytes) 000180 1786416195.157968 write 0x018bb2 thru 0x01e370 (0x57bf bytes) 000182 1786416195.165381 trunc from 0x01e371 to 0x01c0f1 (0x227f bytes) 000183 1786416195.199094 mapread 0x003783 thru 0x00e7a4 (0xb022 bytes) 000185 1786416195.220917 mapwrite 0x03ba20 thru 0x03ffff (0x45e0 bytes) 000189 1786416195.334982 mapread 0x01e731 thru 0x02b3de (0xccae bytes) 000190 1786416195.365593 trunc from 0x040000 to 0x033e53 (0xc1ac bytes) 000191 1786416195.394964 write 0x038689 thru 0x03ffff (0x7977 bytes) 000195 1786416195.417193 read 0x021fd9 thru 0x02aa77 (0x8a9f bytes) 000196 1786416195.418045 trunc from 0x040000 to 0x009d96 (0x36269 bytes) 000197 1786416195.486010 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9d97): Mirror 2: 0x2d01b5c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9d97): Mirror 2: 0x2d01b5c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9d97: skipped 000198 1786416196.034982 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000199 1786416196.190214 write 0x0309d7 thru 0x03d39d (0xc9c7 bytes) 000200 1786416196.193851 read 0x019c3f thru 0x02904a (0xf40c bytes) 000203 1786416196.206455 trunc from 0x03d39e to 0x01c76b (0x20c32 bytes) 000208 1786416196.243154 write 0x02b195 thru 0x02b323 (0x18f bytes) 000209 1786416196.246972 write 0x019089 thru 0x024ebd (0xbe35 bytes) 000210 1786416196.257992 mapwrite 0x013892 thru 0x0163be (0x2b2d bytes) 000211 1786416196.348081 read 0x006b48 thru 0x00eb23 (0x7fdc bytes) 000213 1786416196.373694 MIRROR_EXTEND to 2 mirrors 000214 1786416197.183121 mapwrite 0x035e62 thru 0x03cfc1 (0x7160 bytes) 000218 1786416197.528840 mapwrite 0x026b8a thru 0x02c7df (0x5c56 bytes) 000219 1786416197.631988 MIRROR_RESYNC 2 mirrors 000223 1786416199.122632 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000225 1786416199.476629 mapread 0x036e2c thru 0x03cfc1 (0x6196 bytes) 000226 1786416199.508342 read 0x00c8bb thru 0x010540 (0x3c86 bytes) 000227 1786416199.535993 read 0x015de6 thru 0x022923 (0xcb3e bytes) 000228 1786416199.553406 trunc from 0x03cfc2 to 0x0378d2 (0x56ef bytes) 000231 1786416199.639004 read 00000000 thru 0x00c30b (0xc30c bytes) 000234 1786416199.651190 mapwrite 0x002424 thru 0x003919 (0x14f6 bytes) 000235 1786416199.744809 write 0x02b04c thru 0x030005 (0x4fba bytes) 000239 1786416199.755877 read 0x033fe2 thru 0x0378d2 (0x38f1 bytes) 000241 1786416199.758614 mapwrite 0x027d8e thru 0x02c1bd (0x4430 bytes) 000244 1786416199.899616 write 0x00feb0 thru 0x01ac12 (0xad63 bytes) 000246 1786416199.908789 read 0x017fa9 thru 0x01ec33 (0x6c8b bytes) 000247 1786416199.909568 write 0x01edcf thru 0x0266a7 (0x78d9 bytes) 000249 1786416199.921388 write 0x00a288 thru 0x012a73 (0x87ec bytes) 000251 1786416199.928817 mapwrite 0x023e81 thru 0x032a2f (0xebaf bytes) 000254 1786416200.028832 mapread 0x019537 thru 0x020458 (0x6f22 bytes) 000255 1786416200.030291 write 0x02800b thru 0x02ab20 (0x2b16 bytes) 000256 1786416200.031448 mapread 0x01960e thru 0x027faf (0xe9a2 bytes) 000258 1786416200.039325 mapwrite 0x021d87 thru 0x021ddb (0x55 bytes) 000259 1786416200.090401 mapwrite 0x00b73e thru 0x011c88 (0x654b bytes) 000261 1786416200.139123 read 0x01bffe thru 0x02733f (0xb342 bytes) 000267 1786416200.147116 mapread 0x00f28d thru 0x0174a1 (0x8215 bytes) 000268 1786416200.150034 write 0x026e5e thru 0x02ba46 (0x4be9 bytes) 000269 1786416200.158588 MIRROR_EXTEND to 2 mirrors 000271 1786416201.112505 read 0x025294 thru 0x029c30 (0x499d bytes) 000276 1786416201.146906 mapwrite 0x01ab49 thru 0x01cc69 (0x2121 bytes) 000277 1786416201.323947 mapread 0x008722 thru 0x008ab5 (0x394 bytes) 000278 1786416201.335381 mapwrite 0x0001ee thru 0x00e48a (0xe29d bytes) 000280 1786416201.396050 read 0x00d41d thru 0x01b2ac (0xde90 bytes) 000282 1786416201.408251 mapwrite 0x032030 thru 0x036f7a (0x4f4b bytes) 000289 1786416201.462767 write 0x000e1f thru 0x00dbee (0xcdd0 bytes) 000291 1786416201.470276 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x378d3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378d3): Mirror 2: 0x164f0dab (stale) CRC-32 checksum value for chunk [0, 0x378d3): Mirror 1: 0x5d665e44 Mirror 3: 0x5d665e44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378d3: skipped 000294 1786416203.007080 mapread 0x015c1a thru 0x017a98 (0x1e7f bytes) 000295 1786416203.030399 mapwrite 0x02608e thru 0x033ca2 (0xdc15 bytes) 000296 1786416203.193373 trunc from 0x0378d3 to 0x00a42a (0x2d4a8 bytes) 000297 1786416203.223347 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa42b): Mirror 3: 0xb985cc9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa42b): Mirror 3: 0xb985cc9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa42b: skipped 000299 1786416203.560170 mapread 0x00a0a9 thru 0x00a42a (0x382 bytes) 000301 1786416203.569171 mapread 0x0099be thru 0x00a42a (0xa6d bytes) 000302 1786416203.581658 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xa42b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa42b): Mirror 3: 0xb985cc9e (stale) CRC-32 checksum value for chunk [0, 0xa42b): Mirror 1: 0xb985cc9e Mirror 4: 0xb985cc9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa42b: skipped 000303 1786416204.774241 trunc from 0x00a42b to 0x037505 (0x2d0db bytes) 000304 1786416204.856308 mapread 0x001e18 thru 0x011a11 (0xfbfa bytes) 000306 1786416204.901944 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37506): Mirror 3: 0x48a03932 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37506): Mirror 3: 0x48a03932 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37506: skipped 000307 1786416205.457233 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37506): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37506): Mirror 3: 0x48a03932 (stale) CRC-32 checksum value for chunk [0, 0x37506): Mirror 1: 0x583f36c2 Mirror 4: 0x583f36c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37506: skipped 000308 1786416206.851289 mapread 0x00f2b3 thru 0x01dd7e (0xeacc bytes) 000309 1786416206.911403 read 0x003a31 thru 0x01095a (0xcf2a bytes) 000311 1786416206.928671 mapread 0x020f8f thru 0x028a4f (0x7ac1 bytes) 000312 1786416206.949442 read 0x030e73 thru 0x037505 (0x6693 bytes) 000314 1786416206.965024 mapwrite 0x02d113 thru 0x03aa78 (0xd966 bytes) 000317 1786416207.292012 mapwrite 0x02b021 thru 0x02dd70 (0x2d50 bytes) 000318 1786416207.387877 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aa79): Mirror 4: 0x28e15f34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aa79): Mirror 4: 0x28e15f34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aa79: skipped 000321 1786416207.692945 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000323 1786416207.849402 MIRROR_EXTEND to 2 mirrors 000324 1786416209.105682 mapread 0x024237 thru 0x026e0e (0x2bd8 bytes) 000325 1786416209.193952 read 0x005059 thru 0x014882 (0xf82a bytes) 000326 1786416209.265029 mapwrite 0x01b10d thru 0x022120 (0x7014 bytes) 000327 1786416209.689828 mapread 0x03a17c thru 0x03aa78 (0x8fd bytes) 000328 1786416209.710001 write 0x024473 thru 0x02b032 (0x6bc0 bytes) 000330 1786416209.774845 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000334 1786416210.077708 MIRROR_EXTEND to 2 mirrors 000336 1786416211.417924 read 0x01b7b0 thru 0x01ed22 (0x3573 bytes) 000337 1786416211.436657 trunc from 0x03aa79 to 0x0264e9 (0x1458f bytes) 000339 1786416211.579784 write 0x03534c thru 0x03ffff (0xacb4 bytes) 000340 1786416211.585769 mapread 0x006585 thru 0x00cc28 (0x66a4 bytes) 000342 1786416211.632736 trunc from 0x040000 to 0x0063ea (0x39c15 bytes) 000343 1786416211.666182 write 0x02d670 thru 0x02ded0 (0x861 bytes) 000344 1786416211.667582 mapwrite 0x02ece5 thru 0x036f85 (0x82a1 bytes) 000345 1786416211.882203 mapread 0x0292d5 thru 0x035faf (0xccdb bytes) 000346 1786416211.892910 mapwrite 0x02f896 thru 0x02fe46 (0x5b1 bytes) 000348 1786416211.989899 mapwrite 0x0128f3 thru 0x021c31 (0xf33f bytes) 000349 1786416212.155095 mapwrite 0x022045 thru 0x02d1d8 (0xb194 bytes) 000350 1786416212.229188 mapread 0x0137a7 thru 0x019323 (0x5b7d bytes) 000351 1786416212.238128 MIRROR_RESYNC 2 mirrors 000352 1786416213.458572 read 0x02d0d3 thru 0x035173 (0x80a1 bytes) 000353 1786416213.485828 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000359 1786416213.684746 write 0x0044f0 thru 0x009d14 (0x5825 bytes) 000360 1786416213.759890 trunc from 0x036f86 to 0x02461c (0x12969 bytes) 000361 1786416213.827799 write 0x005f8a thru 0x00af30 (0x4fa7 bytes) 000366 1786416213.843366 trunc from 0x02461d to 0x036367 (0x11d4b bytes) 000367 1786416213.863152 mapread 0x024469 thru 0x02a7c4 (0x635c bytes) 000368 1786416213.901225 write 0x00db64 thru 0x015846 (0x7ce3 bytes) 000369 1786416213.915863 mapread 0x035dea thru 0x036367 (0x57e bytes) 000370 1786416213.928592 mapwrite 0x03c698 thru 0x03ffff (0x3968 bytes) 000372 1786416214.087400 MIRROR_EXTEND to 2 mirrors 000374 1786416214.939540 mapwrite 0x02fa84 thru 0x03daad (0xe02a bytes) 000378 1786416215.177141 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000379 1786416215.347404 trunc from 0x040000 to 0x0222b6 (0x1dd49 bytes) 000380 1786416215.366652 trunc from 0x0222b7 to 0x021560 (0xd56 bytes) 000381 1786416215.388947 MIRROR_EXTEND to 2 mirrors 000383 1786416216.430811 write 0x01bd20 thru 0x01f871 (0x3b52 bytes) 000384 1786416216.554536 write 0x010781 thru 0x01a150 (0x99d0 bytes) 000385 1786416216.582662 read 0x0194b1 thru 0x019c50 (0x7a0 bytes) 000386 1786416216.587570 mapread 0x00a758 thru 0x014321 (0x9bca bytes) 000388 1786416216.652379 read 0x001eeb thru 0x0037d8 (0x18ee bytes) 000393 1786416216.696572 trunc from 0x021561 to 0x02cdbe (0xb85e bytes) 000395 1786416216.753685 trunc from 0x02cdbf to 0x02ac43 (0x217b bytes) 000396 1786416216.799263 mapwrite 0x039d96 thru 0x03ffff (0x626a bytes) 000397 1786416216.990183 mapwrite 0x0047d1 thru 0x010935 (0xc165 bytes) 000398 1786416217.089998 write 0x00daa7 thru 0x018b10 (0xb06a bytes) 000399 1786416217.106918 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x558940c6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc8cd7418 Mirror 3: 0xc8cd7418 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000400 1786416218.789239 mapwrite 0x0316ea thru 0x03e236 (0xcb4d bytes) 000403 1786416219.221881 mapread 0x006928 thru 0x00b205 (0x48de bytes) 000404 1786416219.245899 mapread 0x006a3b thru 0x0154fd (0xeac3 bytes) 000406 1786416219.300159 mapwrite 0x02e738 thru 0x03a8e3 (0xc1ac bytes) 000408 1786416219.447312 mapwrite 0x013b53 thru 0x022c8c (0xf13a bytes) 000409 1786416219.578025 write 0x036724 thru 0x03ffff (0x98dc bytes) 000410 1786416219.582946 mapread 0x02480f thru 0x032619 (0xde0b bytes) 000411 1786416219.644651 MIRROR_RESYNC 3 mirrors 000413 1786416221.030436 read 0x0104ab thru 0x018681 (0x81d7 bytes) 000415 1786416221.076367 write 0x017538 thru 0x02183c (0xa305 bytes) 000416 1786416221.173401 write 0x023f8a thru 0x031c21 (0xdc98 bytes) 000417 1786416221.204618 trunc from 0x040000 to 0x02ccfa (0x13305 bytes) 000419 1786416221.230626 write 0x00fd47 thru 0x01f265 (0xf51f bytes) 000421 1786416221.239386 mapwrite 0x03770a thru 0x03ffff (0x88f6 bytes) 000423 1786416221.403078 mapread 0x021db0 thru 0x028d7d (0x6fce bytes) 000424 1786416221.423386 write 0x039ecd thru 0x03ffff (0x6133 bytes) 000426 1786416221.432981 mapwrite 0x000590 thru 0x00b09a (0xab0b bytes) 000427 1786416221.542588 read 0x00101e thru 0x0090c2 (0x80a5 bytes) 000429 1786416221.544836 trunc from 0x040000 to 0x0346e7 (0xb918 bytes) 000431 1786416221.572648 write 0x0161d2 thru 0x025313 (0xf142 bytes) 000432 1786416221.580663 mapread 0x031eab thru 0x0346e7 (0x283d bytes) 000435 1786416221.639632 write 0x03d771 thru 0x03ecc4 (0x1554 bytes) 000436 1786416221.644646 mapread 0x00f1fb thru 0x01ac98 (0xba9e bytes) 000437 1786416221.651182 read 0x01d45f thru 0x026329 (0x8ecb bytes) 000439 1786416221.655670 mapread 0x03a66a thru 0x03b26c (0xc03 bytes) 000441 1786416221.675569 trunc from 0x03ecc5 to 0x00d3fe (0x318c6 bytes) 000442 1786416221.736565 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xd3ff): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd3ff): Mirror 2: 0x218ba1c5 (stale) Mirror 3: 0x218ba1c5 (stale) CRC-32 checksum value for chunk [0, 0xd3ff): Mirror 1: 0xc0f6c7bf Mirror 4: 0xc0f6c7bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd3ff: skipped 000443 1786416223.339390 write 0x01864b thru 0x025442 (0xcdf8 bytes) 000445 1786416223.413076 write 0x026df7 thru 0x02c980 (0x5b8a bytes) 000448 1786416223.420632 mapread 0x004988 thru 0x007d1e (0x3397 bytes) 000449 1786416223.439339 mapwrite 0x01cf26 thru 0x022909 (0x59e4 bytes) 000450 1786416223.490083 write 0x034423 thru 0x03ffff (0xbbdd bytes) 000452 1786416223.493290 write 0x030021 thru 0x03717d (0x715d bytes) 000453 1786416223.499136 write 0x03ac92 thru 0x03ffff (0x536e bytes) 000455 1786416223.507008 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ff31704 (stale) Mirror 3: 0x3ff31704 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ff31704 (stale) Mirror 3: 0x3ff31704 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000456 1786416223.923471 read 0x017ccb thru 0x0181ad (0x4e3 bytes) 000457 1786416223.931981 write 0x02d7ae thru 0x03273f (0x4f92 bytes) 000459 1786416223.946944 mapread 0x00c74d thru 0x017f7a (0xb82e bytes) 000460 1786416223.973533 read 0x01143a thru 0x013d48 (0x290f bytes) 000463 1786416223.976967 write 0x00ea2e thru 0x01db16 (0xf0e9 bytes) 000464 1786416223.993174 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ff31704 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3ff31704 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000466 1786416224.508144 MIRROR_RESYNC 2 mirrors 000471 1786416225.901623 write 0x01bcba thru 0x026b5c (0xaea3 bytes) 000472 1786416226.071757 mapread 0x03a013 thru 0x03ffff (0x5fed bytes) 000474 1786416226.107564 mapread 0x010379 thru 0x01e98c (0xe614 bytes) 000479 1786416226.173945 read 0x008c49 thru 0x00cb3e (0x3ef6 bytes) truncating to largest ever: 0x3f9ea 000482 1786416226.190894 trunc from 0x040000 to 0x03f9e9 (0x616 bytes) 000483 1786416226.223386 read 0x02d6e8 thru 0x03312a (0x5a43 bytes) 000484 1786416226.245110 mapread 0x026155 thru 0x031c79 (0xbb25 bytes) 000486 1786416226.254741 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f9ea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9ea): Mirror 2: 0xe65823cf (stale) CRC-32 checksum value for chunk [0, 0x3f9ea): Mirror 1: 0x76028ca5 Mirror 3: 0x76028ca5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9ea: skipped 000488 1786416227.589152 read 0x0186bf thru 0x0218c8 (0x920a bytes) 000489 1786416227.606434 mapread 0x0039ff thru 0x008101 (0x4703 bytes) 000491 1786416227.643730 write 0x01a32b thru 0x01f4ac (0x5182 bytes) 000493 1786416227.865093 write 0x00993d thru 0x00d167 (0x382b bytes) 000494 1786416227.883963 trunc from 0x03f9ea to 0x034653 (0xb396 bytes) 000495 1786416227.950153 trunc from 0x034654 to 0x024fa0 (0xf6b3 bytes) 000497 1786416228.005747 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24fa1): Mirror 2: 0x2f0f4c3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24fa1): Mirror 2: 0x2f0f4c3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24fa1: skipped 000499 1786416228.450325 mapwrite 0x03074e thru 0x032b7a (0x242d bytes) 000501 1786416228.554117 mapwrite 0x02e8b8 thru 0x03e2d6 (0xfa1f bytes) 000502 1786416228.670241 write 0x023f12 thru 0x0296cc (0x57bb bytes) 000503 1786416228.684817 trunc from 0x03e2d7 to 0x012e71 (0x2b465 bytes) 000504 1786416228.716469 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000506 1786416228.919773 MIRROR_EXTEND to 2 mirrors 000507 1786416229.877814 mapread 0x010aa1 thru 0x012e71 (0x23d1 bytes) 000508 1786416229.958735 mapread 0x00c670 thru 0x00daa2 (0x1433 bytes) 000509 1786416229.988347 mapread 0x008597 thru 0x012e71 (0xa8db bytes) 000513 1786416230.110645 mapwrite 0x00daf3 thru 0x01341c (0x592a bytes) 000514 1786416230.430967 write 0x038f88 thru 0x03ffff (0x7078 bytes) 000517 1786416230.446013 mapread 0x02bd20 thru 0x03783e (0xbb1f bytes) 000521 1786416230.469895 mapwrite 0x0047c9 thru 0x0084d1 (0x3d09 bytes) 000523 1786416230.677487 mapread 0x0020da thru 0x0101e9 (0xe110 bytes) 000525 1786416230.687939 mapread 0x00bba8 thru 0x012322 (0x677b bytes) 000527 1786416230.714591 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb873fd21 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd8be924d Mirror 3: 0xd8be924d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000528 1786416232.346248 mapread 0x03ea34 thru 0x03ffff (0x15cc bytes) 000530 1786416232.382323 write 0x02402a thru 0x02b8c5 (0x789c bytes) 000531 1786416232.541529 write 0x02efb7 thru 0x0368f7 (0x7941 bytes) 000534 1786416232.561384 write 0x01d0b1 thru 0x029284 (0xc1d4 bytes) 000536 1786416232.570245 mapread 0x02bc2f thru 0x03aa3c (0xee0e bytes) 000538 1786416232.599044 mapwrite 0x01c275 thru 0x0210e3 (0x4e6f bytes) 000540 1786416232.724274 read 0x0044e3 thru 0x0079f0 (0x350e bytes) 000541 1786416232.740924 trunc from 0x040000 to 0x01d4f4 (0x22b0b bytes) 000542 1786416232.800711 trunc from 0x01d4f5 to 0x01aa52 (0x2aa2 bytes) 000543 1786416232.824356 mapwrite 0x02a553 thru 0x03533e (0xadec bytes) 000545 1786416233.098641 write 0x03d660 thru 0x03ffff (0x29a0 bytes) 000547 1786416233.129230 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb873fd21 (stale) Mirror 3: 0xd8be924d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d7dfb4c Mirror 4: 0x7d7dfb4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000551 1786416235.072829 mapwrite 0x0310f6 thru 0x038e15 (0x7d20 bytes) 000553 1786416235.308849 write 0x03a4d2 thru 0x03ffff (0x5b2e bytes) 000555 1786416235.322637 trunc from 0x040000 to 0x024845 (0x1b7ba bytes) 000556 1786416235.398991 write 0x02fcc9 thru 0x0374ef (0x7827 bytes) 000560 1786416235.407065 mapread 0x025b7c thru 0x033d5d (0xe1e2 bytes) 000562 1786416235.449806 write 0x01bdc7 thru 0x02334d (0x7587 bytes) 000564 1786416235.471951 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x374f0): Mirror 3: 0x5bad51c (stale) Mirror 4: 0x343d8fec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374f0): Mirror 3: 0x5bad51c (stale) Mirror 4: 0x343d8fec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374f0: skipped 000565 1786416236.256569 write 0x02a121 thru 0x02e60f (0x44ef bytes) 000566 1786416236.275701 mapread 0x00604c thru 0x007f66 (0x1f1b bytes) 000568 1786416236.294665 mapwrite 0x01982e thru 0x02918a (0xf95d bytes) 000571 1786416236.393932 mapwrite 0x032e11 thru 0x03cbe2 (0x9dd2 bytes) 000573 1786416236.622677 mapread 0x0317b7 thru 0x03268b (0xed5 bytes) 000574 1786416236.625542 mapread 0x0266ac thru 0x02b2e8 (0x4c3d bytes) 000575 1786416236.627030 mapread 0x03c47a thru 0x03cbe2 (0x769 bytes) 000576 1786416236.644118 mapwrite 0x03ab9d thru 0x03ffff (0x5463 bytes) 000578 1786416236.848639 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7d7dfb4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7d7dfb4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000580 1786416237.414310 write 0x0268a7 thru 0x02f1d1 (0x892b bytes) 000581 1786416237.442539 read 0x01f719 thru 0x01fcd8 (0x5c0 bytes) 000582 1786416237.453325 write 0x00a5e8 thru 0x010f53 (0x696c bytes) 000583 1786416237.480396 write 0x00f98d thru 0x01e984 (0xeff8 bytes) 000584 1786416237.500568 read 0x02a741 thru 0x037827 (0xd0e7 bytes) 000586 1786416237.541035 trunc from 0x040000 to 0x03c765 (0x389a bytes) 000587 1786416237.568148 MIRROR_RESYNC 2 mirrors 000588 1786416238.595916 read 0x029a59 thru 0x035c7c (0xc224 bytes) 000589 1786416238.622110 MIRROR_EXTEND to 3 mirrors 000590 1786416239.841136 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000591 1786416240.474307 write 0x02a1e4 thru 0x039202 (0xf01f bytes) 000593 1786416240.650804 mapwrite 0x02cb79 thru 0x02d515 (0x99d bytes) 000594 1786416240.689365 mapwrite 0x016ff6 thru 0x022178 (0xb183 bytes) 000596 1786416240.908334 mapread 0x02fb64 thru 0x034cc2 (0x515f bytes) 000597 1786416240.914564 read 0x02c56d thru 0x0349d6 (0x846a bytes) 000598 1786416240.921551 mapread 0x0390a1 thru 0x03c059 (0x2fb9 bytes) 000599 1786416240.941621 mapread 0x00bc21 thru 0x00dca9 (0x2089 bytes) 000600 1786416240.981844 mapwrite 0x0385dd thru 0x03a169 (0x1b8d bytes) 000601 1786416241.091153 mapread 0x0100e8 thru 0x011f35 (0x1e4e bytes) 000602 1786416241.092372 mapwrite 0x010cf8 thru 0x01f34c (0xe655 bytes) 000605 1786416241.223891 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000606 1786416241.435197 mapread 0x02f75d thru 0x0312d7 (0x1b7b bytes) 000608 1786416241.464632 trunc from 0x03c766 to 0x00ac93 (0x31ad2 bytes) 000610 1786416241.491006 mapwrite 0x028577 thru 0x03597a (0xd404 bytes) 000611 1786416241.722512 mapwrite 0x02ac4d thru 0x038fa9 (0xe35d bytes) 000615 1786416241.944511 mapwrite 0x002d5e thru 0x008ee1 (0x6184 bytes) 000616 1786416242.067613 trunc from 0x038faa to 0x03457a (0x4a2f bytes) 000619 1786416242.101527 read 0x0248e5 thru 0x028606 (0x3d22 bytes) 000620 1786416242.103060 mapwrite 0x0100c4 thru 0x01dfc9 (0xdf06 bytes) 000622 1786416242.233896 read 0x02c9ef thru 0x02e323 (0x1935 bytes) 000623 1786416242.238026 trunc from 0x03457b to 0x00e01b (0x2655f bytes) 000626 1786416242.269968 mapwrite 0x02b489 thru 0x03a9d3 (0xf54b bytes) 000629 1786416242.433257 trunc from 0x03a9d4 to 0x0374c8 (0x350b bytes) 000631 1786416242.460577 read 0x015d0c thru 0x01ae10 (0x5105 bytes) 000633 1786416242.465559 read 0x0052cb thru 0x0111d2 (0xbf08 bytes) 000637 1786416242.469722 mapwrite 0x00fa07 thru 0x015639 (0x5c33 bytes) 000639 1786416242.537667 trunc from 0x0374c9 to 0x018f81 (0x1e547 bytes) 000641 1786416242.560745 trunc from 0x018f82 to 0x014e47 (0x413a bytes) 000643 1786416242.602920 read 0x01170e thru 0x011a9e (0x391 bytes) 000644 1786416242.604007 write 0x03294c thru 0x035c0a (0x32bf bytes) 000650 1786416242.617964 mapread 0x01b42a thru 0x029e57 (0xea2e bytes) 000653 1786416242.639543 trunc from 0x035c0b to 0x01c15f (0x19aab bytes) 000655 1786416242.668720 write 0x030fd9 thru 0x036844 (0x586c bytes) 000656 1786416242.671048 trunc from 0x036845 to 0x03b8b7 (0x5073 bytes) 000657 1786416242.703257 mapwrite 0x012462 thru 0x015d91 (0x3930 bytes) 000658 1786416242.789252 trunc from 0x03b8b8 to 0x035f96 (0x5921 bytes) 000659 1786416242.816889 mapread 0x0193f2 thru 0x01e4f9 (0x5108 bytes) 000664 1786416242.845210 MIRROR_EXTEND to 2 mirrors 000666 1786416244.077100 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000667 1786416244.233504 write 0x00b13d thru 0x0192a1 (0xe165 bytes) 000669 1786416244.270405 mapread 0x0081ff thru 0x0100d5 (0x7ed7 bytes) 000670 1786416244.298315 write 0x025ac6 thru 0x02a314 (0x484f bytes) 000673 1786416244.312667 mapwrite 0x017895 thru 0x01f167 (0x78d3 bytes) 000674 1786416244.387866 trunc from 0x035f97 to 0x03d76f (0x77d9 bytes) 000675 1786416244.410506 MIRROR_EXTEND to 2 mirrors 000677 1786416245.304406 write 0x018334 thru 0x024132 (0xbdff bytes) 000686 1786416245.399972 mapread 0x017f6e thru 0x01c800 (0x4893 bytes) 000687 1786416245.406645 trunc from 0x03d770 to 0x015d99 (0x279d6 bytes) 000689 1786416245.435126 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15d9a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15d9a): Mirror 2: 0x2c6ef02f (stale) CRC-32 checksum value for chunk [0, 0x15d9a): Mirror 1: 0x2c6ef02f Mirror 3: 0x2c6ef02f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15d9a: skipped 000691 1786416246.883574 mapwrite 0x032ff6 thru 0x03c843 (0x984e bytes) 000692 1786416247.164843 trunc from 0x03c844 to 0x00e31e (0x2e525 bytes) 000693 1786416247.209324 write 0x033dc5 thru 0x034da5 (0xfe1 bytes) 000694 1786416247.212621 read 0x006835 thru 0x012d83 (0xc54f bytes) 000698 1786416247.275139 write 0x0270c1 thru 0x03688d (0xf7cd bytes) 000700 1786416247.310882 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3688e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3688e): Mirror 2: 0x7b0abc6a (stale) Mirror 3: 0x832bc508 (stale) CRC-32 checksum value for chunk [0, 0x3688e): Mirror 1: 0xc711befc Mirror 4: 0xc711befc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3688e: skipped 000702 1786416249.142238 mapread 0x01a0d3 thru 0x01d139 (0x3067 bytes) 000703 1786416249.188217 read 0x01a42a thru 0x01e32d (0x3f04 bytes) 000705 1786416249.208558 read 0x0096ea thru 0x00dc71 (0x4588 bytes) 000707 1786416249.258093 write 0x03f892 thru 0x03ffff (0x76e bytes) 000708 1786416249.356677 trunc from 0x040000 to 0x01be42 (0x241bd bytes) 000709 1786416249.388105 read 0x014e9c thru 0x016045 (0x11aa bytes) 000711 1786416249.406280 mapwrite 0x02e1f4 thru 0x03419e (0x5fab bytes) 000714 1786416249.527354 mapread 0x00df8f thru 0x01de6a (0xfedc bytes) 000716 1786416249.530854 write 0x027d12 thru 0x02e697 (0x6986 bytes) 000717 1786416249.533239 mapwrite 0x03aeff thru 0x03ffff (0x5101 bytes) 000718 1786416249.633630 read 0x017de7 thru 0x0269b0 (0xebca bytes) 000722 1786416249.639900 mapread 0x01cbb7 thru 0x025a68 (0x8eb2 bytes) 000723 1786416249.653146 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe5d6f2ab (stale) Mirror 4: 0x15c0c53e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe5d6f2ab (stale) Mirror 4: 0x15c0c53e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000725 1786416250.133465 mapread 0x02b934 thru 0x03a84a (0xef17 bytes) 000726 1786416250.180468 mapwrite 0x010d85 thru 0x01d10a (0xc386 bytes) 000727 1786416250.277885 MIRROR_RESYNC 3 mirrors 000728 1786416251.248339 read 0x00d2da thru 0x011b91 (0x48b8 bytes) 000729 1786416251.259009 trunc from 0x040000 to 0x0185fc (0x27a03 bytes) 000730 1786416251.363685 read 0x00544a thru 0x00c7fb (0x73b2 bytes) 000737 1786416251.381176 mapread 0x014763 thru 0x0185fc (0x3e9a bytes) 000739 1786416251.388730 trunc from 0x0185fd to 0x000aa3 (0x17b59 bytes) 000740 1786416251.412024 mapwrite 0x007d63 thru 0x00de29 (0x60c7 bytes) 000743 1786416251.520180 read 0x00a350 thru 0x00b9f1 (0x16a2 bytes) 000745 1786416251.524098 trunc from 0x00de2a to 0x02a514 (0x1c6eb bytes) 000747 1786416251.540769 read 0x027159 thru 0x0271ba (0x62 bytes) 000748 1786416251.551485 mapread 0x02a3f5 thru 0x02a514 (0x120 bytes) 000749 1786416251.554034 write 0x0142a9 thru 0x01e3a7 (0xa0ff bytes) 000751 1786416251.561930 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a515): Mirror 4: 0x729d19a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a515): Mirror 4: 0x729d19a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a515: skipped 000753 1786416251.866103 mapread 0x024f95 thru 0x0260dc (0x1148 bytes) 000754 1786416251.908078 read 0x0262e2 thru 0x02a514 (0x4233 bytes) 000755 1786416251.910139 mapwrite 0x019346 thru 0x028274 (0xef2f bytes) 000756 1786416251.994920 read 0x008aec thru 0x010951 (0x7e66 bytes) 000758 1786416252.011227 mapwrite 0x003656 thru 0x007602 (0x3fad bytes) 000759 1786416252.082994 mapread 0x00a39d thru 0x00dc2e (0x3892 bytes) 000760 1786416252.084139 trunc from 0x02a515 to 0x002cca (0x2784a bytes) 000761 1786416252.104214 mapwrite 0x02397a thru 0x02eaea (0xb171 bytes) 000763 1786416252.265404 mapwrite 0x031aee thru 0x03681e (0x4d31 bytes) 000767 1786416252.400689 mapread 0x01928d thru 0x028c5a (0xf9ce bytes) 000770 1786416252.477066 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000773 1786416252.648644 mapread 0x02a1c4 thru 0x02f8ab (0x56e8 bytes) 000778 1786416252.678518 read 0x023a0a thru 0x02cfdf (0x95d6 bytes) 000782 1786416252.692245 trunc from 0x03681f to 0x02ed84 (0x7a9a bytes) 000783 1786416252.721471 mapread 0x01cef1 thru 0x01d900 (0xa10 bytes) 000784 1786416252.734532 write 0x029a0e thru 0x02ba5a (0x204d bytes) 000785 1786416252.735716 read 0x00df29 thru 0x0131cd (0x52a5 bytes) 000786 1786416252.750895 write 0x029d64 thru 0x038f45 (0xf1e2 bytes) 000787 1786416252.754867 mapread 0x02a4de thru 0x02dc4f (0x3772 bytes) 000790 1786416252.762805 mapwrite 0x02514f thru 0x031665 (0xc517 bytes) 000791 1786416252.846338 write 0x00d69f thru 0x011867 (0x41c9 bytes) 000792 1786416252.851485 trunc from 0x038f46 to 0x02a2f2 (0xec53 bytes) 000795 1786416252.879398 trunc from 0x02a2f3 to 0x0315d0 (0x72de bytes) 000796 1786416252.923174 mapwrite 0x03804c thru 0x03e493 (0x6448 bytes) 000797 1786416253.051542 MIRROR_EXTEND to 2 mirrors 000798 1786416253.968083 read 0x0248fc thru 0x02bb93 (0x7298 bytes) 000800 1786416254.017328 mapwrite 0x0162be thru 0x022b43 (0xc886 bytes) 000805 1786416254.279866 trunc from 0x03e494 to 0x02e70d (0xfd86 bytes) 000807 1786416254.324630 mapread 0x01d819 thru 0x023002 (0x57ea bytes) 000809 1786416254.330394 mapwrite 0x019ae5 thru 0x021dc4 (0x82e0 bytes) 000811 1786416254.418455 read 0x0042cf thru 0x00ab24 (0x6856 bytes) 000812 1786416254.442399 read 0x000747 thru 0x00c00a (0xb8c4 bytes) 000813 1786416254.452794 MIRROR_RESYNC 2 mirrors 000816 1786416255.069082 mapwrite 0x016552 thru 0x023f92 (0xda41 bytes) 000817 1786416255.303823 mapwrite 0x028fbf thru 0x036155 (0xd197 bytes) 000818 1786416255.463299 mapwrite 0x00529d thru 0x00a3a5 (0x5109 bytes) 000819 1786416255.513191 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000821 1786416255.823751 trunc from 0x036156 to 0x02c9fb (0x975a bytes) 000822 1786416255.884786 MIRROR_EXTEND to 2 mirrors 000823 1786416256.653510 trunc from 0x02c9fc to 0x00fbca (0x1ce31 bytes) 000825 1786416256.735019 write 0x00a72c thru 0x0137d6 (0x90ab bytes) 000826 1786416256.740077 mapwrite 0x034292 thru 0x03d8be (0x962d bytes) 000827 1786416256.859236 read 0x0282fe thru 0x032f08 (0xac0b bytes) 000829 1786416256.878840 mapwrite 0x025757 thru 0x0318a0 (0xc14a bytes) 000831 1786416256.965994 write 0x039096 thru 0x03fe23 (0x6d8e bytes) 000832 1786416256.968577 mapwrite 0x01ee4f thru 0x020718 (0x18ca bytes) 000833 1786416257.044650 mapwrite 0x024b0e thru 0x02a093 (0x5586 bytes) 000834 1786416257.082913 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000837 1786416257.190278 mapread 0x019d27 thru 0x02724f (0xd529 bytes) 000838 1786416257.251931 write 0x019f19 thru 0x029c07 (0xfcef bytes) 000839 1786416257.254401 mapread 0x00901d thru 0x00cd0e (0x3cf2 bytes) 000842 1786416257.268247 mapread 0x03c631 thru 0x03fe23 (0x37f3 bytes) 000843 1786416257.280546 write 0x01615a thru 0x0227dd (0xc684 bytes) 000844 1786416257.283860 read 0x009eb4 thru 0x00b090 (0x11dd bytes) 000849 1786416257.292941 read 0x017394 thru 0x018d65 (0x19d2 bytes) 000851 1786416257.294758 mapread 0x021a60 thru 0x03093d (0xeede bytes) 000852 1786416257.312908 trunc from 0x03fe24 to 0x010334 (0x2faef bytes) 000855 1786416257.358177 read 0x004a64 thru 0x010334 (0xb8d1 bytes) 000858 1786416257.362873 mapwrite 0x01354e thru 0x01d6d8 (0xa18b bytes) 000859 1786416257.469910 MIRROR_EXTEND to 2 mirrors 000860 1786416258.543808 mapread 0x0178ea thru 0x01d6d8 (0x5def bytes) 000861 1786416258.630003 mapread 0x00ebf7 thru 0x01ae0f (0xc219 bytes) 000862 1786416258.701867 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000865 1786416258.965571 mapread 0x018c31 thru 0x01d6d8 (0x4aa8 bytes) 000866 1786416259.005992 read 0x0149e2 thru 0x019f89 (0x55a8 bytes) 000867 1786416259.007959 trunc from 0x01d6d9 to 0x02f05a (0x11982 bytes) 000868 1786416259.057262 mapread 0x0260d5 thru 0x02766a (0x1596 bytes) 000869 1786416259.079073 MIRROR_EXTEND to 2 mirrors 000870 1786416260.096844 write 0x038a67 thru 0x03ffff (0x7599 bytes) 000872 1786416260.283871 read 0x027123 thru 0x036e75 (0xfd53 bytes) 000873 1786416260.321010 MIRROR_RESYNC 2 mirrors 000874 1786416261.817029 write 0x014fb0 thru 0x01a813 (0x5864 bytes) 000877 1786416261.906039 trunc from 0x040000 to 0x025e7c (0x1a183 bytes) 000880 1786416261.930491 mapwrite 0x01e96b thru 0x025a0c (0x70a2 bytes) 000881 1786416262.025214 mapread 0x005443 thru 0x00e33c (0x8efa bytes) 000883 1786416262.036122 read 0x014bf8 thru 0x0227ce (0xdbd7 bytes) 000886 1786416262.042765 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000887 1786416262.162870 write 0x019ddd thru 0x01b2f8 (0x151c bytes) 000894 1786416262.191173 read 0x00a0df thru 0x014ee6 (0xae08 bytes) 000896 1786416262.228182 write 0x036bff thru 0x03a421 (0x3823 bytes) 000898 1786416262.234365 write 0x0275fa thru 0x02df3f (0x6946 bytes) 000900 1786416262.269466 mapwrite 0x003ce2 thru 0x0127d6 (0xeaf5 bytes) 000901 1786416262.474769 write 0x03b77d thru 0x03ffff (0x4883 bytes) 000903 1786416262.492778 read 0x015497 thru 0x01c5ab (0x7115 bytes) 000904 1786416262.518952 write 0x0399a1 thru 0x03cdfe (0x345e bytes) 000905 1786416262.529630 write 0x0251ae thru 0x0253d2 (0x225 bytes) 000906 1786416262.532461 write 0x00ab46 thru 0x0131d3 (0x868e bytes) 000907 1786416262.537941 write 0x002bfb thru 0x011c0a (0xf010 bytes) 000908 1786416262.540888 mapwrite 0x00e680 thru 0x00f1ac (0xb2d bytes) 000909 1786416262.723514 trunc from 0x040000 to 0x012564 (0x2da9b bytes) 000910 1786416262.770053 MIRROR_EXTEND to 2 mirrors 000912 1786416263.930402 mapread 0x004455 thru 0x00c6aa (0x8256 bytes) 000913 1786416263.959457 mapwrite 0x029c6a thru 0x038978 (0xed0f bytes) 000914 1786416264.189678 mapread 0x01ed18 thru 0x024ed4 (0x61bd bytes) 000917 1786416264.213698 mapread 0x024651 thru 0x025bcb (0x157b bytes) 000918 1786416264.226207 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38979): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38979): Mirror 2: 0x81bebf40 (stale) CRC-32 checksum value for chunk [0, 0x38979): Mirror 1: 0x372ca021 Mirror 3: 0x372ca021 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38979: skipped 000919 1786416265.448882 mapwrite 0x00c4a9 thru 0x01a624 (0xe17c bytes) 000921 1786416265.761989 trunc from 0x038979 to 0x01ee10 (0x19b68 bytes) 000927 1786416265.842361 trunc from 0x01ee11 to 0x036e79 (0x18069 bytes) 000928 1786416265.865917 write 0x00fdd4 thru 0x01e650 (0xe87d bytes) 000929 1786416265.867593 mapwrite 0x01d058 thru 0x0218a7 (0x4850 bytes) 000930 1786416265.974445 mapread 0x00d801 thru 0x00ef17 (0x1717 bytes) 000932 1786416265.987481 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36e7a): Mirror 3: 0x5d32e0cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36e7a): Mirror 3: 0x5d32e0cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36e7a: skipped 000933 1786416266.447378 trunc from 0x036e7a to 0x023e48 (0x13031 bytes) 000934 1786416266.466343 read 0x019924 thru 0x021b4a (0x8227 bytes) 000935 1786416266.477732 mapwrite 0x02cc5e thru 0x03a285 (0xd628 bytes) 000938 1786416266.563494 MIRROR_RESYNC 2 mirrors 000939 1786416267.802117 read 0x02a83b thru 0x03a285 (0xfa4b bytes) 000941 1786416267.840666 MIRROR_RESYNC 2 mirrors 000944 1786416268.311805 write 0x0330c9 thru 0x03ffff (0xcf37 bytes) 000948 1786416268.498375 mapwrite 0x01ccd6 thru 0x020535 (0x3860 bytes) 000949 1786416268.781328 trunc from 0x040000 to 0x013a08 (0x2c5f7 bytes) 000950 1786416268.851274 trunc from 0x013a09 to 0x00bbcc (0x7e3c bytes) 000951 1786416268.888285 mapwrite 0x013226 thru 0x01468e (0x1469 bytes) 000953 1786416269.075054 mapread 0x00c7a3 thru 0x01007c (0x38da bytes) 000954 1786416269.082136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000957 1786416269.276817 mapwrite 0x01f197 thru 0x02e7e3 (0xf64d bytes) 000958 1786416269.418123 write 0x0083d6 thru 0x014286 (0xbeb1 bytes) 000961 1786416269.433222 write 0x02075a thru 0x02a3e0 (0x9c87 bytes) 000962 1786416269.435160 mapwrite 0x0018ba thru 0x00690f (0x5056 bytes) 000963 1786416269.485872 write 0x03434b thru 0x03d9ce (0x9684 bytes) 000968 1786416269.491149 read 0x03970b thru 0x03d9ce (0x42c4 bytes) 000970 1786416269.498582 MIRROR_EXTEND to 2 mirrors 000971 1786416270.119316 mapread 0x01625b thru 0x017bee (0x1994 bytes) 000972 1786416270.143587 write 0x026bca thru 0x02a621 (0x3a58 bytes) 000976 1786416270.241651 read 0x00048f thru 0x00a731 (0xa2a3 bytes) 000978 1786416270.274140 read 0x029bfa thru 0x02d144 (0x354b bytes) 000979 1786416270.289975 mapread 0x003888 thru 0x00bc6c (0x83e5 bytes) 000981 1786416270.292853 trunc from 0x03d9cf to 0x025aa0 (0x17f2e bytes) 000984 1786416270.408958 trunc from 0x025aa1 to 0x020026 (0x5a7a bytes) 000987 1786416270.437689 write 0x03672a thru 0x03ffff (0x98d6 bytes) 000992 1786416270.443151 read 0x03798f thru 0x03ffff (0x8671 bytes) 000993 1786416270.446112 mapread 0x03c40e thru 0x03ffff (0x3bf2 bytes) 000995 1786416270.451373 read 0x033e4a thru 0x03512b (0x12e2 bytes) 000996 1786416270.467609 write 0x02e3bf thru 0x036c42 (0x8884 bytes) All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 22:44:37 (1786416277) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 22:44:42 (1786416282) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244605 s, 8.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336822 s, 3.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 22:44:53 (1786416293) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2665216 .. Extending file size to 6920448 .. Extending file size to 8176800 .. Extending file size to 2719968 .. Extending file size to 8103680 .. Extending file size to 8265920 .. Extending file size to 8849024 .. Extending file size to 8516096 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9184800 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9024256 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57325 57326 57330 57331 57332 done 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) 192.168.202.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:46:21 (1786416381) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7489664 .. Extending file size to 7164160 .. Extending file size to 7330208 .. Extending file size to 8151488 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7731744 .. Extending file size to 8387712 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8654240 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8758016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9211776 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8973408 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9075680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58942 58943 58944 58945 58946 failed 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) 192.168.202.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:47:44 (1786416464) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117775 s, 17.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.523315 s, 4.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xccda78b0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:48:09 (1786416489) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133005 s, 15.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 22:48:17 (1786416497) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:48:26 (1786416506) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169767 s, 12.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:48:35 (1786416515) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284395 s, 7.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } PASS 204b (12s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:48:47 (1786416527) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.34358 s, 6.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:49:23 (1786416563) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298739 s, 3.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } PASS 204d (66s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:50:29 (1786416629) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.430265 s, 4.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:51:30 (1786416690) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg243-server mds-ost sync done. Creating new pool test_204f oleg243-server: Pool lustre.test_204f created Adding targets to pool oleg243-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg243-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.315415 s, 6.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg243-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg243-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg243-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:52:57 (1786416777) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:53:07 (1786416787) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:53:17 (1786416797) oleg243-server: Pool lustre.test_206 created Adding targets to pool oleg243-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg243-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg243-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg243-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg243-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:53:47 (1786416827) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678294 s, 15.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111948 s, 9.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd0cfa0c2 Mirror 3: 0xd0cfa0c2 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:54:00 (1786416840) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:54:04 (1786416844) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:54:09 (1786416849) Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0940417 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xd90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xd90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xd90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1570000 Read this (4 bytes): BBBB 192.168.202.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg243-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:54:51 (1786416891) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0194497 s, 421 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:55:00 (1786416900) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415804 s, 25.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fd lmm_fid: [0x200000404:0x1fd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fd lmm_fid: [0x200000404:0x1fd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834429 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-10 22:55:02.000000000 -0400 Modify: 2026-08-10 22:55:02.000000000 -0400 Change: 2026-08-10 22:55:04.000000000 -0400 Birth: 2026-08-10 22:55:01.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:55:11 (1786416911) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:55:16 (1786416916) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.976189 s, 42.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0777423 s, 52.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 10 22:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ff:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 10 22:55 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:55:27 (1786416927) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:55:38 (1786416938) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 2033 sec =========== 22:55:49 (1786416949) === sanity-flr: start cleanup 22:55:52 (1786416952) === === sanity-flr: finish cleanup 22:56:00 (1786416960) ===