-----============= acceptance-small: sanity-flr ============----- Thu May 1 15:03:56 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg132-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:04:05 (1746126245) === oleg132-client.virtnet: executing check_config_client /mnt/lustre oleg132-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg132-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6f19000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6f19000.idle_timeout=debug disable quota as required oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:04:16 (1746126256) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7081] == sanity-flr test 0a: lfs mirror create with -N option == 15:04:17 (1746126257) PASS 0a (9s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:04:26 (1746126266) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:04:29 (1746126269) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:04:30 (1746126270) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:04:42 (1746126282) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:04:44 (1746126284) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 15:04:45 (1746126285) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f0g.sanity-flr': Invalid argument (22) sanity-flr test_0g: @@@@@@ FAIL: error resync-ing file '/mnt/lustre/f0g.sanity-flr' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:740:test_0g() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:751:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0g.*.1746126287.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0g.debug_log.oleg132-server.1746126287.log.lDJV5q" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0g.dmesg.oleg132-server.1746126287.log.OnnfD3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0g (5s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:04:52 (1746126292) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead lfs mirror mirror: fail to pread 0-4096 of mirror 2: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 2: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr': Invalid argument (22) sanity-flr test_0h: @@@@@@ FAIL: error resync-ing file '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:793:test_0h() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:831:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0h.*.1746126295.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0h.debug_log.oleg132-server.1746126295.log.AER1eU" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0h.dmesg.oleg132-server.1746126295.log.thsYnO" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:05:00 (1746126300) lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f0j.sanity-flr': Invalid argument (22) sanity-flr test_0j: @@@@@@ FAIL: resync /mnt/lustre/f0j.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:837:test_0j() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:849:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0j.*.1746126303.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0j.debug_log.oleg132-server.1746126303.log.BotkQX" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0j.dmesg.oleg132-server.1746126303.log.enbCbc" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:05:08 (1746126308) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:05:10 (1746126310) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 15:05:16 (1746126316) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:05:19 (1746126319) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:05:21 (1746126321) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:05:26 (1746126326) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:05:32 (1746126332) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:05:36 (1746126336) 15+0 records in 15+0 records out 15728640 bytes (16 MB) copied, 1.81737 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.13655 s, 7.7 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:05:44 (1746126344) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.147222 s, 57.0 MB/s PASS 22 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:05:50 (1746126350) Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid in DISCONN state after 0 sec Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg132-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (39s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:06:28 (1746126388) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.126196 s, 24.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (24s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:06:52 (1746126412) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg132-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (40s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:07:32 (1746126452) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.67331 s, 31.3 MB/s -rw-r--r-- 1 root root 52428800 May 1 15:07 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.770s user 0m0.000s sys 0m0.177s Stopping /mnt/lustre-ost1 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b6f19000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.771s user 0m0.001s sys 0m0.179s Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg132-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg132-server oleg132-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.906s user 0m0.000s sys 0m0.191s Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg132-server: oleg132-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg132-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg132-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6f19000.ost_server_uuid in FULL state after 0 sec PASS 33b (49s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:08:21 (1746126501) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:08:23 (1746126503) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:08:26 (1746126506) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 15:08:28 (1746126508) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 15:08:32 (1746126512) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.403811 s, 31.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:08:40 (1746126520) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195601 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0220458 s, 47.6 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:08:44 (1746126524) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:08:49 (1746126529) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00862303 s, 119 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00377151 s, 272 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f36d.sanity-flr': Invalid argument (22) sanity-flr test_36d: @@@@@@ FAIL: mirror resync /mnt/lustre/f36d.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1643:test_36d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1674:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_36d.*.1746126533.log rsync: chown "/tmp/testlogs/.sanity-flr.test_36d.debug_log.oleg132-server.1746126533.log.WTwTJQ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_36d.dmesg.oleg132-server.1746126533.log.AemJrV" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 15:09:01 (1746126541) create test files with size 15747397 .. a41fdec422c1f582e6e57e00873608f0 - 065e79c0dc119dd1e85c56f691bbb707 - fe10240f6e575147b0bae3c00be95d86 - 3 Verifying mirror read .. lfs mirror mirror: fail to pread 12582912-4194304 of mirror 1: Invalid argument (22) lfs read: fail to read data from mirror 1: Invalid argument sanity-flr test_37: @@@@@@ FAIL: 1: mismatch: \'a41fdec422c1f582e6e57e00873608f0 -\' vs. \'ff3fbf19d33969a6ee1c3e89783650bf -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1730:test_37() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1759:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_37.*.1746126548.log rsync: chown "/tmp/testlogs/.sanity-flr.test_37.debug_log.oleg132-server.1746126548.log.ERtKtV" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_37.dmesg.oleg132-server.1746126548.log.fPgdBz" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 37 (10s) == sanity-flr test 38: resync ============================ 15:09:15 (1746126555) 4 mirror IDs: 1 2 3 4 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs read: fail to read data from mirror 1: Invalid argument lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument lfs mirror mirror: fail to pread 0-4194304 of mirror 3: Invalid argument (22) lfs read: fail to read data from mirror 3: Invalid argument lfs mirror mirror: fail to pread 0-4194304 of mirror 4: Invalid argument (22) lfs read: fail to read data from mirror 4: Invalid argument sanity-flr test_38: @@@@@@ FAIL: valid mirror doesn't exist Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1811:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1839:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_38.*.1746126558.log rsync: chown "/tmp/testlogs/.sanity-flr.test_38.debug_log.oleg132-server.1746126558.log.GwkJl9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_38.dmesg.oleg132-server.1746126558.log.VcY0bd" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:09:24 (1746126564) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:09:28 (1746126568) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0766972 s, 13.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x121 lmm_fid: 0x200000401:0x121:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x121 lmm_fid: 0x200000401:0x121:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x121 lmm_fid: 0x200000401:0x121:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x121 lmm_fid: 0x200000401:0x121:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x121 lmm_fid: 0x200000401:0x121:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x121 lmm_fid: 0x200000401:0x121:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x121 lmm_fid: 0x200000401:0x121:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0357208 s, 29.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x123 lmm_fid: 0x200000401:0x123:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x123 lmm_fid: 0x200000401:0x123:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x123 lmm_fid: 0x200000401:0x123:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x123 lmm_fid: 0x200000401:0x123:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x123 lmm_fid: 0x200000401:0x123:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x123 lmm_fid: 0x200000401:0x123:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x123 lmm_fid: 0x200000401:0x123:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (5s) == sanity-flr test 41: lfs mirror resync check =========== 15:09:33 (1746126573) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0941515 s, 22.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.252203 s, 16.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.18457 s, 11.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 15:09:41 (1746126581) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 15:09:43 (1746126583) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:09:45 (1746126585) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171531 s, 61.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (5s) == sanity-flr test 44a: lfs mirror split check =========== 15:09:50 (1746126590) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:09:52 (1746126592) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537 sanity-flr test_44b: @@@@@@ FAIL: split --mirror-id=1 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr should succeed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2290:test_44b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2308:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_44b.*.1746126595.log rsync: chown "/tmp/testlogs/.sanity-flr.test_44b.debug_log.oleg132-server.1746126595.log.qKHBBP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_44b.dmesg.oleg132-server.1746126595.log.HARuht" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 44b (6s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:10:04 (1746126604) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.194414 s, 53.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000401:0x131:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg132-client.virtnet /mnt/lustre (opts:) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:10:12 (1746126612) llite.lustre-ffff8800a2d4a000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.184646 s, 56.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 Stopping client oleg132-client.virtnet /mnt/lustre (opts:) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40974 flags: 1 llite.lustre-ffff8800a2f23800.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:10:21 (1746126621) getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:10:39 (1746126639) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:10:54 (1746126654) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 15:10:56 (1746126656) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.154892 s, 20.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.128798 s, 24.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: d09ea95f673a76d3b07139ab82f0b0a9 - mirror 1 checksum: 4cb20036f412df49d905edc4ea663bed - mirror 2 checksum: d09ea95f673a76d3b07139ab82f0b0a9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 1 15:10:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x11 lmm_fid: [0x200000403:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu May 1 15:10:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x11 lmm_fid: [0x200000403:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d09ea95f673a76d3b07139ab82f0b0a9 - mirror 1 checksum: 4cb20036f412df49d905edc4ea663bed - mirror 2 checksum: 4cb20036f412df49d905edc4ea663bed - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 1 15:10:58 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x11 lmm_fid: [0x200000403:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu May 1 15:10:58 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x11 lmm_fid: [0x200000403:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (6s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:11:03 (1746126663) Failing mds1 on oleg132-server Stopping /mnt/lustre-mds1 (opts:) on oleg132-server 15:11:12 (1746126672) shut down facet: mds1 facet_host: oleg132-server facet_failover_host: oleg132-server Failover mds1 to oleg132-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 15:11:28 (1746126688) targets are mounted 15:11:28 (1746126688) facet_failover done oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399056 s, 26.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x15 lmm_fid: [0x200000403:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (40s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:11:43 (1746126703) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:11:46 (1746126706) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:11:48 (1746126708) 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00994136 s, 1.6 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:11:52 (1746126712) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:11:54 (1746126714) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:11:56 (1746126716) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x1e:0x0]: Invalid argument (22) PASS 60b (15s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:12:12 (1746126732) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps lfs migrate: error reading src bytes 0-67108864: Invalid argument (22) error: lfs migrate: /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: data copy failed: Invalid argument sanity-flr test_61a: @@@@@@ FAIL: cannot migrate /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3189:test_61a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3196:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_61a.*.1746126752.log rsync: chown "/tmp/testlogs/.sanity-flr.test_61a.debug_log.oleg132-server.1746126752.log.FnE1ld" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_61a.dmesg.oleg132-server.1746126752.log.pMAg6u" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] mdd.lustre-MDT0000.atime_diff=60 FAIL 61a (24s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:12:38 (1746126758) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x4e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (20s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:12:58 (1746126778) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr': Invalid argument (22) sanity-flr test_61c: @@@@@@ FAIL: cannot resync mirror /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3261:test_61c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3271:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_61c.*.1746126796.log rsync: chown "/tmp/testlogs/.sanity-flr.test_61c.debug_log.oleg132-server.1746126796.log.ecvzIJ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_61c.dmesg.oleg132-server.1746126796.log.2B4PZP" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:13:21 (1746126801) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 15:13:26 (1746126806) mirror create pid 23676 mirror split pid 23678 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 23676 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 23678 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 15:14:29 (1746126869) Seed set to 6870 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1b885 000002 1746126870.459282 trunc from 00000000 to 0x01b884 (0x1b885 bytes) 000006 1746126870.472609 mapread 0x01895c thru 0x01b884 (0x2f29 bytes) 000012 1746126870.493461 write 0x02be4e thru 0x02da1f (0x1bd2 bytes) 000014 1746126870.494778 mapread 0x02d215 thru 0x02da1f (0x80b bytes) 000015 1746126870.496547 mapwrite 0x01cc9d thru 0x02537c (0x86e0 bytes) 000016 1746126870.641658 mapwrite 0x007b10 thru 0x010de5 (0x92d6 bytes) 000017 1746126870.709959 trunc from 0x02da20 to 0x014958 (0x190c7 bytes) 000018 1746126870.726441 mapwrite 0x03ab22 thru 0x03ffff (0x54de bytes) 000019 1746126870.805813 mapread 0x03b17f thru 0x03b644 (0x4c6 bytes) 000025 1746126870.809609 mapwrite 0x01968d thru 0x01ea30 (0x53a4 bytes) 000027 1746126870.863445 mapwrite 0x02cc57 thru 0x038832 (0xbbdc bytes) 000029 1746126870.943056 write 0x01f1c4 thru 0x02771c (0x8559 bytes) 000034 1746126870.946208 mapread 0x028fbb thru 0x031ff8 (0x903e bytes) truncating to largest ever: 0x2e5c7 000036 1746126870.949296 trunc from 0x040000 to 0x02e5c6 (0x11a39 bytes) 000038 1746126870.965210 write 0x011c15 thru 0x01cf2c (0xb318 bytes) 000039 1746126870.974869 write 0x0160e0 thru 0x01e734 (0x8655 bytes) 000040 1746126870.976774 write 0x027996 thru 0x033c02 (0xc26d bytes) 000042 1746126870.979100 mapread 0x025ea2 thru 0x02bf45 (0x60a4 bytes) 000045 1746126870.982436 write 0x0214f0 thru 0x02753d (0x604e bytes) truncating to largest ever: 0x3e2ea 000047 1746126870.983607 trunc from 0x033c03 to 0x03e2e9 (0xa6e7 bytes) 000049 1746126871.014270 read 0x00565f thru 0x00a66f (0x5011 bytes) 000051 1746126871.015269 write 0x015680 thru 0x01becb (0x684c bytes) 000052 1746126871.016037 mapread 0x024663 thru 0x028a97 (0x4435 bytes) 000054 1746126871.018340 mapread 0x0175d3 thru 0x025155 (0xdb83 bytes) 000055 1746126871.020487 mapwrite 0x01eea9 thru 0x02671f (0x7877 bytes) 000059 1746126871.088196 trunc from 0x03e2ea to 0x02ad01 (0x135e8 bytes) 000060 1746126871.101820 write 0x012130 thru 0x01412a (0x1ffb bytes) 000061 1746126871.103537 read 0x023d6d thru 0x02ad01 (0x6f95 bytes) 000063 1746126871.104494 read 0x004b8a thru 0x011231 (0xc6a8 bytes) 000065 1746126871.105595 mapwrite 0x03a199 thru 0x03b4b0 (0x1318 bytes) 000066 1746126871.196806 mapwrite 0x027a5e thru 0x02a3fb (0x299e bytes) 000068 1746126871.252538 write 0x03b68c thru 0x03ffff (0x4974 bytes) 000069 1746126871.254713 mapwrite 0x006899 thru 0x013c2e (0xd396 bytes) 000071 1746126871.316272 mapwrite 0x0295ce thru 0x036c68 (0xd69b bytes) 000072 1746126871.369968 read 0x01e516 thru 0x02924f (0xad3a bytes) 000073 1746126871.371187 mapread 0x03566c thru 0x03ffff (0xa994 bytes) 000075 1746126871.371801 mapread 0x00104b thru 0x00ba29 (0xa9df bytes) 000079 1746126871.375849 read 0x007e78 thru 0x00d83d (0x59c6 bytes) 000083 1746126871.379018 mapwrite 0x0267c5 thru 0x02f528 (0x8d64 bytes) 000085 1746126871.446438 trunc from 0x040000 to 0x013bb8 (0x2c447 bytes) 000086 1746126871.461346 mapread 0x013189 thru 0x013bb8 (0xa30 bytes) 000087 1746126871.463448 mapread 0x01154a thru 0x013bb8 (0x266f bytes) 000088 1746126871.464908 write 0x02ea66 thru 0x035d50 (0x72eb bytes) 000089 1746126871.467027 mapwrite 0x023f20 thru 0x0288ea (0x49cb bytes) 000091 1746126871.534040 read 0x034ef5 thru 0x035d50 (0xe5c bytes) 000092 1746126871.534545 write 0x01d7eb thru 0x023bb1 (0x63c7 bytes) 000095 1746126871.538140 read 0x013fa4 thru 0x01b06b (0x70c8 bytes) truncating to largest ever: 0x3f63d 000099 1746126871.540551 trunc from 0x035d51 to 0x03f63c (0x98ec bytes) 000100 1746126871.552431 trunc from 0x03f63d to 0x00e148 (0x314f4 bytes) 000101 1746126871.565902 trunc from 0x00e149 to 0x00d424 (0xd24 bytes) 000102 1746126871.586461 trunc from 0x00d425 to 0x021270 (0x13e4c bytes) 000104 1746126871.608920 mapread 0x0125d5 thru 0x01d04e (0xaa7a bytes) 000105 1746126871.624554 trunc from 0x021271 to 0x00df73 (0x132fd bytes) 000106 1746126871.639723 mapread 0x0005b6 thru 0x006f48 (0x6993 bytes) 000108 1746126871.640688 trunc from 0x00df74 to 0x02ab53 (0x1cbe0 bytes) 000109 1746126871.654289 trunc from 0x02ab54 to 0x016bdf (0x13f74 bytes) 000110 1746126871.688852 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16be0: skipped LOG DUMP (92 total operations): 1: 1746126870.449142 SKIPPED (no operation) 2: 1746126870.459282 TRUNC_UP 0x00000 to 0x1b885 3: 1746126870.471418 SKIPPED (no operation) 4: 1746126870.472609 MAPREAD 0x1895c thru 0x1b884 (0x02f29 bytes) 5: 1746126870.492215 SKIPPED (no operation) 6: 1746126870.492416 SKIPPED (no operation) 7: 1746126870.492960 SKIPPED (no operation) 8: 1746126870.493280 SKIPPED (no operation) 9: 1746126870.493461 WRITE 0x2be4e thru 0x2da1f (0x01bd2 bytes) HOLE 10: 1746126870.494459 SKIPPED (no operation) 11: 1746126870.494778 MAPREAD 0x2d215 thru 0x2da1f (0x0080b bytes) 12: 1746126870.496547 MAPWRITE 0x1cc9d thru 0x2537c (0x086e0 bytes) 13: 1746126870.641658 MAPWRITE 0x07b10 thru 0x10de5 (0x092d6 bytes) 14: 1746126870.709959 TRUNC_DN 0x2da20 to 0x14959 15: 1746126870.726441 MAPWRITE 0x3ab22 thru 0x3ffff (0x054de bytes) 16: 1746126870.805813 MAPREAD 0x3b17f thru 0x3b644 (0x004c6 bytes) 17: 1746126870.806445 SKIPPED (no operation) 18: 1746126870.807294 SKIPPED (no operation) 19: 1746126870.807650 SKIPPED (no operation) 20: 1746126870.809609 MAPWRITE 0x1968d thru 0x1ea30 (0x053a4 bytes) 21: 1746126870.863445 MAPWRITE 0x2cc57 thru 0x38832 (0x0bbdc bytes) 22: 1746126870.942688 SKIPPED (no operation) 23: 1746126870.943056 WRITE 0x1f1c4 thru 0x2771c (0x08559 bytes) 24: 1746126870.945083 SKIPPED (no operation) 25: 1746126870.945523 SKIPPED (no operation) 26: 1746126870.946208 MAPREAD 0x28fbb thru 0x31ff8 (0x0903e bytes) 27: 1746126870.948864 SKIPPED (no operation) 28: 1746126870.949296 TRUNC_DN 0x40000 to 0x2e5c7 29: 1746126870.965210 WRITE 0x11c15 thru 0x1cf2c (0x0b318 bytes) 30: 1746126870.974869 WRITE 0x160e0 thru 0x1e734 (0x08655 bytes) 31: 1746126870.976774 WRITE 0x27996 thru 0x33c02 (0x0c26d bytes) EXTEND 32: 1746126870.978561 SKIPPED (no operation) 33: 1746126870.979100 MAPREAD 0x25ea2 thru 0x2bf45 (0x060a4 bytes) 34: 1746126870.981537 SKIPPED (no operation) 35: 1746126870.982078 SKIPPED (no operation) 36: 1746126870.982436 WRITE 0x214f0 thru 0x2753d (0x0604e bytes) 37: 1746126870.983261 SKIPPED (no operation) 38: 1746126870.983607 TRUNC_UP 0x33c03 to 0x3e2ea 39: 1746126871.013475 SKIPPED (no operation) 40: 1746126871.014270 READ 0x0565f thru 0x0a66f (0x05011 bytes) 41: 1746126871.015269 WRITE 0x15680 thru 0x1becb (0x0684c bytes) 42: 1746126871.016037 MAPREAD 0x24663 thru 0x28a97 (0x04435 bytes) 43: 1746126871.017938 SKIPPED (no operation) 44: 1746126871.018340 MAPREAD 0x175d3 thru 0x25155 (0x0db83 bytes) 45: 1746126871.020487 MAPWRITE 0x1eea9 thru 0x2671f (0x07877 bytes) 46: 1746126871.084647 SKIPPED (no operation) 47: 1746126871.086761 SKIPPED (no operation) 48: 1746126871.088196 TRUNC_DN 0x3e2ea to 0x2ad02 49: 1746126871.101820 WRITE 0x12130 thru 0x1412a (0x01ffb bytes) 50: 1746126871.103537 READ 0x23d6d thru 0x2ad01 (0x06f95 bytes) 51: 1746126871.104494 READ 0x04b8a thru 0x11231 (0x0c6a8 bytes) 52: 1746126871.105220 SKIPPED (no operation) 53: 1746126871.105595 MAPWRITE 0x3a199 thru 0x3b4b0 (0x01318 bytes) 54: 1746126871.196806 MAPWRITE 0x27a5e thru 0x2a3fb (0x0299e bytes) 55: 1746126871.252538 WRITE 0x3b68c thru 0x3ffff (0x04974 bytes) HOLE 56: 1746126871.254713 MAPWRITE 0x06899 thru 0x13c2e (0x0d396 bytes) 57: 1746126871.315621 SKIPPED (no operation) 58: 1746126871.316272 MAPWRITE 0x295ce thru 0x36c68 (0x0d69b bytes) 59: 1746126871.369968 READ 0x1e516 thru 0x2924f (0x0ad3a bytes) 60: 1746126871.371187 MAPREAD 0x3566c thru 0x3ffff (0x0a994 bytes) 61: 1746126871.371608 SKIPPED (no operation) 62: 1746126871.371801 MAPREAD 0x0104b thru 0x0ba29 (0x0a9df bytes) 63: 1746126871.374846 SKIPPED (no operation) 64: 1746126871.375522 SKIPPED (no operation) 65: 1746126871.375849 READ 0x07e78 thru 0x0d83d (0x059c6 bytes) 66: 1746126871.377854 SKIPPED (no operation) 67: 1746126871.378191 SKIPPED (no operation) 68: 1746126871.378517 SKIPPED (no operation) 69: 1746126871.379018 MAPWRITE 0x267c5 thru 0x2f528 (0x08d64 bytes) 70: 1746126871.446238 SKIPPED (no operation) 71: 1746126871.446438 TRUNC_DN 0x40000 to 0x13bb9 72: 1746126871.461346 MAPREAD 0x13189 thru 0x13bb8 (0x00a30 bytes) 73: 1746126871.463448 MAPREAD 0x1154a thru 0x13bb8 (0x0266f bytes) 74: 1746126871.464908 WRITE 0x2ea66 thru 0x35d50 (0x072eb bytes) HOLE 75: 1746126871.467027 MAPWRITE 0x23f20 thru 0x288ea (0x049cb bytes) 76: 1746126871.534040 READ 0x34ef5 thru 0x35d50 (0x00e5c bytes) 77: 1746126871.534545 WRITE 0x1d7eb thru 0x23bb1 (0x063c7 bytes) 78: 1746126871.538140 READ 0x13fa4 thru 0x1b06b (0x070c8 bytes) 79: 1746126871.538829 SKIPPED (no operation) 80: 1746126871.539421 SKIPPED (no operation) 81: 1746126871.539934 SKIPPED (no operation) 82: 1746126871.540551 TRUNC_UP 0x35d51 to 0x3f63d 83: 1746126871.552431 TRUNC_DN 0x3f63d to 0x0e149 84: 1746126871.565902 TRUNC_DN 0x0e149 to 0x0d425 85: 1746126871.586461 TRUNC_UP 0x0d425 to 0x21271 86: 1746126871.608332 SKIPPED (no operation) 87: 1746126871.608920 MAPREAD 0x125d5 thru 0x1d04e (0x0aa7a bytes) 88: 1746126871.624554 TRUNC_DN 0x21271 to 0x0df74 89: 1746126871.639723 MAPREAD 0x005b6 thru 0x06f48 (0x06993 bytes) 90: 1746126871.640688 TRUNC_UP 0x0df74 to 0x2ab54 91: 1746126871.654289 TRUNC_DN 0x2ab54 to 0x16be0 92: 1746126871.688852 MIRROR_EXTEND to 2 mirrors Using seed 6870 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746126873.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg132-server.1746126873.log.Hu1QUB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg132-server.1746126873.log.hxUiLY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 70a (6s) == sanity-flr test 71: check mirror extend parent fid ==== 15:14:38 (1746126878) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 200a: stress test ===================== 15:14:41 (1746126881) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3411744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 686304 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5552672 .. Extending file size to 7598880 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9062112 .. Extending file size to 5646464 .. Extending file size to 6545600 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7308128 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8043840 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8617824 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8688928 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8979584 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 26945 26946 26948 26952 26953 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) 192.168.201.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 0-9064448 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-9064448 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200a.sanity-flr': Invalid argument (22) sanity-flr test_200a: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3534:test_200a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3546:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200a.*.1746126947.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.debug_log.oleg132-server.1746126947.log.SYkQ4r" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.dmesg.oleg132-server.1746126947.log.G0orRY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:15:54 (1746126954) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4846944 .. Extending file size to 3668160 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8337632 .. Extending file size to 5589472 .. Extending file size to 5784320 .. Extending file size to 6736704 .. Extending file size to 7380160 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8650944 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8473920 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8919424 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8857088 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8902368 .. failed Extending file size to 9319456 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 29747 29748 29751 29752 29753 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) 192.168.201.132@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 0-9322496 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-9322496 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:06 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:05 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? S 0:36 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:13 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:04 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:13 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:11 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? R 0:03 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:11 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 46 ? S 0:00 [kworker/1:1] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 49 ? S 0:04 [kworker/2:1] 53 ? S 0:00 [kswapd0] 54 ? SN 0:00 [ksmd] 55 ? SN 0:00 [khugepaged] 56 ? S< 0:00 [nfsiod] 57 ? S< 0:00 [crypto] 65 ? S< 0:00 [kthrotld] 67 ? S< 0:00 [kmpath_rdacd] 68 ? S< 0:00 [kpsmoused] 70 ? S< 0:00 [ipv6_addrconf] 83 ? S< 0:00 [deferwq] 127 ? S 0:01 [kworker/3:2] 293 ? S< 0:00 [ata_sff] 303 ? S 0:00 [scsi_eh_0] 305 ? S< 0:00 [scsi_tmf_0] 306 ? S 0:00 [scsi_eh_1] 309 ? S< 0:00 [scsi_tmf_1] 314 ? S 0:00 [kworker/u8:3] 452 ? Ss 0:02 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 454 ? S< 0:02 [nbd0] 483 ? S 0:00 [jbd2/nbd0-8] 485 ? S< 0:00 [ext4-rsv-conver] 525 ? S< 0:00 [kworker/2:1H] 526 ? S< 0:00 [kworker/0:1H] 571 ? Ss 0:02 /usr/lib/systemd/systemd-journald 604 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 726 ? S< 0:00 [kvm-irqfd-clean] 733 ? S< 0:00 [edac-poller] 744 ? S< 0:00 [kworker/3:1H] 784 ? S 0:00 [nfsv4.1-svc] 819 ? Ss 0:01 /usr/lib/systemd/systemd-logind 821 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 824 ? Ssl 0:00 /usr/sbin/gssproxy -D 832 ? Ssl 0:01 /usr/sbin/NetworkManager --no-daemon 848 ? Ss 0:00 /usr/sbin/sshd -D 864 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 867 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 868 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 882 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 889 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-e078118f-4129-4075-86e5-754bd02f6347-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 916 ? Ssl 0:02 /usr/sbin/rsyslogd -n 966 ? S< 0:00 [kworker/1:1H] 1150 ? Ss 0:00 sshd: root@pts/0 1195 pts/0 Ss+ 0:01 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1984 ? S 0:00 [ptlrpc_hr00_000] 1985 ? S 0:00 [ptlrpc_hr01_000] 1986 ? S 0:00 [socknal_cd00] 1987 ? S 0:00 [socknal_cd01] 1988 ? S 0:00 [socknal_cd02] 1989 ? S 0:00 [socknal_cd03] 1990 ? S 0:00 [socknal_reaper] 1991 ? S 0:00 [socknal_sd00_00] 1992 ? S 0:00 [socknal_sd00_01] 1993 ? S 0:18 [socknal_sd01_00] 1994 ? S 0:18 [socknal_sd01_01] 1995 ? S 0:00 [acceptor_001] 1996 ? S 0:00 [monitor_thread] 1997 ? S 0:00 [lnet_discovery] 1998 ? S 0:00 [ptlrpcd_rcv] 1999 ? S 0:06 [ptlrpcd_00_00] 2000 ? S 0:06 [ptlrpcd_00_01] 2001 ? S 0:03 [ptlrpcd_01_00] 2002 ? S 0:03 [ptlrpcd_01_01] 6990 ? S 0:00 [kworker/u8:4] 7081 pts/0 S+ 0:02 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 7745 ? S 0:00 [ldlm_cb00_000] 7746 ? S 0:00 [ldlm_cb00_001] 7747 ? S 0:01 [ldlm_cb01_000] 7748 ? S 0:01 [ldlm_cb01_001] 7749 ? S 0:00 [ldlm_cn00_000] 7750 ? S 0:00 [ldlm_cn00_001] 7751 ? S 0:00 [ldlm_cn00_002] 7752 ? S 0:00 [ldlm_cn01_000] 7753 ? S 0:00 [ldlm_cn01_001] 7754 ? S 0:00 [ldlm_cn01_002] 7755 ? S 0:01 [ldlm_bl_01] 7756 ? S 0:01 [ldlm_bl_02] 7757 ? S 0:00 [ldlm_elt] 7758 ? S 0:00 [ll_cfg_requeue] 15478 ? S 0:01 [ldlm_bl_03] 15479 ? S 0:01 [ldlm_cb01_002] 17069 ? S 0:03 [kworker/3:0] 19386 ? S 0:00 [kworker/0:1] 21581 ? S 0:00 [kworker/2:2] 21913 ? S 0:00 [kworker/1:0] 23743 ? S 0:00 [kworker/0:0] 23763 ? S 0:01 [ldlm_bl_04] 24695 ? S 0:00 [kworker/1:3] 24812 ? S 0:00 [ldlm_cb00_002] 25774 ? S 0:00 [kworker/2:0] 27090 ? S 0:00 [ldlm_bl_05] 27091 ? S 0:00 [ldlm_bl_06] 27092 ? S 0:00 [ldlm_bl_07] 27093 ? S 0:00 [ldlm_bl_08] 29517 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 29518 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg132-client.log 31550 ? S 0:00 /usr/lib/systemd/systemd-udevd 31572 pts/0 R+ 0:00 ps ax 32756 ? S 0:00 [kworker/u8:2] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746127019.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg132-server.1746127019.log.TR5CdP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg132-server.1746127019.log.xe82gp" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 200b (68s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:17:06 (1746127026) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3757056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3755008 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 26624 7512064 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 12288 3757056 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 14336 3755008 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 26624 7512064 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0934965 s, 22.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.273116 s, 7.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) PASS 200c (10s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:17:17 (1746127037) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0735358 s, 28.5 MB/s PASS 202 (4s) == sanity-flr test 203: mirror file preserve mirror ID === 15:17:21 (1746127041) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x2b8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:17:25 (1746127045) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.117967 s, 17.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x350:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:17:30 (1746127050) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.101277 s, 20.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x351:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x351:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x352:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:17:34 (1746127054) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0724831 s, 28.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x353:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x353:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x354:0x0] } PASS 204c (26s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:18:00 (1746127080) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg132-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0545677 s, 19.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x356:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x356:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x355:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x353:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:18:48 (1746127128) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg132-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x354:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x355:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.084981 s, 24.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x354:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x355:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x357:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x354:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x355:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x357:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:19:37 (1746127177) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg132-server mds-ost sync done. Creating new pool test_204f oleg132-server: Pool lustre.test_204f created Adding targets to pool oleg132-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg132-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0664145 s, 31.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x356:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x358:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x356:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x358:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg132-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg132-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg132-server: Pool lustre.test_204f destroyed PASS 204f (61s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:20:38 (1746127238) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:20:42 (1746127242) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:20:45 (1746127245) oleg132-server: Pool lustre.test_206 created Adding targets to pool oleg132-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg132-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg132-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg132-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg132-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:21:00 (1746127260) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0558296 s, 18.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0285294 s, 36.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x680744a9 Mirror 3: 0x680744a9 PASS 207 (6s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:21:06 (1746127266) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:21:08 (1746127268) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:21:10 (1746127270) Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.102379 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x16d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ad0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ad0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x11b0000 Read this (4 bytes): BBBB 192.168.201.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg132-client.virtnet /mnt/lustre2 (opts:) PASS 209a (23s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:21:33 (1746127293) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0197813 s, 414 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:21:37 (1746127297) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0312679 s, 33.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x2dd lmm_fid: [0x200000403:0x2dd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x2dd lmm_fid: [0x200000403:0x2dd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x363:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205306057437 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-05-01 15:21:38.000000000 -0400 Modify: 2025-05-01 15:21:38.000000000 -0400 Change: 2025-05-01 15:21:39.000000000 -0400 Birth: - PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:21:42 (1746127302) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:21:44 (1746127304) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.512772 s, 79.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0696721 s, 58.8 kB/s size after second write -rw-r--r-- 1 root root 4096 May 1 15:21 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000403:0x2df:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 1 15:21 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test complete, duration 1072 sec =========== 15:21:49 (1746127309) sanity-flr: FAIL: test_0g error resync-ing file '/mnt/lustre/f0g.sanity-flr' sanity-flr: FAIL: test_0h error resync-ing file '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr' sanity-flr: FAIL: test_0j resync /mnt/lustre/f0j.sanity-flr failed sanity-flr: FAIL: test_36d mirror resync /mnt/lustre/f36d.sanity-flr failed sanity-flr: FAIL: test_37 1: mismatch: \'a41fdec422c1f582e6e57e00873608f0 -\' vs. \'ff3fbf19d33969a6ee1c3e89783650bf -\' sanity-flr: FAIL: test_38 valid mirror doesn't exist sanity-flr: FAIL: test_44b split --mirror-id=1 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr should succeed sanity-flr: FAIL: test_61a cannot migrate /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr sanity-flr: FAIL: test_61c cannot resync mirror /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sanity-flr: FAIL: test_70a fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed sanity-flr: FAIL: test_200a final resync failed sanity-flr: FAIL: test_200b final resync failed === sanity-flr: start cleanup 15:21:50 (1746127310) === === sanity-flr: finish cleanup 15:21:54 (1746127314) ===