-----============= acceptance-small: sanity-flr ============----- Wed Apr 30 21:54:59 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg227-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex: No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:55:04 (1746064504) === oleg227-client.virtnet: executing check_config_client /mnt/lustre oleg227-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg227-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b3dc9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3dc9800.idle_timeout=debug disable quota as required oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:55:24 (1746064524) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanity-flr test 0a: lfs mirror create with -N option == 21:55:25 (1746064525) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:55:36 (1746064536) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:55:38 (1746064538) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:55:40 (1746064540) PASS 0d (13s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:55:53 (1746064553) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:55:55 (1746064555) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 21:55:57 (1746064557) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f0g.sanity-flr': Invalid argument (22) sanity-flr test_0g: @@@@@@ FAIL: error resync-ing file '/mnt/lustre/f0g.sanity-flr' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:740:test_0g() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:751:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0g.*.1746064560.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0g.debug_log.oleg227-server.1746064560.log.1YW3YS" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0g.dmesg.oleg227-server.1746064560.log.WzmEJH" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:56:03 (1746064563) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead lfs mirror mirror: fail to pread 0-4096 of mirror 2: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 2: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr': Invalid argument (22) sanity-flr test_0h: @@@@@@ FAIL: error resync-ing file '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:793:test_0h() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:831:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0h.*.1746064566.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0h.debug_log.oleg227-server.1746064566.log.zOxq82" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0h.dmesg.oleg227-server.1746064566.log.lOB2a3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:56:08 (1746064568) lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f0j.sanity-flr': Invalid argument (22) sanity-flr test_0j: @@@@@@ FAIL: resync /mnt/lustre/f0j.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:837:test_0j() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:849:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_0j.*.1746064570.log rsync: chown "/tmp/testlogs/.sanity-flr.test_0j.debug_log.oleg227-server.1746064570.log.jLul3m" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_0j.dmesg.oleg227-server.1746064570.log.xaXcSx" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:56:12 (1746064572) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:56:14 (1746064574) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 21:56:18 (1746064578) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:56:22 (1746064582) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:56:26 (1746064586) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:56:30 (1746064590) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:56:37 (1746064597) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:56:41 (1746064601) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.25314 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0391261 s, 26.8 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:56:46 (1746064606) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB) copied, 0.183508 s, 40.0 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:56:52 (1746064612) Stopping /mnt/lustre-ost1 (opts:) on oleg227-server oleg227-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid in DISCONN state after 0 sec Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg227-server: oleg227-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg227-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg227-server oleg227-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid in DISCONN state after 1 sec Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg227-server: oleg227-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg227-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (40s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:57:32 (1746064652) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.141817 s, 37.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg227-server oleg227-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg227-server: oleg227-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg227-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:57:52 (1746064672) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg227-server oleg227-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg227-server: oleg227-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg227-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg227-server oleg227-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg227-server: oleg227-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg227-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (37s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:58:29 (1746064709) 25+0 records in 25+0 records out 52428800 bytes (52 MB) copied, 1.56395 s, 33.5 MB/s -rw-r--r-- 1 root root 52428800 Apr 30 21:58 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.755s user 0m0.000s sys 0m0.133s Stopping /mnt/lustre-ost1 (opts:) on oleg227-server oleg227-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3dc9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.597s user 0m0.000s sys 0m0.136s Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg227-server: oleg227-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg227-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg227-server oleg227-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.592s user 0m0.000s sys 0m0.136s Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg227-server: oleg227-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg227-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b3dc9800.ost_server_uuid in FULL state after 0 sec PASS 33b (42s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:59:11 (1746064751) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:59:13 (1746064753) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:59:15 (1746064755) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 21:59:16 (1746064756) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 21:59:21 (1746064761) 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.320587 s, 39.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:59:28 (1746064768) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0281057 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015463 s, 67.8 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:59:32 (1746064772) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:59:37 (1746064777) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00109983 s, 931 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00176072 s, 582 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f36d.sanity-flr': Invalid argument (22) sanity-flr test_36d: @@@@@@ FAIL: mirror resync /mnt/lustre/f36d.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1643:test_36d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1674:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_36d.*.1746064780.log rsync: chown "/tmp/testlogs/.sanity-flr.test_36d.debug_log.oleg227-server.1746064780.log.29sFwd" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_36d.dmesg.oleg227-server.1746064780.log.jfRbZk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 36d (6s) == sanity-flr test 37: mirror I/O API verification ======= 21:59:43 (1746064783) create test files with size 15749810 .. f1e627b38335c24a5ca5a5d71092ea50 - f47350db83ce9f4a76d909e3225b1050 - 8113589e137f1bd0af40895b9f252019 - 3 Verifying mirror read .. lfs mirror mirror: fail to pread 12582912-4194304 of mirror 1: Invalid argument (22) lfs read: fail to read data from mirror 1: Invalid argument sanity-flr test_37: @@@@@@ FAIL: 1: mismatch: \'f1e627b38335c24a5ca5a5d71092ea50 -\' vs. \'0808d7dbb34e1f0432dbc3c72e4d3271 -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1730:test_37() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1759:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_37.*.1746064790.log rsync: chown "/tmp/testlogs/.sanity-flr.test_37.debug_log.oleg227-server.1746064790.log.eGWawu" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_37.dmesg.oleg227-server.1746064790.log.RmL1KZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 37 (10s) == sanity-flr test 38: resync ============================ 21:59:53 (1746064793) 4 mirror IDs: 1 2 3 4 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs read: fail to read data from mirror 1: Invalid argument lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument lfs mirror mirror: fail to pread 0-4194304 of mirror 3: Invalid argument (22) lfs read: fail to read data from mirror 3: Invalid argument lfs mirror mirror: fail to pread 0-4194304 of mirror 4: Invalid argument (22) lfs read: fail to read data from mirror 4: Invalid argument sanity-flr test_38: @@@@@@ FAIL: valid mirror doesn't exist Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1811:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:1839:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_38.*.1746064797.log rsync: chown "/tmp/testlogs/.sanity-flr.test_38.debug_log.oleg227-server.1746064797.log.MJQK90" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_38.dmesg.oleg227-server.1746064797.log.1LjgfN" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:59:59 (1746064799) PASS 39 (4s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:00:03 (1746064803) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0525074 s, 20.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11e lmm_fid: 0x200000402:0x11e:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x36:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11e lmm_fid: 0x200000402:0x11e:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x37:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11e lmm_fid: 0x200000402:0x11e:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11e lmm_fid: 0x200000402:0x11e:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11e lmm_fid: 0x200000402:0x11e:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11e lmm_fid: 0x200000402:0x11e:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x11e lmm_fid: 0x200000402:0x11e:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127648 s, 82.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x120 lmm_fid: 0x200000402:0x120:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x120 lmm_fid: 0x200000402:0x120:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x120 lmm_fid: 0x200000402:0x120:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x120 lmm_fid: 0x200000402:0x120:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x38:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x120 lmm_fid: 0x200000402:0x120:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x120 lmm_fid: 0x200000402:0x120:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x120 lmm_fid: 0x200000402:0x120:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 22:00:07 (1746064807) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0411113 s, 51.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.128258 s, 32.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0852957 s, 24.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 22:00:12 (1746064812) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 22:00:14 (1746064814) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:00:15 (1746064815) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123545 s, 84.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 22:00:18 (1746064818) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.152238 s, 20.7 MB/s PASS 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:00:22 (1746064822) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x12:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x13:0x0] } split mirror_id 1 id 65537 sanity-flr test_44b: @@@@@@ FAIL: split --mirror-id=1 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr should succeed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2290:test_44b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2308:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_44b.*.1746064825.log rsync: chown "/tmp/testlogs/.sanity-flr.test_44b.debug_log.oleg227-server.1746064825.log.AP7g5z" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_44b.dmesg.oleg227-server.1746064825.log.idlRVy" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 44b (5s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:00:28 (1746064828) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.132215 s, 79.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x12c:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg227-client.virtnet /mnt/lustre (opts:) Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:00:43 (1746064843) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8800a9085000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.179037 s, 58.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg227-client.virtnet /mnt/lustre (opts:) Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8800a9ddb000.xattr_cache=1 PASS 44e (8s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:00:51 (1746064851) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy compare PASS 45 (17s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:01:08 (1746064868) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:01:22 (1746064882) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 22:01:24 (1746064884) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.160877 s, 19.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB) copied, 0.127751 s, 24.6 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: a72e9b64e9e9065d356a107b623cf4ef - mirror 1 checksum: 0348f1f917401140c3c58490732c4684 - mirror 2 checksum: a72e9b64e9e9065d356a107b623cf4ef - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 30 22:01:25 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa lmm_fid: [0x200000404:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Apr 30 22:01:25 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa lmm_fid: [0x200000404:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a72e9b64e9e9065d356a107b623cf4ef - mirror 1 checksum: 0348f1f917401140c3c58490732c4684 - mirror 2 checksum: 0348f1f917401140c3c58490732c4684 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 30 22:01:25 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa lmm_fid: [0x200000404:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4c:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Apr 30 22:01:25 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0xa lmm_fid: [0x200000404:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4d:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:01:29 (1746064889) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125899 s, 83.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 527360.. 528383: 1024: 0001: net,eof 1: 0.. 1023: 525312.. 526335: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:01:32 (1746064892) Failing mds1 on oleg227-server Stopping /mnt/lustre-mds1 (opts:) on oleg227-server 22:01:41 (1746064901) shut down facet: mds1 facet_host: oleg227-server facet_failover_host: oleg227-server Failover mds1 to oleg227-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 22:01:57 (1746064917) targets are mounted 22:01:57 (1746064917) facet_failover done oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206262 s, 50.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x10 lmm_fid: [0x200000404:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4f:0x0] } PASS 50A (33s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:02:06 (1746064926) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0208047 s, 984 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.00769076 s, 2.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument sanity-flr test_50a: @@@@@@ FAIL: data mismatch: \'b46fdc980b505c0e8074fc7c58bc788b -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2846:test_50a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2895:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_50a.*.1746064928.log rsync: chown "/tmp/testlogs/.sanity-flr.test_50a.debug_log.oleg227-server.1746064928.log.TdyqV8" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_50a.dmesg.oleg227-server.1746064928.log.QHM3gu" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:02:11 (1746064931) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0284563 s, 720 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument sanity-flr test_50b: @@@@@@ FAIL: data mismatch: \'5aa4604d16ac73000c99b67cca4ecbd9 -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2939:test_50b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:2983:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_50b.*.1746064933.log rsync: chown "/tmp/testlogs/.sanity-flr.test_50b.debug_log.oleg227-server.1746064933.log.YETXfO" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_50b.dmesg.oleg227-server.1746064933.log.J4edBk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:02:15 (1746064935) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.0102195 s, 1.6 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:02:19 (1746064939) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB) copied, 0.0317349 s, 645 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 lfs mirror mirror: fail to pread 1048576-20480 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 1048576-20480 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d50d.sanity-flr/f50d.sanity-flr': Invalid argument (22) lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs read: fail to read data from mirror 2: Invalid argument sanity-flr test_50d: @@@@@@ FAIL: data mismatch: \'407c530a3cfca55b8bbbd3863159e7e3 -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3054:test_50d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3066:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_50d.*.1746064941.log rsync: chown "/tmp/testlogs/.sanity-flr.test_50d.debug_log.oleg227-server.1746064941.log.97UHvL" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_50d.dmesg.oleg227-server.1746064941.log.5njUHB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:02:23 (1746064943) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00660436 s, 620 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0252689 s, 162 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:02:26 (1746064946) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x11:0x0]: Invalid argument (22) PASS 60b (11s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:02:37 (1746064957) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps lfs migrate: error reading src bytes 0-67108864: Invalid argument (22) error: lfs migrate: /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: data copy failed: Invalid argument sanity-flr test_61a: @@@@@@ FAIL: cannot migrate /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3189:test_61a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3196:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_61a.*.1746064976.log rsync: chown "/tmp/testlogs/.sanity-flr.test_61a.debug_log.oleg227-server.1746064976.log.kRESPA" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_61a.dmesg.oleg227-server.1746064976.log.KweXRW" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 FAIL 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:02:59 (1746064979) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:03:18 (1746064998) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps lfs mirror mirror: fail to pread 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 0-4096 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr': Invalid argument (22) sanity-flr test_61c: @@@@@@ FAIL: cannot resync mirror /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3261:test_61c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3271:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_61c.*.1746065015.log rsync: chown "/tmp/testlogs/.sanity-flr.test_61c.debug_log.oleg227-server.1746065015.log.1TJA2j" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_61c.dmesg.oleg227-server.1746065015.log.Gmaqpj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:03:37 (1746065017) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 22:03:41 (1746065021) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 26180 mirror split pid 26181 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26180 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4880: 26181 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &>/dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 22:04:45 (1746065085) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 5086 000001 1746065086.589752 write 0x0256ac thru 0x029bfe (0x4553 bytes) fallocating to largest ever: 0x16b74 000002 1746065086.596453 fallocate 0x00c463 thru 0x016b73 (0xa711 bytes) zero_range to largest ever: 0x29bff 000003 1746065086.600561 trunc from 0x023e38 to 0x029bfe (0x5dc7 bytes) 000004 1746065086.609298 fallocate 0x009293 thru 0x0177c9 (0xe537 bytes) 000006 1746065086.612750 mapread 0x0209f2 thru 0x025f41 (0x5550 bytes) 000007 1746065086.626715 mapwrite 0x03fcca thru 0x03ffff (0x336 bytes) 000008 1746065086.670143 trunc from 0x01ba21 to 0x028f8e (0xd56e bytes) truncating to largest ever: 0x3293a 000009 1746065086.674623 trunc from 0x040000 to 0x032939 (0xd6c6 bytes) 000010 1746065086.692103 mapread 0x004240 thru 0x00da1a (0x97db bytes) 000011 1746065086.704249 mapwrite 0x0106a4 thru 0x015af1 (0x544e bytes) 000013 1746065086.720892 mapwrite 0x03cd42 thru 0x03ffff (0x32be bytes) 000014 1746065086.750628 mapwrite 0x00b5fe thru 0x0172d4 (0xbcd7 bytes) 000015 1746065086.772585 mapwrite 0x03fe81 thru 0x03ffff (0x17f bytes) 000016 1746065086.781630 read 0x0244f0 thru 0x02a89b (0x63ac bytes) 000017 1746065086.786279 read 0x01706a thru 0x0171c7 (0x15e bytes) truncating to largest ever: 0x39d06 000019 1746065086.787239 trunc from 0x040000 to 0x039d05 (0x62fa bytes) 000020 1746065086.801022 trunc from 0x00145f to 0x00b410 (0x9fb2 bytes) 000021 1746065086.805956 trunc from 0x039d06 to 0x028a73 (0x11292 bytes) 000022 1746065086.816944 trunc from 0x00f042 to 0x01c5b7 (0xd576 bytes) 000023 1746065086.821350 mapwrite 0x0095b8 thru 0x009fa4 (0x9ed bytes) 000024 1746065086.837633 punch from 0x0231f4 to 0x0236c6 (0x4d3 bytes) 000026 1746065086.841636 write 0x0163c1 thru 0x025348 (0xef88 bytes) 000027 1746065086.842697 read 0x023ff6 thru 0x025c4f (0x1c5a bytes) fallocating to largest ever: 0x40000 000028 1746065086.843085 fallocate 0x03f3c7 thru 0x03ffff (0xc39 bytes) 000029 1746065086.847459 read 0x00e99c thru 0x014308 (0x596d bytes) 000030 1746065086.847850 write 0x0322e0 thru 0x03fd4b (0xda6c bytes) 000031 1746065086.856659 trunc from 0x012f16 to 0x01e089 (0xb174 bytes) 000032 1746065086.872549 mapread 0x01d0ed thru 0x02041d (0x3331 bytes) 000034 1746065086.891188 mapread 0x02a446 thru 0x0370c6 (0xcc81 bytes) 000035 1746065086.891781 trunc from 0x016361 to 0x01789a (0x153a bytes) 000036 1746065086.896264 trunc from 0x00f15e to 0x01e10f (0xefb2 bytes) 000037 1746065086.901367 mapwrite 0x015057 thru 0x01d803 (0x87ad bytes) 000038 1746065086.972269 trunc from 0x014e46 to 0x024330 (0xf4eb bytes) 000039 1746065086.980651 mapwrite 0x02161b thru 0x0287fa (0x71e0 bytes) 000040 1746065087.025780 write 0x01d75c thru 0x01efc4 (0x1869 bytes) 000041 1746065087.031745 fallocate 0x008e4e thru 0x00bec6 (0x3079 bytes) 000042 1746065087.035161 read 0x01ddda thru 0x0214f3 (0x371a bytes) 000043 1746065087.040601 mapread 0x034b53 thru 0x03ffff (0xb4ad bytes) 000044 1746065087.041174 trunc from 0x027b2b to 0x0335c6 (0xba9c bytes) 000045 1746065087.045579 punch from 0x03bd2e to 0x03ffff (0x42d2 bytes) 000046 1746065087.049491 fallocate 0x002594 thru 0x00f408 (0xce75 bytes) 000049 1746065087.054838 fallocate 0x0172ca thru 0x01a8e5 (0x361c bytes) 000050 1746065087.058319 punch from 0x03c9e8 to 0x03cd3c (0x355 bytes) 000051 1746065087.060966 read 0x00e3de thru 0x016481 (0x80a4 bytes) 000052 1746065087.061901 write 0x025321 thru 0x031867 (0xc547 bytes) 000053 1746065087.067805 trunc from 0x040000 to 0x0106be (0x2f941 bytes) 000054 1746065087.082236 read 0x007453 thru 0x007900 (0x4ae bytes) 000055 1746065087.082642 trunc from 0x001658 to 0x00c044 (0xa9ed bytes) 000059 1746065087.088475 trunc from 0x00536d to 0x00cf46 (0x7bda bytes) 000061 1746065087.093842 mapread 0x00e27e thru 0x0106be (0x2441 bytes) 000062 1746065087.094778 trunc from 0x0106bf to 0x00dae4 (0x2bda bytes) 000063 1746065087.108226 write 0x02a39c thru 0x02ae3b (0xaa0 bytes) 000064 1746065087.108915 fallocate 0x0305ab thru 0x03267e (0x20d4 bytes) 000065 1746065087.115507 mapwrite 0x004501 thru 0x006168 (0x1c68 bytes) 000067 1746065087.142913 read 0x01781c thru 0x024112 (0xc8f7 bytes) 000068 1746065087.143449 trunc from 0x02ae3c to 0x01b1e3 (0xfc58 bytes) 000069 1746065087.159371 trunc from 0x01b1e4 to 0x02acd3 (0xfaf0 bytes) 000070 1746065087.171101 trunc from 0x0210c4 to 0x02acd3 (0x9c10 bytes) 000071 1746065087.175107 read 0x017674 thru 0x0268a3 (0xf230 bytes) 000072 1746065087.181003 mapread 0x00550b thru 0x005cba (0x7b0 bytes) 000073 1746065087.182188 punch from 0x025cbf to 0x02acd3 (0x5015 bytes) 000074 1746065087.185735 read 0x010b4c thru 0x013123 (0x25d8 bytes) 000075 1746065087.186495 trunc from 0x01b77d to 0x02154b (0x5dcf bytes) 000076 1746065087.189538 punch from 0x01b2e6 to 0x025338 (0xa053 bytes) 000077 1746065087.192967 mapread 0x01176d thru 0x01bd8c (0xa620 bytes) 000078 1746065087.199805 trunc from 0x02acd4 to 0x027420 (0x38b3 bytes) 000079 1746065087.210689 mapread 0x00c25e thru 0x01b321 (0xf0c4 bytes) 000080 1746065087.212232 mapwrite 0x031303 thru 0x037b9f (0x689d bytes) 000081 1746065087.248280 trunc from 0x0142f6 to 0x015bb8 (0x18c3 bytes) 000082 1746065087.250850 punch from 0x02554c to 0x030761 (0xb216 bytes) 000084 1746065087.254433 write 0x00938e thru 0x00dd3e (0x49b1 bytes) 000085 1746065087.255202 trunc from 0x037ba0 to 0x00043d (0x37762 bytes) 000086 1746065087.269490 mapread 0x000350 thru 0x00043d (0xee bytes) 000087 1746065087.271717 read 0x0001fa thru 0x00043d (0x244 bytes) 000088 1746065087.274778 trunc from 0x0003cd to 0x00043d (0x71 bytes) 000089 1746065087.279189 punch from 0x0002c1 to 0x00043d (0x17d bytes) 000090 1746065087.288083 MIRROR_EXTEND to 2 mirrors lfs mirror mirror: fail to pread 0-4194304 of mirror 2: Invalid argument (22) lfs verify: failed to read data from mirror 2: Invalid argument. mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22 lfs mirror mirror: fail to pread 0-4194304 of mirror 1: Invalid argument (22) lfs verify: failed to read data from mirror 1: Invalid argument. lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x43e: skipped LOG DUMP (77 total operations): 1: 1746065086.589752 WRITE 0x256ac thru 0x29bfe (0x04553 bytes) HOLE 2: 1746065086.596453 FALLOC 0x0c463 thru 0x16b74 (0x0a711 bytes)INTERIOR 3: 1746065086.600561 ZERO 0x23e38 thru 0x29bfe (0x05dc7 bytes) 4: 1746065086.609298 FALLOC 0x09293 thru 0x177ca (0x0e537 bytes)INTERIOR 5: 1746065086.612750 MAPREAD 0x209f2 thru 0x25f41 (0x05550 bytes) 6: 1746065086.626715 MAPWRITE 0x3fcca thru 0x3ffff (0x00336 bytes) 7: 1746065086.670143 ZERO 0x1ba21 thru 0x28f8e (0x0d56e bytes) 8: 1746065086.674623 TRUNC_DN 0x40000 to 0x3293a 9: 1746065086.692103 MAPREAD 0x04240 thru 0x0da1a (0x097db bytes) 10: 1746065086.704249 MAPWRITE 0x106a4 thru 0x15af1 (0x0544e bytes) 11: 1746065086.720892 MAPWRITE 0x3cd42 thru 0x3ffff (0x032be bytes) 12: 1746065086.750628 MAPWRITE 0x0b5fe thru 0x172d4 (0x0bcd7 bytes) 13: 1746065086.772585 MAPWRITE 0x3fe81 thru 0x3ffff (0x0017f bytes) 14: 1746065086.781630 READ 0x244f0 thru 0x2a89b (0x063ac bytes) 15: 1746065086.786279 READ 0x1706a thru 0x171c7 (0x0015e bytes) 16: 1746065086.787239 TRUNC_DN 0x40000 to 0x39d06 17: 1746065086.801022 ZERO 0x0145f thru 0x0b410 (0x09fb2 bytes) 18: 1746065086.805956 TRUNC_DN 0x39d06 to 0x28a74 19: 1746065086.816944 ZERO 0x0f042 thru 0x1c5b7 (0x0d576 bytes) 20: 1746065086.821350 MAPWRITE 0x095b8 thru 0x09fa4 (0x009ed bytes) 21: 1746065086.837633 PUNCH 0x231f4 thru 0x236c6 (0x004d3 bytes) 22: 1746065086.841636 WRITE 0x163c1 thru 0x25348 (0x0ef88 bytes) 23: 1746065086.842697 READ 0x23ff6 thru 0x25c4f (0x01c5a bytes) 24: 1746065086.843085 FALLOC 0x3f3c7 thru 0x40000 (0x00c39 bytes)EXTENDING 25: 1746065086.847459 READ 0x0e99c thru 0x14308 (0x0596d bytes) 26: 1746065086.847850 WRITE 0x322e0 thru 0x3fd4b (0x0da6c bytes) 27: 1746065086.856659 ZERO 0x12f16 thru 0x1e089 (0x0b174 bytes) 28: 1746065086.872549 MAPREAD 0x1d0ed thru 0x2041d (0x03331 bytes) 29: 1746065086.891188 MAPREAD 0x2a446 thru 0x370c6 (0x0cc81 bytes) 30: 1746065086.891781 ZERO 0x16361 thru 0x1789a (0x0153a bytes) 31: 1746065086.896264 ZERO 0x0f15e thru 0x1e10f (0x0efb2 bytes) 32: 1746065086.901367 MAPWRITE 0x15057 thru 0x1d803 (0x087ad bytes) 33: 1746065086.972269 ZERO 0x14e46 thru 0x24330 (0x0f4eb bytes) 34: 1746065086.980651 MAPWRITE 0x2161b thru 0x287fa (0x071e0 bytes) 35: 1746065087.025780 WRITE 0x1d75c thru 0x1efc4 (0x01869 bytes) 36: 1746065087.031745 FALLOC 0x08e4e thru 0x0bec7 (0x03079 bytes)INTERIOR 37: 1746065087.035161 READ 0x1ddda thru 0x214f3 (0x0371a bytes) 38: 1746065087.040601 MAPREAD 0x34b53 thru 0x3ffff (0x0b4ad bytes) 39: 1746065087.041174 ZERO 0x27b2b thru 0x335c6 (0x0ba9c bytes) 40: 1746065087.045579 PUNCH 0x3bd2e thru 0x3ffff (0x042d2 bytes) 41: 1746065087.049491 FALLOC 0x02594 thru 0x0f409 (0x0ce75 bytes)INTERIOR 42: 1746065087.054838 FALLOC 0x172ca thru 0x1a8e6 (0x0361c bytes)INTERIOR 43: 1746065087.058319 PUNCH 0x3c9e8 thru 0x3cd3c (0x00355 bytes) 44: 1746065087.060966 READ 0x0e3de thru 0x16481 (0x080a4 bytes) 45: 1746065087.061901 WRITE 0x25321 thru 0x31867 (0x0c547 bytes) 46: 1746065087.067805 TRUNC_DN 0x40000 to 0x106bf 47: 1746065087.082236 READ 0x07453 thru 0x07900 (0x004ae bytes) 48: 1746065087.082642 ZERO 0x01658 thru 0x0c044 (0x0a9ed bytes) 49: 1746065087.088475 ZERO 0x0536d thru 0x0cf46 (0x07bda bytes) 50: 1746065087.093842 MAPREAD 0x0e27e thru 0x106be (0x02441 bytes) 51: 1746065087.094778 TRUNC_DN 0x106bf to 0x0dae5 52: 1746065087.108226 WRITE 0x2a39c thru 0x2ae3b (0x00aa0 bytes) HOLE 53: 1746065087.108915 FALLOC 0x305ab thru 0x3267f (0x020d4 bytes)PAST_EOF 54: 1746065087.115507 MAPWRITE 0x04501 thru 0x06168 (0x01c68 bytes) 55: 1746065087.142913 READ 0x1781c thru 0x24112 (0x0c8f7 bytes) 56: 1746065087.143449 TRUNC_DN 0x2ae3c to 0x1b1e4 57: 1746065087.159371 TRUNC_UP 0x1b1e4 to 0x2acd4 58: 1746065087.171101 ZERO 0x210c4 thru 0x2acd3 (0x09c10 bytes) 59: 1746065087.175107 READ 0x17674 thru 0x268a3 (0x0f230 bytes) 60: 1746065087.181003 MAPREAD 0x0550b thru 0x05cba (0x007b0 bytes) 61: 1746065087.182188 PUNCH 0x25cbf thru 0x2acd3 (0x05015 bytes) 62: 1746065087.185735 READ 0x10b4c thru 0x13123 (0x025d8 bytes) 63: 1746065087.186495 ZERO 0x1b77d thru 0x2154b (0x05dcf bytes) 64: 1746065087.189538 PUNCH 0x1b2e6 thru 0x25338 (0x0a053 bytes) 65: 1746065087.192967 MAPREAD 0x1176d thru 0x1bd8c (0x0a620 bytes) 66: 1746065087.199805 TRUNC_DN 0x2acd4 to 0x27421 67: 1746065087.210689 MAPREAD 0x0c25e thru 0x1b321 (0x0f0c4 bytes) 68: 1746065087.212232 MAPWRITE 0x31303 thru 0x37b9f (0x0689d bytes) 69: 1746065087.248280 ZERO 0x142f6 thru 0x15bb8 (0x018c3 bytes) 70: 1746065087.250850 PUNCH 0x2554c thru 0x30761 (0x0b216 bytes) 71: 1746065087.254433 WRITE 0x0938e thru 0x0dd3e (0x049b1 bytes) 72: 1746065087.255202 TRUNC_DN 0x37ba0 to 0x0043e 73: 1746065087.269490 MAPREAD 0x00350 thru 0x0043d (0x000ee bytes) 74: 1746065087.271717 READ 0x001fa thru 0x0043d (0x00244 bytes) 75: 1746065087.274778 ZERO 0x003cd thru 0x0043d (0x00071 bytes) 76: 1746065087.279189 PUNCH 0x002c1 thru 0x0043d (0x0017d bytes) 77: 1746065087.288083 MIRROR_EXTEND to 2 mirrors Using seed 5086 Correct content saved for comparison (maybe hexdump "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr" vs "/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood") sanity-flr test_70a: @@@@@@ FAIL: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3351:test_70a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3353:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_70a.*.1746065088.log rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.debug_log.oleg227-server.1746065088.log.vZ0NGN" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_70a.dmesg.oleg227-server.1746065088.log.il23YU" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 70a (6s) == sanity-flr test 71: check mirror extend parent fid ==== 22:04:51 (1746065091) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000403:0x369:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (5s) == sanity-flr test 200a: stress test ===================== 22:04:56 (1746065096) Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre2 Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8070112 .. Extending file size to 4802016 .. failed Extending file size to 8892864 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7337504 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7445600 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8094496 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9007680 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9215520 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 30250 30251 30253 30254 30255 failed 192.168.202.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg227-client.virtnet /mnt/lustre2 (opts:) 192.168.202.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg227-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 4096-9211904 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 4096-9211904 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200a.sanity-flr': Invalid argument (22) sanity-flr test_200a: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3534:test_200a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3546:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200a.*.1746065162.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.debug_log.oleg227-server.1746065162.log.NEWy4m" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200a.dmesg.oleg227-server.1746065162.log.aZ90RA" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:06:05 (1746065165) Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre2 Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2555168 .. Extending file size to 3972896 .. done Extending file size to 5885120 .. Extending file size to 7255104 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7946720 .. Extending file size to 8276544 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8497216 .. Extending file size to 8076416 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8864864 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8703616 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9091552 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9228896 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 805 814 815 820 821 failed 192.168.202.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg227-client.virtnet /mnt/lustre2 (opts:) 192.168.202.127@tcp:/lustre /mnt/lustre3 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg227-client.virtnet /mnt/lustre3 (opts:) lfs mirror mirror: fail to pread 4096-9228288 of mirror 1: Invalid argument (22) lfs mirror mirror: error reading bytes 4096-9228288 of mirror 1: Invalid argument (22) lfs mirror mirror: fail to mirror resync '/mnt/lustre/f200b.sanity-flr': Invalid argument (22) PID TTY STAT TIME COMMAND 1 ? Ss 0:06 /usr/lib/systemd/systemd --switched-root --system --deserialize 22 2 ? S 0:00 [kthreadd] 3 ? S 0:00 [kworker/0:0] 4 ? S< 0:00 [kworker/0:0H] 5 ? S 0:01 [kworker/u8:0] 6 ? S 0:02 [ksoftirqd/0] 7 ? S 0:04 [migration/0] 8 ? S 0:00 [rcu_bh] 9 ? R 0:33 [rcu_sched] 10 ? S 0:00 [rcuob/0] 11 ? S 0:12 [rcuos/0] 12 ? S< 0:00 [lru-add-drain] 13 ? S 0:00 [watchdog/0] 14 ? S 0:00 [watchdog/1] 15 ? S 0:03 [migration/1] 16 ? S 0:00 [ksoftirqd/1] 18 ? S< 0:00 [kworker/1:0H] 19 ? S 0:00 [rcuob/1] 20 ? S 0:12 [rcuos/1] 21 ? S 0:00 [watchdog/2] 22 ? S 0:02 [migration/2] 23 ? S 0:00 [ksoftirqd/2] 25 ? S< 0:00 [kworker/2:0H] 26 ? S 0:00 [rcuob/2] 27 ? S 0:11 [rcuos/2] 28 ? S 0:00 [watchdog/3] 29 ? S 0:02 [migration/3] 30 ? S 0:00 [ksoftirqd/3] 31 ? S 0:01 [kworker/3:0] 32 ? S< 0:00 [kworker/3:0H] 33 ? S 0:00 [rcuob/3] 34 ? S 0:10 [rcuos/3] 35 ? S 0:00 [kdevtmpfs] 36 ? S< 0:00 [netns] 37 ? S 0:00 [khungtaskd] 38 ? S< 0:00 [writeback] 39 ? S< 0:00 [kintegrityd] 40 ? S< 0:00 [bioset] 41 ? S< 0:00 [bioset] 42 ? S< 0:00 [bioset] 43 ? S< 0:00 [kblockd] 44 ? S< 0:00 [md] 45 ? S< 0:00 [watchdogd] 47 ? S< 0:00 [rpciod] 48 ? S< 0:00 [xprtiod] 51 ? S 0:00 [kswapd0] 52 ? SN 0:00 [ksmd] 53 ? SN 0:00 [khugepaged] 54 ? S< 0:00 [nfsiod] 55 ? S< 0:00 [crypto] 63 ? S< 0:00 [kthrotld] 64 ? S 0:00 [kworker/u8:1] 66 ? S< 0:00 [kmpath_rdacd] 67 ? S< 0:00 [kpsmoused] 69 ? S< 0:00 [ipv6_addrconf] 82 ? S< 0:00 [deferwq] 280 ? S< 0:00 [ata_sff] 286 ? S 0:00 [scsi_eh_0] 288 ? S< 0:00 [scsi_tmf_0] 290 ? S 0:00 [scsi_eh_1] 291 ? S< 0:00 [scsi_tmf_1] 305 ? S 0:01 [kworker/u8:3] 358 ? S 0:00 [kworker/u8:4] 448 ? Ss 0:01 @bd-client --systemd-mark 192.168.200.253 -N centos7 /dev/nbd0 -p -b4096 450 ? S< 0:01 [nbd0] 480 ? S 0:00 [jbd2/nbd0-8] 481 ? S< 0:00 [ext4-rsv-conver] 540 ? S 0:04 [kworker/2:2] 564 pts/0 S+ 0:00 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 565 pts/0 S+ 0:00 tee -i /tmp/testlogs//sanity-flr.test_200b.test_log.oleg227-client.log 567 ? Ss 0:02 /usr/lib/systemd/systemd-journald 594 ? S< 0:00 [kworker/1:1H] 595 ? S< 0:00 [kworker/0:1H] 601 ? Ss 0:00 /usr/lib/systemd/systemd-udevd 714 ? S< 0:00 [kvm-irqfd-clean] 719 ? S< 0:00 [edac-poller] 771 ? S 0:00 [nfsv4.1-svc] 773 ? S< 0:00 [kworker/3:1H] 806 ? Ss 0:01 /usr/lib/systemd/systemd-logind 807 ? Ss 0:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 808 ? Ssl 0:00 /usr/sbin/gssproxy -D 816 ? Ssl 0:00 /usr/sbin/NetworkManager --no-daemon 827 ? Ss 0:00 /usr/sbin/sshd -D 845 ttyS1 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS1 vt220 846 ttyS0 Ss+ 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220 847 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 linux 863 ? Ssl 0:01 /usr/lib/polkit-1/polkitd --no-debug 869 ? S 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-4b58f25c-c88a-4cbd-a719-8448f53633ed-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0 895 ? Ssl 0:01 /usr/sbin/rsyslogd -n 999 ? S< 0:00 [kworker/2:1H] 1127 ? Ss 0:00 sshd: root@pts/0 1165 pts/0 Ss+ 0:00 /bin/bash /home/green/git/lustre-release/lustre/tests/auster -D /tmp/testlogs/ -r -k sanity-flr 1989 ? S 0:00 [ptlrpc_hr00_000] 1990 ? S 0:00 [ptlrpc_hr01_000] 1991 ? S 0:00 [socknal_cd00] 1992 ? S 0:00 [socknal_cd01] 1993 ? S 0:00 [socknal_cd02] 1994 ? S 0:00 [socknal_cd03] 1995 ? S 0:00 [socknal_reaper] 1996 ? S 0:00 [socknal_sd00_00] 1997 ? S 0:00 [socknal_sd00_01] 1998 ? S 0:18 [socknal_sd01_00] 1999 ? S 0:18 [socknal_sd01_01] 2000 ? S 0:00 [acceptor_001] 2001 ? S 0:00 [monitor_thread] 2002 ? S 0:00 [lnet_discovery] 2003 ? S 0:01 [ptlrpcd_rcv] 2004 ? S 0:06 [ptlrpcd_00_00] 2005 ? S 0:06 [ptlrpcd_00_01] 2006 ? S 0:03 [ptlrpcd_01_00] 2007 ? S 0:03 [ptlrpcd_01_01] 2779 ? S 0:00 /usr/lib/systemd/systemd-udevd 2800 pts/0 R+ 0:00 ps ax 5084 ? S 0:00 [kworker/1:0] 5924 ? S 0:00 [kworker/1:2] 7623 pts/0 S+ 0:01 bash /home/green/git/lustre-release/lustre/tests/sanity-flr.sh 8511 ? S 0:00 [ldlm_cb00_000] 8512 ? S 0:00 [ldlm_cb00_001] 8513 ? S 0:01 [ldlm_cb01_000] 8514 ? S 0:01 [ldlm_cb01_001] 8515 ? S 0:00 [ldlm_cn00_000] 8516 ? S 0:00 [ldlm_cn00_001] 8517 ? S 0:00 [ldlm_cn00_002] 8518 ? S 0:00 [ldlm_cn01_000] 8519 ? S 0:00 [ldlm_cn01_001] 8520 ? S 0:00 [ldlm_cn01_002] 8521 ? S 0:01 [ldlm_bl_01] 8522 ? S 0:01 [ldlm_bl_02] 8523 ? S 0:00 [ldlm_elt] 8524 ? S 0:00 [ll_cfg_requeue] 16305 ? S 0:01 [ldlm_bl_03] 22497 ? S 0:01 [ldlm_cb01_002] 25046 ? S 0:00 [kworker/3:2] 25193 ? S 0:00 [kworker/0:2] 26898 ? S 0:02 [kworker/3:1] 27067 ? S 0:00 [kworker/2:0] 27071 ? S 0:00 [kworker/0:1] 27241 ? S 0:00 [ldlm_bl_04] 27963 ? S 0:00 [ldlm_cb00_002] 28293 ? S 0:00 [kworker/2:1] 30397 ? S 0:00 [ldlm_bl_05] 30398 ? S 0:00 [ldlm_bl_06] 30399 ? S 0:00 [ldlm_bl_07] 31231 ? S 0:00 [ldlm_bl_08] 31232 ? S 0:00 [ldlm_bl_09] 31828 ? S 0:00 [kworker/u8:2] sanity-flr test_200b: @@@@@@ FAIL: final resync failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3606:test_200b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-flr.sh:3618:main() Dumping lctl log to /tmp/testlogs//sanity-flr.test_200b.*.1746065230.log rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.debug_log.oleg227-server.1746065230.log.kZ1V6d" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-flr.test_200b.dmesg.oleg227-server.1746065230.log.hvotkm" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 200b (69s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:07:13 (1746065233) Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2316 1285372 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14172 3592848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18104 3588916 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32276 7181764 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2316 1285372 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 14172 3592848 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18104 3588916 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 32276 7181764 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0592304 s, 35.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.130085 s, 16.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg227-client.virtnet /mnt/lustre2 (opts:) PASS 200c (8s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:07:22 (1746065242) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0656081 s, 32.0 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 22:07:25 (1746065245) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x38:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x9f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa0:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x81:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:07:29 (1746065249) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0420744 s, 49.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x38f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x38f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x390:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:07:32 (1746065252) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0407194 s, 51.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x82:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x83:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:07:35 (1746065255) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.049038 s, 42.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x84:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x85:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:08:01 (1746065281) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg227-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0392341 s, 26.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x390:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x392:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x391:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x390:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x392:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x391:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x391:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:08:36 (1746065316) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg227-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0406123 s, 51.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x86:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x86:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (39s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:09:15 (1746065355) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg227-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg227-server: Pool lustre.test_204f created Adding targets to pool oleg227-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg227-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0349035 s, 60.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x87:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xa6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x87:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg227-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg227-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg227-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:10:04 (1746065404) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:10:08 (1746065408) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:10:12 (1746065412) oleg227-server: Pool lustre.test_206 created Adding targets to pool oleg227-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg227-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg227-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg227-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg227-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:10:30 (1746065430) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0370224 s, 28.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164586 s, 63.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x793d541a Mirror 3: 0x793d541a PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:10:35 (1746065435) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:10:37 (1746065437) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:10:38 (1746065438) Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes (4 B) copied, 0.0499336 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1160000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1310000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1310000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1310000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xef0000 Read this (4 bytes): BBBB 192.168.202.127@tcp:/lustre /mnt/lustre2 lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg227-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:11:01 (1746065461) 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0111671 s, 734 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:11:04 (1746065464) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191446 s, 54.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x59 lmm_fid: [0x200000404:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x59 lmm_fid: [0x200000404:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x91:0x0] } File: '/mnt/lustre/f210a.sanity-flr' Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834009 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-04-30 22:11:05.000000000 -0400 Modify: 2025-04-30 22:11:05.000000000 -0400 Change: 2025-04-30 22:11:05.000000000 -0400 Birth: - PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:11:07 (1746065467) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177713 s, 59.0 MB/s oleg227-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg227-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps fail_loc=0x1428 Waiting for MDT destroys to complete oleg227-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg227-server: /dev/mapper/ost1_flakey: catastrophic mode - not reading inode or group bitmaps oleg227-server: O/280000401/d17/b1: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:11:20 (1746065480) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.101518 s, 403 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0126505 s, 324 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 30 22:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 30 22:11 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test complete, duration 985 sec ============ 22:11:24 (1746065484) sanity-flr: FAIL: test_0g error resync-ing file '/mnt/lustre/f0g.sanity-flr' sanity-flr: FAIL: test_0h error resync-ing file '/mnt/lustre/d0h.sanity-flr/f0h.sanity-flr' sanity-flr: FAIL: test_0j resync /mnt/lustre/f0j.sanity-flr failed sanity-flr: FAIL: test_36d mirror resync /mnt/lustre/f36d.sanity-flr failed sanity-flr: FAIL: test_37 1: mismatch: \'f1e627b38335c24a5ca5a5d71092ea50 -\' vs. \'0808d7dbb34e1f0432dbc3c72e4d3271 -\' sanity-flr: FAIL: test_38 valid mirror doesn't exist sanity-flr: FAIL: test_44b split --mirror-id=1 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr should succeed sanity-flr: FAIL: test_50a data mismatch: \'b46fdc980b505c0e8074fc7c58bc788b -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' sanity-flr: FAIL: test_50b data mismatch: \'5aa4604d16ac73000c99b67cca4ecbd9 -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' sanity-flr: FAIL: test_50d data mismatch: \'407c530a3cfca55b8bbbd3863159e7e3 -\' vs. \'d41d8cd98f00b204e9800998ecf8427e -\' sanity-flr: FAIL: test_61a cannot migrate /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr sanity-flr: FAIL: test_61c cannot resync mirror /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sanity-flr: FAIL: test_70a fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed sanity-flr: FAIL: test_200a final resync failed sanity-flr: FAIL: test_200b final resync failed === sanity-flr: start cleanup 22:11:25 (1746065485) === === sanity-flr: finish cleanup 22:11:29 (1746065489) ===