-----============= acceptance-small: sanity-dom ============----- Wed May 8 18:02:14 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: === sanity-dom: start setup 18:02:18 (1715205738) === Starting client oleg433-client.virtnet: -o user_xattr,flock oleg433-server@tcp:/lustre /mnt/lustre2 Started clients oleg433-client.virtnet: 192.168.204.133@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg433-client.virtnet: executing check_config_client /mnt/lustre oleg433-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg433-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a98fb800.idle_timeout=debug osc.lustre-OST0000-osc-ffff880136899800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a98fb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff880136899800.idle_timeout=debug disable quota as required oleg433-server: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-dom: finish setup 18:02:24 (1715205744) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6866] 192.168.204.133@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 1: write a file on one mount, truncate on the other, write again ========================================================== 18:02:25 (1715205745) 1+0 records in 1+0 records out 7168 bytes (7.2 kB) copied, 0.00164509 s, 4.4 MB/s dd: failed to open '/mnt/lustre/f1.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_1: @@@@@@ FAIL: write 2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:50:test_1() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:54:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_1.*.1715205845.log rsync: chown "/tmp/testlogs/.sanity-dom.test_1.debug_log.oleg433-server.1715205845.log.cni3hc" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_1.dmesg.oleg433-server.1715205845.log.X2qtXG" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1 (101s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 2: Write with a seek, append, read from a single mountpoint ========================================================== 18:04:09 (1715205849) dd: failed to open '/mnt/lustre/f2.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_2: @@@@@@ FAIL: test_2 failed with 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:64:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_2.*.1715205850.log rsync: chown "/tmp/testlogs/.sanity-dom.test_2.debug_log.oleg433-server.1715205850.log.vkOxq0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_2.dmesg.oleg433-server.1715205850.log.ktjZLG" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 2 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 3: Truncate over DoM size on different nodes ========================================================== 18:04:14 (1715205854) dd: failed to open '/mnt/lustre/f3.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_3: @@@@@@ FAIL: test_3 failed with 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:83:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_3.*.1715205854.log rsync: chown "/tmp/testlogs/.sanity-dom.test_3.debug_log.oleg433-server.1715205854.log.Bk90Im" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_3.dmesg.oleg433-server.1715205854.log.hbzlge" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 4: DoM: glimpse doesn't produce duplicated locks ========================================================== 18:04:18 (1715205858) dd: failed to open '/mnt/lustre/f4.sanity-dom': Cannot send after transport endpoint shutdown fail_loc=0x80000808 tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 5: DoM truncate deadlock ============== 18:04:21 (1715205861) dd: failed to open '/mnt/lustre/f5.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_5: @@@@@@ FAIL: test_5 failed with 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:142:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_5.*.1715205862.log rsync: chown "/tmp/testlogs/.sanity-dom.test_5.debug_log.oleg433-server.1715205862.log.HVyd8m" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_5.dmesg.oleg433-server.1715205862.log.5FjKGy" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 6: Race two writes, check file size === 18:04:26 (1715205866) open(O_RDWR|O_CREAT): Cannot send after transport endpoint shutdown open: No such file or directory /home/green/git/lustre-release/lustre/tests/sanity-dom.sh: line 151: kill: (13242) - No such process open: No such file or directory Can't lstat /mnt/lustre2/f6.sanity-dom: No such file or directory sanity-dom test_6: @@@@@@ FAIL: wrong size Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:154:test_6() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:156:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_6.*.1715205867.log rsync: chown "/tmp/testlogs/.sanity-dom.test_6.debug_log.oleg433-server.1715205867.log.BRee28" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_6.dmesg.oleg433-server.1715205867.log.UanN8y" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 6 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 7: Stale pages after read-on-open ===== 18:04:31 (1715205871) dd: failed to open '/mnt/lustre/f7.sanity-dom': Cannot send after transport endpoint shutdown open: Cannot send after transport endpoint shutdown 1+0 records in 1+0 records out 1000 bytes (1.0 kB) copied, 0.00134763 s, 742 kB/s md5sum: /mnt/lustre/f7.sanity-dom: Cannot send after transport endpoint shutdown sanity-dom test_7: @@@@@@ FAIL: Client reads stale page Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:167:test_7() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:169:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_7.*.1715205872.log rsync: chown "/tmp/testlogs/.sanity-dom.test_7.debug_log.oleg433-server.1715205872.log.KZC3pR" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_7.dmesg.oleg433-server.1715205872.log.UMASEt" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 7 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test fsx: Dual-mount fsx with DoM files ==== 18:04:36 (1715205876) fallocate on zfs doesn't consume space fallocate not supported touch: cannot touch '/mnt/lustre/ffsx.sanity-dom': Cannot send after transport endpoint shutdown Chance of close/open is 1 in 50 Seed set to 5876 open_test_files: /mnt/lustre/ffsx.sanity-dom: Cannot send after transport endpoint shutdown sanity-dom test_fsx: @@@@@@ FAIL: test_fsx failed with 91 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:180:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_fsx.*.1715205877.log rsync: chown "/tmp/testlogs/.sanity-dom.test_fsx.debug_log.oleg433-server.1715205877.log.bnEsCG" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_fsx.dmesg.oleg433-server.1715205877.log.DR7vEu" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL fsx (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test sanity: Run sanity with Data-on-MDT files ========================================================== 18:04:41 (1715205881) mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 excepting tests: 42a 42c 42b 118c 118d 407 411b 130b 130c 130d 130e 130f 130g 312 skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b === sanity: start setup 18:04:45 (1715205885) === Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:) lsof: WARNING: can't stat() lustre file system /mnt/lustre Output information may be incomplete. Stopping client oleg433-client.virtnet /mnt/lustre2 opts: oleg433-client.virtnet: executing check_config_client /mnt/lustre oleg433-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg433-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff880136899800.idle_timeout=debug osc.lustre-OST0001-osc-ffff880136899800.idle_timeout=debug disable quota as required oleg433-server: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity: finish setup 18:04:51 (1715205891) === rm: cannot remove '/mnt/lustre/[Rdfs][0-9]*': Cannot send after transport endpoint shutdown Trace dump: = /home/green/git/lustre-release/lustre/tests/sanity.sh:130:main() sanity: FAIL: test-framework exiting on error PASS sanity (11s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test sanityn: Run sanityn with Data-on-MDT files ========================================================== 18:04:53 (1715205893) mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 excepting tests: 28 102 skipping tests SLOW=no: 33a === sanityn: start setup 18:04:57 (1715205897) === Starting client oleg433-client.virtnet: -o user_xattr,flock oleg433-server@tcp:/lustre /mnt/lustre2 Started clients oleg433-client.virtnet: 192.168.204.133@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg433-client.virtnet: executing check_config_client /mnt/lustre oleg433-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg433-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a98fe800.idle_timeout=debug osc.lustre-OST0000-osc-ffff880136899800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a98fe800.idle_timeout=debug osc.lustre-OST0001-osc-ffff880136899800.idle_timeout=debug disable quota as required oleg433-server: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 18:05:03 (1715205903) === rm: cannot remove '/mnt/lustre/[df][0-9]*': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/lnk': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/[df].sanityn*': Cannot send after transport endpoint shutdown Trace dump: = /home/green/git/lustre-release/lustre/tests/sanityn.sh:47:main() sanityn: FAIL: test-framework exiting on error PASS sanityn (11s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-dom test complete, duration 170 sec ============ 18:05:04 (1715205904) sanity-dom: FAIL: test_1 write 2 sanity-dom: FAIL: test_2 test_2 failed with 1 sanity-dom: FAIL: test_3 test_3 failed with 1 sanity-dom: FAIL: test_5 test_5 failed with 1 sanity-dom: FAIL: test_6 wrong size sanity-dom: FAIL: test_7 Client reads stale page sanity-dom: FAIL: test_fsx test_fsx failed with 91 === sanity-dom: start cleanup 18:05:05 (1715205905) === rm: cannot remove '/mnt/lustre/[Rdfs][0-9]*': Cannot send after transport endpoint shutdown sanity-dom : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6436:do_check_and_cleanup_lustre() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6473:check_and_cleanup_lustre() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:222:main() Dumping lctl log to /tmp/testlogs//sanity-dom..*.1715205905.log rsync: chown "/tmp/testlogs/.sanity-dom..debug_log.oleg433-server.1715205905.log.AeykCf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom..dmesg.oleg433-server.1715205905.log.tT1Uig" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]