-----============= acceptance-small: sanity-dom ============----- Wed May 8 18:02:37 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: === sanity-dom: start setup 18:02:41 (1715205761) === Starting client oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg435-client.virtnet: executing check_config_client /mnt/lustre oleg435-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg435-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9ca3000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b3e33000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9ca3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3e33000.idle_timeout=debug disable quota as required oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-dom: finish setup 18:02:49 (1715205769) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7497] 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 1: write a file on one mount, truncate on the other, write again ========================================================== 18:02:51 (1715205771) 1+0 records in 1+0 records out 7168 bytes (7.2 kB) copied, 0.00299831 s, 2.4 MB/s dd: failed to open '/mnt/lustre/f1.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_1: @@@@@@ FAIL: write 2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:50:test_1() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:54:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_1.*.1715205872.log rsync: chown "/tmp/testlogs/.sanity-dom.test_1.debug_log.oleg435-server.1715205872.log.PrC7VA" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_1.dmesg.oleg435-server.1715205872.log.VVxun3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1 (101s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 2: Write with a seek, append, read from a single mountpoint ========================================================== 18:04:34 (1715205874) dd: failed to open '/mnt/lustre/f2.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_2: @@@@@@ FAIL: test_2 failed with 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:64:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_2.*.1715205875.log rsync: chown "/tmp/testlogs/.sanity-dom.test_2.debug_log.oleg435-server.1715205875.log.VHT15k" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_2.dmesg.oleg435-server.1715205875.log.K6pQrU" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 2 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 3: Truncate over DoM size on different nodes ========================================================== 18:04:37 (1715205877) dd: failed to open '/mnt/lustre/f3.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_3: @@@@@@ FAIL: test_3 failed with 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:83:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_3.*.1715205877.log rsync: chown "/tmp/testlogs/.sanity-dom.test_3.debug_log.oleg435-server.1715205877.log.x60zhf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_3.dmesg.oleg435-server.1715205877.log.DCgRXV" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 3 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 4: DoM: glimpse doesn't produce duplicated locks ========================================================== 18:04:40 (1715205880) dd: failed to open '/mnt/lustre/f4.sanity-dom': Cannot send after transport endpoint shutdown fail_loc=0x80000808 tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory tail: cannot open '/mnt/lustre/f4.sanity-dom' for reading: Cannot send after transport endpoint shutdown stat: cannot read file system information for '/mnt/lustre2/f4.sanity-dom': No such file or directory PASS 4 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 5: DoM truncate deadlock ============== 18:04:43 (1715205883) dd: failed to open '/mnt/lustre/f5.sanity-dom': Cannot send after transport endpoint shutdown sanity-dom test_5: @@@@@@ FAIL: test_5 failed with 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:142:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_5.*.1715205884.log rsync: chown "/tmp/testlogs/.sanity-dom.test_5.debug_log.oleg435-server.1715205884.log.AY9GX0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_5.dmesg.oleg435-server.1715205884.log.lYxjsY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 5 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 6: Race two writes, check file size === 18:04:46 (1715205886) open(O_RDWR|O_CREAT): Cannot send after transport endpoint shutdown open: No such file or directory /home/green/git/lustre-release/lustre/tests/sanity-dom.sh: line 151: kill: (13681) - No such process open: No such file or directory Can't lstat /mnt/lustre2/f6.sanity-dom: No such file or directory sanity-dom test_6: @@@@@@ FAIL: wrong size Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:154:test_6() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:156:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_6.*.1715205888.log rsync: chown "/tmp/testlogs/.sanity-dom.test_6.debug_log.oleg435-server.1715205888.log.ef2Ode" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_6.dmesg.oleg435-server.1715205888.log.ozCThm" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 6 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test 7: Stale pages after read-on-open ===== 18:04:50 (1715205890) dd: failed to open '/mnt/lustre/f7.sanity-dom': Cannot send after transport endpoint shutdown open: Cannot send after transport endpoint shutdown 1+0 records in 1+0 records out 1000 bytes (1.0 kB) copied, 0.00132545 s, 754 kB/s md5sum: /mnt/lustre/f7.sanity-dom: Cannot send after transport endpoint shutdown sanity-dom test_7: @@@@@@ FAIL: Client reads stale page Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:167:test_7() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:169:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_7.*.1715205891.log rsync: chown "/tmp/testlogs/.sanity-dom.test_7.debug_log.oleg435-server.1715205891.log.I4XWoj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_7.dmesg.oleg435-server.1715205891.log.0JCZez" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 7 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test fsx: Dual-mount fsx with DoM files ==== 18:04:53 (1715205893) keep default fallocate mode: 0 touch: cannot touch '/mnt/lustre/ffsx.sanity-dom': Cannot send after transport endpoint shutdown Chance of close/open is 1 in 50 Seed set to 5894 open_test_files: /mnt/lustre/ffsx.sanity-dom: Cannot send after transport endpoint shutdown sanity-dom test_fsx: @@@@@@ FAIL: test_fsx failed with 91 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:180:main() Dumping lctl log to /tmp/testlogs//sanity-dom.test_fsx.*.1715205894.log rsync: chown "/tmp/testlogs/.sanity-dom.test_fsx.debug_log.oleg435-server.1715205894.log.Kz82PE" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom.test_fsx.dmesg.oleg435-server.1715205894.log.9ua852" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL fsx (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test sanity: Run sanity with Data-on-MDT files ========================================================== 18:04:56 (1715205896) mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 excepting tests: 42a 42c 42b 118c 118d 407 411b skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 === sanity: start setup 18:05:00 (1715205900) === Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:) lsof: WARNING: can't stat() lustre file system /mnt/lustre Output information may be incomplete. Stopping client oleg435-client.virtnet /mnt/lustre2 opts: oleg435-client.virtnet: executing check_config_client /mnt/lustre oleg435-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg435-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b3e33000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3e33000.idle_timeout=debug disable quota as required oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity: finish setup 18:05:07 (1715205907) === rm: cannot remove '/mnt/lustre/[Rdfs][0-9]*': Cannot send after transport endpoint shutdown Trace dump: = /home/green/git/lustre-release/lustre/tests/sanity.sh:130:main() sanity: FAIL: test-framework exiting on error PASS sanity (12s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == sanity-dom test sanityn: Run sanityn with Data-on-MDT files ========================================================== 18:05:09 (1715205909) mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 excepting tests: 28 skipping tests SLOW=no: 33a === sanityn: start setup 18:05:13 (1715205913) === Starting client oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg435-client.virtnet: executing check_config_client /mnt/lustre oleg435-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg435-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9d1f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b3e33000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9d1f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b3e33000.idle_timeout=debug disable quota as required oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 18:05:20 (1715205920) === rm: cannot remove '/mnt/lustre/[df][0-9]*': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/lnk': Cannot send after transport endpoint shutdown rm: cannot remove '/mnt/lustre/[df].sanityn*': Cannot send after transport endpoint shutdown Trace dump: = /home/green/git/lustre-release/lustre/tests/sanityn.sh:47:main() sanityn: FAIL: test-framework exiting on error PASS sanityn (12s) debug_raw_pointers=0 debug_raw_pointers=0 == sanity-dom test complete, duration 164 sec ============ 18:05:22 (1715205922) sanity-dom: FAIL: test_1 write 2 sanity-dom: FAIL: test_2 test_2 failed with 1 sanity-dom: FAIL: test_3 test_3 failed with 1 sanity-dom: FAIL: test_5 test_5 failed with 1 sanity-dom: FAIL: test_6 wrong size sanity-dom: FAIL: test_7 Client reads stale page sanity-dom: FAIL: test_fsx test_fsx failed with 91 === sanity-dom: start cleanup 18:05:22 (1715205922) === rm: cannot remove '/mnt/lustre/[Rdfs][0-9]*': Cannot send after transport endpoint shutdown sanity-dom : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6436:do_check_and_cleanup_lustre() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6473:check_and_cleanup_lustre() = /home/green/git/lustre-release/lustre/tests/sanity-dom.sh:222:main() Dumping lctl log to /tmp/testlogs//sanity-dom..*.1715205923.log rsync: chown "/tmp/testlogs/.sanity-dom..debug_log.oleg435-server.1715205923.log.kBaJuB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-dom..dmesg.oleg435-server.1715205923.log.lJl57c" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]