-----============= acceptance-small: racer ============----- Thu May 1 15:25:22 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg306-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/racer.*ex: No such file or directory excepting tests: racer: /home/green/git/lustre-release/lustre/tests/racer/racer.sh with 1 MDTs === racer: start setup 15:25:28 (1746127528) === Starting client oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre2 Started clients oleg306-client.virtnet: 192.168.203.106@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg306-client.virtnet: executing check_config_client /mnt/lustre oleg306-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg306-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9f0e000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b64a9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9f0e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b64a9800.idle_timeout=debug disable quota as required oleg306-server: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === racer: finish setup 15:25:39 (1746127539) === RACERDIRS=/mnt/lustre /mnt/lustre2 fallocate on zfs doesn't consume space fallocate not supported == racer test 1: racer on clients: oleg306-client.virtnet DURATION=300 ========================================================== 15:25:39 (1746127539) Generating /etc/ldev.conf on MGS ... oleg306-server - lustre-MDT0000 zfs:/dev/lustre-mdt1/mdt1 - - oleg306-server - lustre-OST0000 zfs:/dev/lustre-ost1/ost1 - - oleg306-server - lustre-OST0001 zfs:/dev/lustre-ost2/ost2 - - racers pids: 9535 9536 ./file_symlink.sh: line 1: kill: (16120) - No such process ./file_list.sh: line 1: kill: (15259) - No such process ./file_list.sh: line 1: kill: (15332) - No such process ./file_list.sh: line 1: kill: (15333) - No such process ./file_list.sh: line 1: kill: (15339) - No such process ./file_exec.sh: line 1: kill: (9588) - No such process ./file_exec.sh: line 1: kill: (9678) - No such process ./file_exec.sh: line 1: kill: (9826) - No such process ./file_exec.sh: line 1: kill: (9773) - No such process ./file_exec.sh: line 1: kill: (9729) - No such process ./file_exec.sh: line 1: kill: (9659) - No such process ./file_getxattr.sh: line 1: kill: (16248) - No such process ./file_setxattr.sh: line 1: kill: (16311) - No such process ./file_setxattr.sh: line 1: kill: (16310) - No such process ./file_fallocate.sh: line 1: kill: (16329) - No such process file_create.sh: no process found dir_create.sh: no process found file_create.sh: no process found file_rm.sh: no process found dir_create.sh: no process found file_rename.sh: no process found file_rm.sh: no process found file_link.sh: no process found file_rename.sh: no process found file_symlink.sh: no process found file_list.sh: no process found file_link.sh: no process found file_symlink.sh: no process found file_concat.sh: no process found file_list.sh: no process found file_exec.sh: no process found file_concat.sh: no process found file_chown.sh: no process found file_exec.sh: no process found file_chmod.sh: no process found file_chown.sh: no process found file_chmod.sh: no process found file_mknod.sh: no process found file_mknod.sh: no process found file_truncate.sh: no process found file_truncate.sh: no process found file_delxattr.sh: no process found file_delxattr.sh: no process found file_getxattr.sh: no process found file_getxattr.sh: no process found file_setxattr.sh: no process found file_setxattr.sh: no process found file_fallocate.sh: no process found file_fallocate.sh: no process found Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.106@tcp:/lustre 7471104 24576 7413760 1% /mnt/lustre2 We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.106@tcp:/lustre 7471104 24576 7413760 1% /mnt/lustre We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. pid=9535 rc=0 pid=9536 rc=0 /home/green/git/lustre-release/lustre/tests/racer.sh: line 109: 10889 Terminated $LUSTRE/tests/racer/lss_create.sh /home/green/git/lustre-release/lustre/tests/racer.sh: line 215: 10890 Terminated $LUSTRE/tests/racer/lss_destroy.sh Cleaning test environment ... oleg306-server: Can't destroy the snapshot lss_24565 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 22 Thu May 1 15:25:45 2025 (15962:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_8411 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:26:03 2025 (21937:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_15916 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:26:04 2025 (23293:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_8411 successfully with force Thu May 1 15:26:24 2025 (27058:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_9293 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:26:26 2025 (28459:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_15916 successfully with force Thu May 1 15:26:44 2025 (482:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_9293 successfully with force Thu May 1 15:26:46 2025 (1397:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_12635 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:27:06 2025 (8383:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_12635 successfully with force Thu May 1 15:27:06 2025 (8388:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_4264 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:27:25 2025 (14052:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_1244 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:27:30 2025 (16379:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_1244 successfully with force Thu May 1 15:27:40 2025 (18875:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_10517 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:27:52 2025 (21323:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_10517 successfully with force Thu May 1 15:27:54 2025 (21840:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_4033 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:28:14 2025 (27165:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_4033 successfully with force Thu May 1 15:28:15 2025 (27260:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_22459 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:28:32 2025 (31899:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_22459 successfully with force Thu May 1 15:28:34 2025 (32603:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_14376 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:28:53 2025 (3597:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_14376 successfully with force Thu May 1 15:28:55 2025 (4301:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_22941 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:29:13 2025 (8237:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_25198 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:29:16 2025 (9934:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_22941 successfully with force Thu May 1 15:29:28 2025 (12915:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_21761 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:29:38 2025 (16054:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_21761 successfully with force Thu May 1 15:29:47 2025 (18624:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_17328 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:30:05 2025 (21925:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_28602 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:30:06 2025 (22990:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_17328 successfully with force Thu May 1 15:30:21 2025 (27133:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_9652 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:30:25 2025 (28653:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_28602 successfully with force Thu May 1 15:30:36 2025 (31624:jt_snapshot_create:1538:lustre:ssh): Create snapshot lss_24565 successfully with comment <(null)>, barrier , timeout <30> Thu May 1 15:30:47 2025 (1978:jt_snapshot_destroy:1798:lustre:ssh): Destroy snapshot lss_24565 successfully with force Thu May 1 15:30:54 2025 (3699:jt_snapshot_destroy:1794:lustre:ssh): Can't destroy snapshot lss_24565 with force : -22 racer test_1: @@@@@@ FAIL: Fail to destroy lss_24565 by force Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7249:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:10780:lss_err() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:10795:lss_cleanup() = /home/green/git/lustre-release/lustre/tests/racer.sh:218:test_1() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7622:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7685:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7503:run_test() = /home/green/git/lustre-release/lustre/tests/racer.sh:223:main() Dumping lctl log to /tmp/testlogs//racer.test_1.*.1746127856.log rsync: chown "/tmp/testlogs/.racer.test_1.debug_log.oleg306-server.1746127856.log.0D4L7P" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.racer.test_1.dmesg.oleg306-server.1746127856.log.FDaYqn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 1 (320s) == racer test 2: racer rename: oleg306-client.virtnet DURATION=300 ========================================================== 15:31:02 (1746127862) SKIP: racer test_2 need at least 2 MDTs SKIP 2 (1s) == racer test complete, duration 341 sec ================= 15:31:03 (1746127863) racer: FAIL: test_1 Fail to destroy lss_24565 by force === racer: start cleanup 15:31:04 (1746127864) === Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg306-client.virtnet /mnt/lustre2 opts: === racer: finish cleanup 15:31:11 (1746127871) ===