== sanity-scrub test 5: OI scrub state machine =========== 08:40:59 (1782218459) oleg351-server: oleg351-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Using TIMEOUT=20 preparing... Tue Jun 23 08:44:23 EDT 2026 creating 100 files on mds1 striped dir -i0 -c1 -H crush /mnt/lustre/d5.sanity-scrub/mds1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267741 s, 39.2 MB/s creating 100 files on mds2 striped dir -i1 -c1 -H crush2 /mnt/lustre/d5.sanity-scrub/mds2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435176 s, 24.1 MB/s prepared Tue Jun 23 08:44:35 EDT 2026. stop mds1 stop mds2 file-level backup/restore on mds1:/dev/mapper/mds1_flakey backup data reformat new device Format mds1: /dev/mapper/mds1_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' file-level backup/restore on mds2:/dev/mapper/mds2_flakey backup data reformat new device Format mds2: /dev/mapper/mds2_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' starting MDTs with OI scrub disabled (1) oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre fail_val=3 fail_loc=0x190 192.168.203.151@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg351-client.virtnet /mnt/lustre (opts:) fail_loc=0x191 stopping mds1 stopping mds2 fail_loc=0 fail_val=0 starting MDTs with OI scrub disabled (2) oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 stopping mds1 stopping mds2 fail_val=3 fail_loc=0x190 starting MDTs without disabling OI scrub oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 oleg351-server: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 fail_loc=0x192 Waiting 6s for 'failed' Updated after 2s: want 'failed' got 'failed' Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre fail_loc=0 fail_val=0 File: /mnt/lustre/d5.sanity-scrub/mds1/sanity-scrub.sh Size: 42517 Blocks: 88 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205289279557 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-23 08:45:03.000000000 -0400 Modify: 2026-06-23 08:44:26.000000000 -0400 Change: 2026-06-23 08:45:03.000000000 -0400 Birth: 2026-06-23 08:45:03.000000000 -0400 File: /mnt/lustre/d5.sanity-scrub/mds2/sanity-scrub.sh Size: 42517 Blocks: 88 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 162129603798761540 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-23 08:45:20.000000000 -0400 Modify: 2026-06-23 08:44:33.000000000 -0400 Change: 2026-06-23 08:45:20.000000000 -0400 Birth: 2026-06-23 08:45:20.000000000 -0400 Waiting 6s for 'completed' Update not seen after 6s: want 'completed' got 'failed' sanity-scrub test_5: @@@@@@ FAIL: (19) Expected 'completed' on mds1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/sanity-scrub.sh:204:scrub_check_status() = /home/green/git/lustre-release/lustre/tests/sanity-scrub.sh:768:test_5() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7701:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-scrub.sh:771:main() Dumping lctl log to /tmp/testlogs//sanity-scrub.test_5.*.1782218885.log rsync: chown "/tmp/testlogs/.sanity-scrub.test_5.debug_log.oleg351-server.1782218885.log.hkKv7n" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-scrub.test_5.dmesg.oleg351-server.1782218885.log.W3cwjZ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]