-----============= acceptance-small: sanity-scrub ============----- Fri Aug 14 13:32:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg144-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-scrub.*ex': No such file or directory excepting tests: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions client=34683138 MDS=34683138 OSS=34683138 Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:) Stopping client oleg144-client.virtnet /mnt/lustre opts: Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local oleg144-server: modules unloaded. === sanity-scrub: start setup 13:34:45 (1786728885) === Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 oleg144-server: oleg144-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg144-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/vdc pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Format mds2: /dev/vdd pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Format ost1: /dev/vde pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Format ost2: /dev/vdf pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Checking servers environments Checking clients oleg144-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vdf Started lustre-OST0001 Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre Starting client oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre Started clients oleg144-client.virtnet: 192.168.201.144@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9448075d7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9448075d7000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 6s: want 'procname_uid' got 'procname_uid' disable quota as required osd-ldiskfs.track_declares_assert=1 === sanity-scrub: finish setup 13:38:17 (1786729097) === == sanity-scrub test 23: lctl iterate_objects can iterate objects on OST ========================================================== 13:38:20 (1786729100) Initial objects found: 65 - open/close 524 (time 1786729115.58 total 10.01 last 52.34) - open/close 1200 (time 1786729125.58 total 20.01 last 67.59) - open/close 2015 (time 1786729135.59 total 30.02 last 81.46) total: 2048 open/close in 30.44 seconds: 67.28 ops/second 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160066 s, 6.6 MB/s Parent FID: [0x200000402:0x803:0x0] OST FID: 0x280000401:0x802:0x0 New objects found: 2113 Verifying all created files are returned... Waiting for parent FIDs to propagate to OST... 1429 of 2049 parent FIDs are still missing on the OST... Verification successful: all created files found! Testing limit 2000 (multiple pages)... oleg144-server: iterate_objects: lustre-OST0000: resume with --resume 1896 Returned 2000 objects with limit 2000 Verification successful: limited iteration valid! PASS 23 (58s) == sanity-scrub test 24: object iteration copes with objects created in the middle of it ========================================================== 13:39:18 (1786729158) Creating 500 files... total: 500 open/close in 6.58 seconds: 76.00 ops/second - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2593 prealloc_next_id: 2551 count: 43 lustre-OST0000 holds 2625 objects before the walk Starting /home/green/git/lustre-release/lustre/tests/scrub_iterate_test on ost1... /home/green/git/lustre-release/lustre/tests/scrub_iterate_test runs on ost1 as PID 24614 Waiting for the walk to pause... Creating 543 files while the walk is paused... total: 543 open/close in 4.97 seconds: 109.19 ops/second Resuming the walk... pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 the walk reported 3169 objects, 544 objects were created while it was paused 544 of the new objects were reported by the walk lustre-MDT0000 holds 3094 objects before the walk Starting /home/green/git/lustre-release/lustre/tests/scrub_iterate_test on mds1... /home/green/git/lustre-release/lustre/tests/scrub_iterate_test runs on mds1 as PID 25297 Waiting for the walk to pause... Creating 200 files while the walk is paused... total: 200 open/close in 2.26 seconds: 88.52 ops/second Resuming the walk... pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 the walk reported 3294 objects, 200 objects were created while it was paused 200 of the new objects were reported by the walk pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 PASS 24 (61s) == sanity-scrub test 25: lctl iterate_objects --count limit support ========================================================== 13:40:19 (1786729219) Creating 10 files... total: 10 open/close in 0.18 seconds: 55.86 ops/second Testing limit < total (count = 5)... oleg144-server: iterate_objects: lustre-OST0000: resume with --resume 0 Returned 5 objects (expect 5) Testing limit > total (count = 3371)... Returned 3361 objects with limit 3371 (expect 3361) Testing invalid limit (0)... oleg144-server: error: iterate_objects: invalid count '0' pdsh@oleg144-client: oleg144-server: ssh exited with exit code 4 list the objects of a local OST or MDT usage: iterate_objects [--print-parent] [--print-size] [--print-time] [--print-owner] [--count COUNT] [--resume COOKIE] Testing invalid limit (-1)... oleg144-server: error: iterate_objects: invalid count '-1' pdsh@oleg144-client: oleg144-server: ssh exited with exit code 4 list the objects of a local OST or MDT usage: iterate_objects [--print-parent] [--print-size] [--print-time] [--print-owner] [--count COUNT] [--resume COOKIE] Success! PASS 25 (14s) == sanity-scrub test 26: lctl iterate_objects on MDT (regular files only) ========================================================== 13:40:33 (1786729233) Directory FID: [0x200000402:0xce3:0x0] Creating 10 files... total: 10 open/close in 0.13 seconds: 79.39 ops/second Expected MDT FIDs: 0x200000402:0xce4:0x0 0x200000402:0xce5:0x0 0x200000402:0xce6:0x0 0x200000402:0xce7:0x0 0x200000402:0xce8:0x0 0x200000402:0xce9:0x0 0x200000402:0xcea:0x0 0x200000402:0xceb:0x0 0x200000402:0xcec:0x0 0x200000402:0xced:0x0 Running lctl iterate_objects on lustre-MDT0000... lustre-MDT0000 reported 3306 objects Testing limit (count = 5)... oleg144-server: iterate_objects: lustre-MDT0000: resume with --resume 0 Returned 5 objects (expect 5) Testing size... Testing parent (should be rejected)... oleg144-server: error: iterate_objects: failed to iterate lustre-MDT0000: Operation not supported oleg144-server: iterate_objects: lustre-MDT0000: resume with --resume 0 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 95 Success! PASS 26 (14s) == sanity-scrub test 27: iterate_objects and OI scrub do not disturb each other ========================================================== 13:40:47 (1786729247) preparing... Fri Aug 14 13:41:23 EDT 2026 creating 0 files on mds1 striped dir -i0 -c1 -H crush2 /mnt/lustre/d27.sanity-scrub/mds1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379075 s, 27.7 MB/s creating 0 files on mds2 striped dir -i1 -c1 -H fnv_1a_64 /mnt/lustre/d27.sanity-scrub/mds2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399974 s, 26.2 MB/s prepared Fri Aug 14 13:41:31 EDT 2026. stop mds1 stop mds2 removing OI files on mds1: idx= removed '/mnt/lustre-brpt/oi.16.0' removed '/mnt/lustre-brpt/oi.16.1' removed '/mnt/lustre-brpt/oi.16.10' removed '/mnt/lustre-brpt/oi.16.11' removed '/mnt/lustre-brpt/oi.16.12' removed '/mnt/lustre-brpt/oi.16.13' removed '/mnt/lustre-brpt/oi.16.14' removed '/mnt/lustre-brpt/oi.16.15' removed '/mnt/lustre-brpt/oi.16.16' removed '/mnt/lustre-brpt/oi.16.17' removed '/mnt/lustre-brpt/oi.16.18' removed '/mnt/lustre-brpt/oi.16.19' removed '/mnt/lustre-brpt/oi.16.2' removed '/mnt/lustre-brpt/oi.16.20' removed '/mnt/lustre-brpt/oi.16.21' removed '/mnt/lustre-brpt/oi.16.22' removed '/mnt/lustre-brpt/oi.16.23' removed '/mnt/lustre-brpt/oi.16.24' removed '/mnt/lustre-brpt/oi.16.25' removed '/mnt/lustre-brpt/oi.16.26' removed '/mnt/lustre-brpt/oi.16.27' removed '/mnt/lustre-brpt/oi.16.28' removed '/mnt/lustre-brpt/oi.16.29' removed '/mnt/lustre-brpt/oi.16.3' removed '/mnt/lustre-brpt/oi.16.30' removed '/mnt/lustre-brpt/oi.16.31' removed '/mnt/lustre-brpt/oi.16.32' removed '/mnt/lustre-brpt/oi.16.33' removed '/mnt/lustre-brpt/oi.16.34' removed '/mnt/lustre-brpt/oi.16.35' removed '/mnt/lustre-brpt/oi.16.36' removed '/mnt/lustre-brpt/oi.16.37' removed '/mnt/lustre-brpt/oi.16.38' removed '/mnt/lustre-brpt/oi.16.39' removed '/mnt/lustre-brpt/oi.16.4' removed '/mnt/lustre-brpt/oi.16.40' removed '/mnt/lustre-brpt/oi.16.41' removed '/mnt/lustre-brpt/oi.16.42' removed '/mnt/lustre-brpt/oi.16.43' removed '/mnt/lustre-brpt/oi.16.44' removed '/mnt/lustre-brpt/oi.16.45' removed '/mnt/lustre-brpt/oi.16.46' removed '/mnt/lustre-brpt/oi.16.47' removed '/mnt/lustre-brpt/oi.16.48' removed '/mnt/lustre-brpt/oi.16.49' removed '/mnt/lustre-brpt/oi.16.5' removed '/mnt/lustre-brpt/oi.16.50' removed '/mnt/lustre-brpt/oi.16.51' removed '/mnt/lustre-brpt/oi.16.52' removed '/mnt/lustre-brpt/oi.16.53' removed '/mnt/lustre-brpt/oi.16.54' removed '/mnt/lustre-brpt/oi.16.55' removed '/mnt/lustre-brpt/oi.16.56' removed '/mnt/lustre-brpt/oi.16.57' removed '/mnt/lustre-brpt/oi.16.58' removed '/mnt/lustre-brpt/oi.16.59' removed '/mnt/lustre-brpt/oi.16.6' removed '/mnt/lustre-brpt/oi.16.60' removed '/mnt/lustre-brpt/oi.16.61' removed '/mnt/lustre-brpt/oi.16.62' removed '/mnt/lustre-brpt/oi.16.63' removed '/mnt/lustre-brpt/oi.16.7' removed '/mnt/lustre-brpt/oi.16.8' removed '/mnt/lustre-brpt/oi.16.9' removing OI files on mds2: idx= removed '/mnt/lustre-brpt/oi.16.0' removed '/mnt/lustre-brpt/oi.16.1' removed '/mnt/lustre-brpt/oi.16.10' removed '/mnt/lustre-brpt/oi.16.11' removed '/mnt/lustre-brpt/oi.16.12' removed '/mnt/lustre-brpt/oi.16.13' removed '/mnt/lustre-brpt/oi.16.14' removed '/mnt/lustre-brpt/oi.16.15' removed '/mnt/lustre-brpt/oi.16.16' removed '/mnt/lustre-brpt/oi.16.17' removed '/mnt/lustre-brpt/oi.16.18' removed '/mnt/lustre-brpt/oi.16.19' removed '/mnt/lustre-brpt/oi.16.2' removed '/mnt/lustre-brpt/oi.16.20' removed '/mnt/lustre-brpt/oi.16.21' removed '/mnt/lustre-brpt/oi.16.22' removed '/mnt/lustre-brpt/oi.16.23' removed '/mnt/lustre-brpt/oi.16.24' removed '/mnt/lustre-brpt/oi.16.25' removed '/mnt/lustre-brpt/oi.16.26' removed '/mnt/lustre-brpt/oi.16.27' removed '/mnt/lustre-brpt/oi.16.28' removed '/mnt/lustre-brpt/oi.16.29' removed '/mnt/lustre-brpt/oi.16.3' removed '/mnt/lustre-brpt/oi.16.30' removed '/mnt/lustre-brpt/oi.16.31' removed '/mnt/lustre-brpt/oi.16.32' removed '/mnt/lustre-brpt/oi.16.33' removed '/mnt/lustre-brpt/oi.16.34' removed '/mnt/lustre-brpt/oi.16.35' removed '/mnt/lustre-brpt/oi.16.36' removed '/mnt/lustre-brpt/oi.16.37' removed '/mnt/lustre-brpt/oi.16.38' removed '/mnt/lustre-brpt/oi.16.39' removed '/mnt/lustre-brpt/oi.16.4' removed '/mnt/lustre-brpt/oi.16.40' removed '/mnt/lustre-brpt/oi.16.41' removed '/mnt/lustre-brpt/oi.16.42' removed '/mnt/lustre-brpt/oi.16.43' removed '/mnt/lustre-brpt/oi.16.44' removed '/mnt/lustre-brpt/oi.16.45' removed '/mnt/lustre-brpt/oi.16.46' removed '/mnt/lustre-brpt/oi.16.47' removed '/mnt/lustre-brpt/oi.16.48' removed '/mnt/lustre-brpt/oi.16.49' removed '/mnt/lustre-brpt/oi.16.5' removed '/mnt/lustre-brpt/oi.16.50' removed '/mnt/lustre-brpt/oi.16.51' removed '/mnt/lustre-brpt/oi.16.52' removed '/mnt/lustre-brpt/oi.16.53' removed '/mnt/lustre-brpt/oi.16.54' removed '/mnt/lustre-brpt/oi.16.55' removed '/mnt/lustre-brpt/oi.16.56' removed '/mnt/lustre-brpt/oi.16.57' removed '/mnt/lustre-brpt/oi.16.58' removed '/mnt/lustre-brpt/oi.16.59' removed '/mnt/lustre-brpt/oi.16.6' removed '/mnt/lustre-brpt/oi.16.60' removed '/mnt/lustre-brpt/oi.16.61' removed '/mnt/lustre-brpt/oi.16.62' removed '/mnt/lustre-brpt/oi.16.63' removed '/mnt/lustre-brpt/oi.16.7' removed '/mnt/lustre-brpt/oi.16.8' removed '/mnt/lustre-brpt/oi.16.9' fail_val=3 fail_loc=0x190 starting MDTs without disabling OI scrub oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 oleg144-server: error: iterate_objects: failed to iterate lustre-MDT0000: Device or resource busy oleg144-server: iterate_objects: lustre-MDT0000: resume with --resume 0 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 16 fail_loc=0 fail_val=0 [0x200000400:0x1:0x0] Started LFSCK on the device lustre-MDT0000: scrub Stopped LFSCK on the device lustre-MDT0000. Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre fail_loc=0 fail_val=0 PASS 27 (124s) == sanity-scrub test 28: object iteration is the same whatever the ioctl batch size ========================================================== 13:42:51 (1786729371) Creating 2560 files... - open/close 899 (time 1786729386.50 total 10.01 last 89.85) - open/close 1723 (time 1786729396.50 total 20.01 last 82.40) - open/close 2490 (time 1786729406.50 total 30.01 last 76.67) total: 2560 open/close in 30.91 seconds: 82.83 ops/second lustre-OST0000 holds 2704 objects Walking lustre-OST0000 with batch 0... Walking lustre-OST0000 with batch 1... Walking lustre-OST0000 with batch 3... Walking lustre-OST0000 with batch 255... Walking lustre-OST0000 with batch 1024... Walking lustre-OST0000 with batch 4096... Asking lustre-OST0000 for metadata it does not know... oleg144-server: /home/green/git/lustre-release/lustre/tests/scrub_iterate_test: cannot iterate lustre-OST0000: Operation not supported pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Success! PASS 28 (64s) == sanity-scrub test 29: object iteration can be resumed where it stopped ========================================================== 13:43:55 (1786729435) Creating 500 files... total: 500 open/close in 6.25 seconds: 80.03 ops/second lustre-OST0000 holds 3216 objects, stopping after 1608 stopped after 1608 objects, resume point '1640' Success! PASS 29 (23s) == sanity-scrub test 30: object iteration copes with objects unlinked under it ========================================================== 13:44:18 (1786729458) Creating 500 files... total: 500 open/close in 5.51 seconds: 90.77 ops/second Unlinking 400 files while the walk is paused... - unlinked 0 (time 1786729479 ; total 0 ; last 0) total: 400 unlinks in 2 seconds: 200.000000 unlinks/second Waiting for MDT destroys to complete pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Success! pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 PASS 30 (50s) debug=0 Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:) Stopping client oleg144-client.virtnet /mnt/lustre opts: Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local oleg144-server: modules unloaded. === sanity-scrub: start setup 13:46:24 (1786729584) === Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 oleg144-server: oleg144-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg144-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/vdc pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Format mds2: /dev/vdd pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Format ost1: /dev/vde pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Format ost2: /dev/vdf pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Checking servers environments Checking clients oleg144-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on /dev/vdf Started lustre-OST0001 Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre Starting client oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre Started clients oleg144-client.virtnet: 192.168.201.144@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9448185c4000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9448185c4000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 2s: want 'procname_uid' got 'procname_uid' disable quota as required osd-ldiskfs.track_declares_assert=1 === sanity-scrub: finish setup 13:50:00 (1786729800) === == sanity-scrub test complete, duration 1027 sec ========= 13:50:02 (1786729802) === sanity-scrub: start cleanup 13:50:04 (1786729804) === === sanity-scrub: finish cleanup 13:50:07 (1786729807) ===