-----============= acceptance-small: sanity-scrub ============----- Fri Aug 14 13:35:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg427-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-scrub.*ex': No such file or directory excepting tests: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions client=34683138 MDS=34683138 OSS=34683138 Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:) Stopping client oleg427-client.virtnet /mnt/lustre opts: Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded. === sanity-scrub: start setup 13:37:43 (1786729063) === Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: oleg427-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/vdc pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Format mds2: /dev/vdd pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Format ost1: /dev/vde pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Format ost2: /dev/vdf pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Checking servers environments Checking clients oleg427-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vdf Started lustre-OST0001 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre Starting client oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre Started clients oleg427-client.virtnet: 192.168.204.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9105460b9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9105460b9800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 7s: want 'procname_uid' got 'procname_uid' disable quota as required osd-ldiskfs.track_declares_assert=1 === sanity-scrub: finish setup 13:41:28 (1786729288) === == sanity-scrub test 23: lctl iterate_objects can iterate objects on OST ========================================================== 13:41:30 (1786729290) Initial objects found: 65 - open/close 424 (time 1786729306.93 total 10.02 last 42.33) - open/close 817 (time 1786729316.94 total 20.02 last 39.29) - open/close 1282 (time 1786729326.94 total 30.03 last 46.47) - open/close 1727 (time 1786729336.97 total 40.05 last 44.38) total: 2048 open/close in 46.46 seconds: 44.08 ops/second 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200195 s, 5.2 MB/s Parent FID: [0x200000402:0x803:0x0] OST FID: 0x280000401:0x802:0x0 New objects found: 2113 Verifying all created files are returned... Waiting for parent FIDs to propagate to OST... 666 of 2049 parent FIDs are still missing on the OST... 666 of 2049 parent FIDs are still missing on the OST... 606 of 2049 parent FIDs are still missing on the OST... 370 of 2049 parent FIDs are still missing on the OST... 100 of 2049 parent FIDs are still missing on the OST... Verification successful: all created files found! Testing limit 2000 (multiple pages)... oleg427-server: iterate_objects: lustre-OST0000: resume with --resume 1908 Returned 2000 objects with limit 2000 Verification successful: limited iteration valid! PASS 23 (106s) == sanity-scrub test 24: object iteration copes with objects created in the middle of it ========================================================== 13:43:16 (1786729396) Creating 500 files... total: 500 open/close in 8.65 seconds: 57.82 ops/second - precreated_ost_obj_count lustre-OST0000-osc-MDT0000 prealloc_last_id: 2593 prealloc_next_id: 2551 count: 43 lustre-OST0000 holds 2625 objects before the walk Starting /home/green/git/lustre-release/lustre/tests/scrub_iterate_test on ost1... /home/green/git/lustre-release/lustre/tests/scrub_iterate_test runs on ost1 as PID 24850 Waiting for the walk to pause... Creating 543 files while the walk is paused... - open/close 451 (time 1786729437.71 total 10.01 last 45.04) total: 543 open/close in 12.09 seconds: 44.90 ops/second Resuming the walk... pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 the walk reported 3169 objects, 544 objects were created while it was paused 544 of the new objects were reported by the walk lustre-MDT0000 holds 3094 objects before the walk Starting /home/green/git/lustre-release/lustre/tests/scrub_iterate_test on mds1... /home/green/git/lustre-release/lustre/tests/scrub_iterate_test runs on mds1 as PID 25532 Waiting for the walk to pause... Creating 200 files while the walk is paused... total: 200 open/close in 4.11 seconds: 48.69 ops/second Resuming the walk... pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 the walk reported 3294 objects, 200 objects were created while it was paused 200 of the new objects were reported by the walk pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 PASS 24 (93s) == sanity-scrub test 25: lctl iterate_objects --count limit support ========================================================== 13:44:49 (1786729489) Creating 10 files... total: 10 open/close in 0.31 seconds: 32.09 ops/second Testing limit < total (count = 5)... oleg427-server: iterate_objects: lustre-OST0000: resume with --resume 0 Returned 5 objects (expect 5) Testing limit > total (count = 3371)... Returned 3361 objects with limit 3371 (expect 3361) Testing invalid limit (0)... oleg427-server: error: iterate_objects: invalid count '0' pdsh@oleg427-client: oleg427-server: ssh exited with exit code 4 list the objects of a local OST or MDT usage: iterate_objects [--print-parent] [--print-size] [--print-time] [--print-owner] [--count COUNT] [--resume COOKIE] Testing invalid limit (-1)... oleg427-server: error: iterate_objects: invalid count '-1' pdsh@oleg427-client: oleg427-server: ssh exited with exit code 4 list the objects of a local OST or MDT usage: iterate_objects [--print-parent] [--print-size] [--print-time] [--print-owner] [--count COUNT] [--resume COOKIE] Success! PASS 25 (13s) == sanity-scrub test 26: lctl iterate_objects on MDT (regular files only) ========================================================== 13:45:02 (1786729502) Directory FID: [0x200000402:0xce3:0x0] Creating 10 files... total: 10 open/close in 0.23 seconds: 43.88 ops/second Expected MDT FIDs: 0x200000402:0xce4:0x0 0x200000402:0xce5:0x0 0x200000402:0xce6:0x0 0x200000402:0xce7:0x0 0x200000402:0xce8:0x0 0x200000402:0xce9:0x0 0x200000402:0xcea:0x0 0x200000402:0xceb:0x0 0x200000402:0xcec:0x0 0x200000402:0xced:0x0 Running lctl iterate_objects on lustre-MDT0000... lustre-MDT0000 reported 3306 objects Testing limit (count = 5)... oleg427-server: iterate_objects: lustre-MDT0000: resume with --resume 0 Returned 5 objects (expect 5) Testing size... Testing parent (should be rejected)... oleg427-server: error: iterate_objects: failed to iterate lustre-MDT0000: Operation not supported oleg427-server: iterate_objects: lustre-MDT0000: resume with --resume 0 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 95 Success! PASS 26 (14s) == sanity-scrub test 27: iterate_objects and OI scrub do not disturb each other ========================================================== 13:45:16 (1786729516) preparing... Fri Aug 14 13:45:57 EDT 2026 creating 0 files on mds1 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d27.sanity-scrub/mds1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445931 s, 23.5 MB/s creating 0 files on mds2 striped dir -i1 -c1 -H crush2 /mnt/lustre/d27.sanity-scrub/mds2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030868 s, 34.0 MB/s prepared Fri Aug 14 13:46:10 EDT 2026. stop mds1 stop mds2 removing OI files on mds1: idx= removed '/mnt/lustre-brpt/oi.16.0' removed '/mnt/lustre-brpt/oi.16.1' removed '/mnt/lustre-brpt/oi.16.10' removed '/mnt/lustre-brpt/oi.16.11' removed '/mnt/lustre-brpt/oi.16.12' removed '/mnt/lustre-brpt/oi.16.13' removed '/mnt/lustre-brpt/oi.16.14' removed '/mnt/lustre-brpt/oi.16.15' removed '/mnt/lustre-brpt/oi.16.16' removed '/mnt/lustre-brpt/oi.16.17' removed '/mnt/lustre-brpt/oi.16.18' removed '/mnt/lustre-brpt/oi.16.19' removed '/mnt/lustre-brpt/oi.16.2' removed '/mnt/lustre-brpt/oi.16.20' removed '/mnt/lustre-brpt/oi.16.21' removed '/mnt/lustre-brpt/oi.16.22' removed '/mnt/lustre-brpt/oi.16.23' removed '/mnt/lustre-brpt/oi.16.24' removed '/mnt/lustre-brpt/oi.16.25' removed '/mnt/lustre-brpt/oi.16.26' removed '/mnt/lustre-brpt/oi.16.27' removed '/mnt/lustre-brpt/oi.16.28' removed '/mnt/lustre-brpt/oi.16.29' removed '/mnt/lustre-brpt/oi.16.3' removed '/mnt/lustre-brpt/oi.16.30' removed '/mnt/lustre-brpt/oi.16.31' removed '/mnt/lustre-brpt/oi.16.32' removed '/mnt/lustre-brpt/oi.16.33' removed '/mnt/lustre-brpt/oi.16.34' removed '/mnt/lustre-brpt/oi.16.35' removed '/mnt/lustre-brpt/oi.16.36' removed '/mnt/lustre-brpt/oi.16.37' removed '/mnt/lustre-brpt/oi.16.38' removed '/mnt/lustre-brpt/oi.16.39' removed '/mnt/lustre-brpt/oi.16.4' removed '/mnt/lustre-brpt/oi.16.40' removed '/mnt/lustre-brpt/oi.16.41' removed '/mnt/lustre-brpt/oi.16.42' removed '/mnt/lustre-brpt/oi.16.43' removed '/mnt/lustre-brpt/oi.16.44' removed '/mnt/lustre-brpt/oi.16.45' removed '/mnt/lustre-brpt/oi.16.46' removed '/mnt/lustre-brpt/oi.16.47' removed '/mnt/lustre-brpt/oi.16.48' removed '/mnt/lustre-brpt/oi.16.49' removed '/mnt/lustre-brpt/oi.16.5' removed '/mnt/lustre-brpt/oi.16.50' removed '/mnt/lustre-brpt/oi.16.51' removed '/mnt/lustre-brpt/oi.16.52' removed '/mnt/lustre-brpt/oi.16.53' removed '/mnt/lustre-brpt/oi.16.54' removed '/mnt/lustre-brpt/oi.16.55' removed '/mnt/lustre-brpt/oi.16.56' removed '/mnt/lustre-brpt/oi.16.57' removed '/mnt/lustre-brpt/oi.16.58' removed '/mnt/lustre-brpt/oi.16.59' removed '/mnt/lustre-brpt/oi.16.6' removed '/mnt/lustre-brpt/oi.16.60' removed '/mnt/lustre-brpt/oi.16.61' removed '/mnt/lustre-brpt/oi.16.62' removed '/mnt/lustre-brpt/oi.16.63' removed '/mnt/lustre-brpt/oi.16.7' removed '/mnt/lustre-brpt/oi.16.8' removed '/mnt/lustre-brpt/oi.16.9' removing OI files on mds2: idx= removed '/mnt/lustre-brpt/oi.16.0' removed '/mnt/lustre-brpt/oi.16.1' removed '/mnt/lustre-brpt/oi.16.10' removed '/mnt/lustre-brpt/oi.16.11' removed '/mnt/lustre-brpt/oi.16.12' removed '/mnt/lustre-brpt/oi.16.13' removed '/mnt/lustre-brpt/oi.16.14' removed '/mnt/lustre-brpt/oi.16.15' removed '/mnt/lustre-brpt/oi.16.16' removed '/mnt/lustre-brpt/oi.16.17' removed '/mnt/lustre-brpt/oi.16.18' removed '/mnt/lustre-brpt/oi.16.19' removed '/mnt/lustre-brpt/oi.16.2' removed '/mnt/lustre-brpt/oi.16.20' removed '/mnt/lustre-brpt/oi.16.21' removed '/mnt/lustre-brpt/oi.16.22' removed '/mnt/lustre-brpt/oi.16.23' removed '/mnt/lustre-brpt/oi.16.24' removed '/mnt/lustre-brpt/oi.16.25' removed '/mnt/lustre-brpt/oi.16.26' removed '/mnt/lustre-brpt/oi.16.27' removed '/mnt/lustre-brpt/oi.16.28' removed '/mnt/lustre-brpt/oi.16.29' removed '/mnt/lustre-brpt/oi.16.3' removed '/mnt/lustre-brpt/oi.16.30' removed '/mnt/lustre-brpt/oi.16.31' removed '/mnt/lustre-brpt/oi.16.32' removed '/mnt/lustre-brpt/oi.16.33' removed '/mnt/lustre-brpt/oi.16.34' removed '/mnt/lustre-brpt/oi.16.35' removed '/mnt/lustre-brpt/oi.16.36' removed '/mnt/lustre-brpt/oi.16.37' removed '/mnt/lustre-brpt/oi.16.38' removed '/mnt/lustre-brpt/oi.16.39' removed '/mnt/lustre-brpt/oi.16.4' removed '/mnt/lustre-brpt/oi.16.40' removed '/mnt/lustre-brpt/oi.16.41' removed '/mnt/lustre-brpt/oi.16.42' removed '/mnt/lustre-brpt/oi.16.43' removed '/mnt/lustre-brpt/oi.16.44' removed '/mnt/lustre-brpt/oi.16.45' removed '/mnt/lustre-brpt/oi.16.46' removed '/mnt/lustre-brpt/oi.16.47' removed '/mnt/lustre-brpt/oi.16.48' removed '/mnt/lustre-brpt/oi.16.49' removed '/mnt/lustre-brpt/oi.16.5' removed '/mnt/lustre-brpt/oi.16.50' removed '/mnt/lustre-brpt/oi.16.51' removed '/mnt/lustre-brpt/oi.16.52' removed '/mnt/lustre-brpt/oi.16.53' removed '/mnt/lustre-brpt/oi.16.54' removed '/mnt/lustre-brpt/oi.16.55' removed '/mnt/lustre-brpt/oi.16.56' removed '/mnt/lustre-brpt/oi.16.57' removed '/mnt/lustre-brpt/oi.16.58' removed '/mnt/lustre-brpt/oi.16.59' removed '/mnt/lustre-brpt/oi.16.6' removed '/mnt/lustre-brpt/oi.16.60' removed '/mnt/lustre-brpt/oi.16.61' removed '/mnt/lustre-brpt/oi.16.62' removed '/mnt/lustre-brpt/oi.16.63' removed '/mnt/lustre-brpt/oi.16.7' removed '/mnt/lustre-brpt/oi.16.8' removed '/mnt/lustre-brpt/oi.16.9' fail_val=3 fail_loc=0x190 starting MDTs without disabling OI scrub oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 oleg427-server: error: iterate_objects: failed to iterate lustre-MDT0000: Device or resource busy oleg427-server: iterate_objects: lustre-MDT0000: resume with --resume 0 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 16 fail_loc=0 fail_val=0 [0x200000400:0x1:0x0] Started LFSCK on the device lustre-MDT0000: scrub Stopped LFSCK on the device lustre-MDT0000. Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre fail_loc=0 fail_val=0 PASS 27 (137s) == sanity-scrub test 28: object iteration is the same whatever the ioctl batch size ========================================================== 13:47:33 (1786729653) Creating 2560 files... - open/close 669 (time 1786729669.59 total 10.01 last 66.85) - open/close 1365 (time 1786729679.59 total 20.01 last 69.59) - open/close 2111 (time 1786729689.60 total 30.02 last 74.54) total: 2560 open/close in 36.23 seconds: 70.65 ops/second lustre-OST0000 holds 2703 objects Walking lustre-OST0000 with batch 0... Walking lustre-OST0000 with batch 1... Walking lustre-OST0000 with batch 3... Walking lustre-OST0000 with batch 255... Walking lustre-OST0000 with batch 1024... Walking lustre-OST0000 with batch 4096... Asking lustre-OST0000 for metadata it does not know... oleg427-server: /home/green/git/lustre-release/lustre/tests/scrub_iterate_test: cannot iterate lustre-OST0000: Operation not supported pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Success! PASS 28 (70s) == sanity-scrub test 29: object iteration can be resumed where it stopped ========================================================== 13:48:44 (1786729724) Creating 500 files... total: 500 open/close in 6.69 seconds: 74.74 ops/second lustre-OST0000 holds 3215 objects, stopping after 1607 stopped after 1607 objects, resume point '1652' Success! PASS 29 (24s) == sanity-scrub test 30: object iteration copes with objects unlinked under it ========================================================== 13:49:08 (1786729748) Creating 500 files... total: 500 open/close in 7.59 seconds: 65.84 ops/second Unlinking 400 files while the walk is paused... - unlinked 0 (time 1786729771 ; total 0 ; last 0) total: 400 unlinks in 3 seconds: 133.333328 unlinks/second Waiting for MDT destroys to complete pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Success! pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 PASS 30 (53s) debug=0 Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:) Stopping client oleg427-client.virtnet /mnt/lustre opts: Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded. === sanity-scrub: start setup 13:51:18 (1786729878) === Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 oleg427-server: oleg427-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/vdc pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Format mds2: /dev/vdd pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Format ost1: /dev/vde pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Format ost2: /dev/vdf pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Checking servers environments Checking clients oleg427-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/vdf Started lustre-OST0001 Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre Starting client oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre Started clients oleg427-client.virtnet: 192.168.204.127@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff910548ce9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff910548ce9800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 5s: want 'procname_uid' got 'procname_uid' disable quota as required osd-ldiskfs.track_declares_assert=1 === sanity-scrub: finish setup 13:55:04 (1786730104) === == sanity-scrub test complete, duration 1157 sec ========= 13:55:06 (1786730106) === sanity-scrub: start cleanup 13:55:08 (1786730108) === === sanity-scrub: finish cleanup 13:55:13 (1786730113) ===