-----============= acceptance-small: conf-sanity ============----- Tue Aug 18 01:49:41 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg408-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682625 > 34553653) for LU-16938, go 81 excepting tests: 32b 32c 32newtarball Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server == conf-sanity test 45: long unlink handling in ptlrpcd == 01:54:01 (1787032441) start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 PASS 45 (313s) == conf-sanity test 69: replace an OST with the same index ========================================================== 01:59:15 (1787032755) start mds service on oleg408-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 694 (time 1787032885.00 total 10.01 last 69.32) - open/close 1397 (time 1787032895.01 total 20.02 last 70.27) - open/close 2145 (time 1787032905.03 total 30.04 last 74.66) - open/close 2958 (time 1787032915.04 total 40.05 last 81.20) - open/close 3635 (time 1787032925.04 total 50.05 last 67.69) - open/close 4305 (time 1787032935.04 total 60.05 last 66.99) - open/close 4944 (time 1787032945.05 total 70.05 last 63.88) - open/close 5500 (time 1787032955.05 total 80.06 last 55.58) - open/close 6079 (time 1787032965.06 total 90.06 last 57.86) - open/close 6772 (time 1787032975.06 total 100.06 last 69.30) - open/close 7411 (time 1787032985.08 total 110.08 last 63.77) - open/close 8076 (time 1787032995.09 total 120.09 last 66.44) - open/close 8782 (time 1787033005.09 total 130.10 last 70.57) - open/close 9556 (time 1787033015.09 total 140.10 last 77.38) total: 10000 open/close in 145.66 seconds: 68.65 ops/second - unlinked 0 (time 1787033025 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 670 (time 1787033126.75 total 10.02 last 66.87) - open/close 1335 (time 1787033136.76 total 20.03 last 66.40) - open/close 1976 (time 1787033146.77 total 30.04 last 64.04) - open/close 2594 (time 1787033156.78 total 40.05 last 61.76) - open/close 3225 (time 1787033166.79 total 50.06 last 63.04) - open/close 3880 (time 1787033176.79 total 60.06 last 65.50) - open/close 4473 (time 1787033186.80 total 70.07 last 59.22) - open/close 5061 (time 1787033196.80 total 80.07 last 58.80) - open/close 5723 (time 1787033206.81 total 90.08 last 66.17) - open/close 6278 (time 1787033216.82 total 100.09 last 55.41) - open/close 6861 (time 1787033226.83 total 110.10 last 58.27) - open/close 7398 (time 1787033236.84 total 120.11 last 53.66) - open/close 8047 (time 1787033246.85 total 130.12 last 64.84) - open/close 8736 (time 1787033256.86 total 140.13 last 68.82) - open/close 9426 (time 1787033266.87 total 150.14 last 68.92) total: 10000 open/close in 158.12 seconds: 63.24 ops/second - unlinked 0 (time 1787033279 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 711 (time 1787033380.84 total 10.00 last 71.09) - open/close 1438 (time 1787033390.85 total 20.01 last 72.63) - open/close 2075 (time 1787033400.87 total 30.03 last 63.60) - open/close 2745 (time 1787033410.88 total 40.04 last 66.94) - open/close 3442 (time 1787033420.88 total 50.04 last 69.67) - open/close 4136 (time 1787033430.89 total 60.05 last 69.31) - open/close 4758 (time 1787033440.90 total 70.06 last 62.19) - open/close 5481 (time 1787033450.90 total 80.06 last 72.26) - open/close 6197 (time 1787033460.90 total 90.06 last 71.58) - open/close 6830 (time 1787033470.91 total 100.07 last 63.24) - open/close 7514 (time 1787033480.92 total 110.08 last 68.36) - open/close 8227 (time 1787033490.93 total 120.09 last 71.23) - open/close 8911 (time 1787033500.94 total 130.10 last 68.33) - open/close 9541 (time 1787033510.95 total 140.11 last 62.96) total: 10000 open/close in 146.31 seconds: 68.35 ops/second - unlinked 0 (time 1787033521 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 628 (time 1787033618.73 total 10.00 last 62.79) - open/close 1267 (time 1787033628.73 total 20.00 last 63.89) - open/close 1959 (time 1787033638.74 total 30.01 last 69.16) - open/close 2630 (time 1787033648.74 total 40.01 last 67.10) - open/close 3313 (time 1787033658.75 total 50.02 last 68.23) - open/close 3947 (time 1787033668.76 total 60.03 last 63.36) - open/close 4600 (time 1787033678.77 total 70.04 last 65.23) - open/close 5255 (time 1787033688.78 total 80.05 last 65.44) - open/close 5957 (time 1787033698.78 total 90.05 last 70.19) - open/close 6629 (time 1787033708.79 total 100.06 last 67.16) - open/close 7350 (time 1787033718.79 total 110.06 last 72.08) - open/close 8088 (time 1787033728.80 total 120.07 last 73.69) - open/close 8801 (time 1787033738.81 total 130.08 last 71.27) - open/close 9549 (time 1787033748.81 total 140.08 last 74.78) total: 10000 open/close in 146.32 seconds: 68.34 ops/second - unlinked 0 (time 1787033759 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 751 (time 1787033853.18 total 10.01 last 75.05) - open/close 1465 (time 1787033863.20 total 20.02 last 71.29) - open/close 2209 (time 1787033873.21 total 30.04 last 74.30) - open/close 2935 (time 1787033883.22 total 40.04 last 72.58) - open/close 3642 (time 1787033893.22 total 50.04 last 70.68) - open/close 4337 (time 1787033903.23 total 60.06 last 69.39) - open/close 5062 (time 1787033913.24 total 70.06 last 72.47) - open/close 5791 (time 1787033923.26 total 80.08 last 72.77) - open/close 6510 (time 1787033933.27 total 90.09 last 71.84) - open/close 7288 (time 1787033943.28 total 100.11 last 77.66) - open/close 8019 (time 1787033953.30 total 110.12 last 72.99) - open/close 8744 (time 1787033963.32 total 120.14 last 72.38) - open/close 9458 (time 1787033973.32 total 130.14 last 71.37) total: 10000 open/close in 138.07 seconds: 72.43 ops/second - unlinked 0 (time 1787033985 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 520 (time 1787034090.52 total 10.01 last 51.96) - open/close 1262 (time 1787034100.52 total 20.02 last 74.14) - open/close 1918 (time 1787034110.53 total 30.02 last 65.58) - open/close 2596 (time 1787034120.54 total 40.03 last 67.69) - open/close 3193 (time 1787034130.55 total 50.04 last 59.66) - open/close 3810 (time 1787034140.55 total 60.05 last 61.67) - open/close 4514 (time 1787034150.56 total 70.05 last 70.35) - open/close 5180 (time 1787034160.57 total 80.06 last 66.55) - open/close 5852 (time 1787034170.57 total 90.07 last 67.15) - open/close 6514 (time 1787034180.58 total 100.08 last 66.14) - open/close 7186 (time 1787034190.59 total 110.08 last 67.16) - open/close 7890 (time 1787034200.60 total 120.09 last 70.33) - open/close 8563 (time 1787034210.61 total 130.11 last 67.21) - open/close 9269 (time 1787034220.62 total 140.11 last 70.55) - open/close 9960 (time 1787034230.64 total 150.13 last 68.98) total: 10000 open/close in 150.65 seconds: 66.38 ops/second - unlinked 0 (time 1787034235 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 713 (time 1787034334.36 total 10.00 last 71.27) - open/close 1461 (time 1787034344.36 total 20.01 last 74.78) - open/close 2176 (time 1787034354.36 total 30.01 last 71.47) - open/close 2872 (time 1787034364.37 total 40.02 last 69.53) - open/close 3568 (time 1787034374.38 total 50.03 last 69.57) - open/close 4241 (time 1787034384.38 total 60.03 last 67.29) - open/close 4849 (time 1787034394.38 total 70.03 last 60.77) - open/close 5491 (time 1787034404.39 total 80.04 last 64.15) - open/close 6157 (time 1787034414.40 total 90.05 last 66.55) - open/close 6774 (time 1787034424.41 total 100.06 last 61.64) - open/close 7509 (time 1787034434.42 total 110.06 last 73.45) - open/close 8191 (time 1787034444.42 total 120.07 last 68.16) - open/close 8887 (time 1787034454.44 total 130.08 last 69.51) - open/close 9521 (time 1787034464.44 total 140.09 last 63.39) total: 10000 open/close in 147.12 seconds: 67.97 ops/second - unlinked 0 (time 1787034476 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 755 (time 1787034572.48 total 10.00 last 75.47) - open/close 1432 (time 1787034582.48 total 20.01 last 67.66) - open/close 2060 (time 1787034592.48 total 30.01 last 62.79) - open/close 2693 (time 1787034602.48 total 40.01 last 63.30) - open/close 3359 (time 1787034612.49 total 50.02 last 66.53) - open/close 4074 (time 1787034622.50 total 60.03 last 71.47) - open/close 4773 (time 1787034632.50 total 70.03 last 69.86) - open/close 5483 (time 1787034642.51 total 80.03 last 70.97) - open/close 6219 (time 1787034652.51 total 90.04 last 73.56) - open/close 6991 (time 1787034662.51 total 100.04 last 77.19) - open/close 7739 (time 1787034672.52 total 110.05 last 74.72) - open/close 8445 (time 1787034682.53 total 120.06 last 70.57) - open/close 9199 (time 1787034692.54 total 130.07 last 75.33) - open/close 9907 (time 1787034702.54 total 140.07 last 70.75) total: 10000 open/close in 141.27 seconds: 70.79 ops/second - unlinked 0 (time 1787034708 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 586 (time 1787034810.86 total 10.01 last 58.55) - open/close 1216 (time 1787034820.86 total 20.01 last 62.99) - open/close 1892 (time 1787034830.88 total 30.02 last 67.52) - open/close 2568 (time 1787034840.88 total 40.03 last 67.55) - open/close 3207 (time 1787034850.89 total 50.04 last 63.86) - open/close 3968 (time 1787034860.89 total 60.04 last 76.06) - open/close 4631 (time 1787034870.89 total 70.04 last 66.29) - open/close 5344 (time 1787034880.90 total 80.05 last 71.24) - open/close 6117 (time 1787034890.90 total 90.05 last 77.29) - open/close 6816 (time 1787034900.91 total 100.06 last 69.85) - open/close 7498 (time 1787034910.92 total 110.07 last 68.14) - open/close 8215 (time 1787034920.93 total 120.08 last 71.62) - open/close 8844 (time 1787034930.95 total 130.09 last 62.80) - open/close 9495 (time 1787034940.96 total 140.11 last 65.02) total: 10000 open/close in 147.98 seconds: 67.57 ops/second - unlinked 0 (time 1787034953 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 757 (time 1787035052.91 total 10.01 last 75.64) - open/close 1414 (time 1787035062.91 total 20.01 last 65.69) - open/close 2153 (time 1787035072.91 total 30.01 last 73.87) - open/close 2943 (time 1787035082.93 total 40.03 last 78.90) - open/close 3643 (time 1787035092.94 total 50.03 last 69.95) - open/close 4373 (time 1787035102.94 total 60.04 last 72.98) - open/close 5050 (time 1787035112.94 total 70.04 last 67.68) - open/close 5592 (time 1787035122.96 total 80.06 last 54.11) - open/close 6290 (time 1787035132.97 total 90.07 last 69.69) - open/close 6964 (time 1787035142.98 total 100.08 last 67.38) - open/close 7594 (time 1787035152.98 total 110.08 last 62.96) - open/close 8226 (time 1787035162.99 total 120.09 last 63.12) - open/close 8880 (time 1787035173.01 total 130.11 last 65.31) - open/close 9510 (time 1787035183.03 total 140.13 last 62.88) total: 10000 open/close in 147.12 seconds: 67.97 ops/second - unlinked 0 (time 1787035194 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.108@tcp sys.timeout=20 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 69 (2691s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 02:44:07 (1787035447) oleg408-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server SKIP 111 (245s) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3576 sec ========== 02:49:19 (1787035759) === conf-sanity: start cleanup 02:49:21 (1787035761) === === conf-sanity: finish cleanup 02:49:25 (1787035765) ===