-----============= acceptance-small: conf-sanity ============----- Tue Jun 23 09:11:22 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg114-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682451 > 34553653) for LU-16938, go 81 excepting tests: 32b 32c 32newtarball Stopping clients: oleg114-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg114-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg114-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg114-server oleg114-server: oleg114-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg114-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server == conf-sanity test 45: long unlink handling in ptlrpcd == 09:15:52 (1782220552) start mds service on oleg114-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg114-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: modules unloaded. pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 PASS 45 (338s) == conf-sanity test 69: replace an OST with the same index ========================================================== 09:21:31 (1782220891) start mds service on oleg114-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg114-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 655 (time 1782221040.19 total 10.01 last 65.46) - open/close 1354 (time 1782221050.19 total 20.01 last 69.89) - open/close 2065 (time 1782221060.20 total 30.02 last 71.04) - open/close 2813 (time 1782221070.20 total 40.02 last 74.78) - open/close 3392 (time 1782221080.21 total 50.02 last 57.89) - open/close 4170 (time 1782221090.22 total 60.03 last 77.71) - open/close 4791 (time 1782221100.23 total 70.04 last 62.02) - open/close 5368 (time 1782221110.25 total 80.06 last 57.61) - open/close 6019 (time 1782221120.25 total 90.06 last 65.09) - open/close 6753 (time 1782221130.25 total 100.06 last 73.38) - open/close 7510 (time 1782221140.26 total 110.07 last 75.63) - open/close 8202 (time 1782221150.26 total 120.08 last 69.18) - open/close 8928 (time 1782221160.27 total 130.08 last 72.58) - open/close 9574 (time 1782221170.28 total 140.10 last 64.50) total: 10000 open/close in 146.81 seconds: 68.12 ops/second - unlinked 0 (time 1782221182 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 691 (time 1782221282.03 total 10.00 last 69.08) - open/close 1513 (time 1782221292.05 total 20.02 last 82.08) - open/close 2174 (time 1782221302.05 total 30.02 last 66.08) - open/close 2766 (time 1782221312.07 total 40.04 last 59.09) - open/close 3230 (time 1782221322.08 total 50.05 last 46.33) - open/close 3698 (time 1782221332.11 total 60.08 last 46.66) - open/close 4333 (time 1782221342.12 total 70.09 last 63.45) - open/close 4883 (time 1782221352.14 total 80.11 last 54.91) - open/close 5517 (time 1782221362.15 total 90.12 last 63.33) - open/close 6178 (time 1782221372.16 total 100.13 last 66.03) - open/close 6751 (time 1782221382.16 total 110.13 last 57.27) - open/close 7593 (time 1782221392.17 total 120.14 last 84.14) - open/close 8181 (time 1782221402.18 total 130.15 last 58.75) - open/close 8634 (time 1782221412.18 total 140.15 last 45.29) - open/close 9182 (time 1782221422.18 total 150.15 last 54.79) - open/close 9866 (time 1782221432.19 total 160.17 last 68.31) total: 10000 open/close in 162.02 seconds: 61.72 ops/second - unlinked 0 (time 1782221439 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 630 (time 1782221543.11 total 10.01 last 62.97) - open/close 1319 (time 1782221553.12 total 20.01 last 68.87) - open/close 2005 (time 1782221563.12 total 30.01 last 68.60) - open/close 2759 (time 1782221573.13 total 40.02 last 75.35) - open/close 3579 (time 1782221583.14 total 50.03 last 81.92) - open/close 4232 (time 1782221593.15 total 60.04 last 65.22) - open/close 4972 (time 1782221603.16 total 70.05 last 73.90) - open/close 5752 (time 1782221613.17 total 80.06 last 77.92) - open/close 6516 (time 1782221623.18 total 90.07 last 76.37) - open/close 7123 (time 1782221633.18 total 100.08 last 60.65) - open/close 7741 (time 1782221643.21 total 110.10 last 61.63) - open/close 8233 (time 1782221653.22 total 120.11 last 49.17) - open/close 8766 (time 1782221663.25 total 130.14 last 53.14) - open/close 9175 (time 1782221673.27 total 140.16 last 40.81) - open/close 9720 (time 1782221683.27 total 150.16 last 54.49) total: 10000 open/close in 154.07 seconds: 64.91 ops/second - unlinked 0 (time 1782221693 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 682 (time 1782221799.66 total 10.02 last 68.07) - open/close 1435 (time 1782221809.67 total 20.02 last 75.26) - open/close 2066 (time 1782221819.68 total 30.04 last 63.01) - open/close 2653 (time 1782221829.68 total 40.04 last 58.70) - open/close 3294 (time 1782221839.70 total 50.05 last 64.02) - open/close 3935 (time 1782221849.70 total 60.05 last 64.09) - open/close 4545 (time 1782221859.72 total 70.08 last 60.87) - open/close 5088 (time 1782221869.73 total 80.09 last 54.23) - open/close 5670 (time 1782221879.74 total 90.10 last 58.16) - open/close 6304 (time 1782221889.75 total 100.11 last 63.29) - open/close 6876 (time 1782221899.76 total 110.12 last 57.17) - open/close 7444 (time 1782221909.76 total 120.12 last 56.79) - open/close 7991 (time 1782221919.79 total 130.15 last 54.55) - open/close 8570 (time 1782221929.81 total 140.17 last 57.79) - open/close 9086 (time 1782221939.81 total 150.17 last 51.59) - open/close 9691 (time 1782221949.82 total 160.18 last 60.45) total: 10000 open/close in 166.45 seconds: 60.08 ops/second - unlinked 0 (time 1782221963 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 674 (time 1782222064.17 total 10.00 last 67.39) - open/close 1407 (time 1782222074.18 total 20.01 last 73.21) - open/close 2066 (time 1782222084.18 total 30.02 last 65.86) - open/close 2821 (time 1782222094.19 total 40.02 last 75.48) - open/close 3393 (time 1782222104.19 total 50.02 last 57.19) - open/close 4062 (time 1782222114.20 total 60.03 last 66.85) - open/close 4684 (time 1782222124.20 total 70.03 last 62.20) - open/close 5311 (time 1782222134.20 total 80.04 last 62.66) - open/close 6021 (time 1782222144.22 total 90.06 last 70.86) - open/close 6735 (time 1782222154.23 total 100.06 last 71.38) - open/close 7537 (time 1782222164.24 total 110.07 last 80.10) - open/close 8245 (time 1782222174.25 total 120.08 last 70.75) - open/close 8832 (time 1782222184.26 total 130.10 last 58.60) - open/close 9584 (time 1782222194.28 total 140.12 last 75.03) total: 10000 open/close in 146.66 seconds: 68.19 ops/second - unlinked 0 (time 1782222207 ; total 0 ; last 0) total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second - open/close 694 (time 1782222317.71 total 10.02 last 69.27) - open/close 1278 (time 1782222327.72 total 20.02 last 58.36) - open/close 1670 (time 1782222337.72 total 30.03 last 39.18) - open/close 2191 (time 1782222347.72 total 40.03 last 52.09) - open/close 2720 (time 1782222357.73 total 50.04 last 52.83) - open/close 3312 (time 1782222367.74 total 60.05 last 59.14) - open/close 3779 (time 1782222377.75 total 70.06 last 46.66) - open/close 4411 (time 1782222387.75 total 80.06 last 63.19) - open/close 4995 (time 1782222397.77 total 90.07 last 58.33) - open/close 5571 (time 1782222407.77 total 100.08 last 57.58) - open/close 6237 (time 1782222417.77 total 110.08 last 66.58) - open/close 6929 (time 1782222427.79 total 120.09 last 69.10) - open/close 7540 (time 1782222437.79 total 130.10 last 61.10) - open/close 8120 (time 1782222447.80 total 140.11 last 57.90) - open/close 8738 (time 1782222457.81 total 150.12 last 61.76) - open/close 9504 (time 1782222467.81 total 160.12 last 76.59) total: 10000 open/close in 166.24 seconds: 60.15 ops/second - unlinked 0 (time 1782222480 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 577 (time 1782222583.57 total 10.01 last 57.64) - open/close 1239 (time 1782222593.58 total 20.02 last 66.16) - open/close 1881 (time 1782222603.58 total 30.02 last 64.18) - open/close 2698 (time 1782222613.59 total 40.02 last 81.65) - open/close 3407 (time 1782222623.59 total 50.03 last 70.89) - open/close 4210 (time 1782222633.59 total 60.03 last 80.29) - open/close 4905 (time 1782222643.60 total 70.04 last 69.43) - open/close 5480 (time 1782222653.60 total 80.04 last 57.48) - open/close 6032 (time 1782222663.61 total 90.05 last 55.14) - open/close 6886 (time 1782222673.62 total 100.06 last 85.35) - open/close 7577 (time 1782222683.62 total 110.06 last 69.08) - open/close 8155 (time 1782222693.63 total 120.07 last 57.74) - open/close 8816 (time 1782222703.66 total 130.09 last 65.94) - open/close 9480 (time 1782222713.67 total 140.10 last 66.31) total: 10000 open/close in 147.71 seconds: 67.70 ops/second - unlinked 0 (time 1782222726 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 781 (time 1782222824.50 total 10.00 last 78.07) - open/close 1497 (time 1782222834.50 total 20.00 last 71.60) - open/close 2142 (time 1782222844.50 total 30.01 last 64.48) - open/close 2856 (time 1782222854.50 total 40.01 last 71.38) - open/close 3716 (time 1782222864.51 total 50.02 last 85.93) - open/close 4447 (time 1782222874.52 total 60.03 last 73.00) - open/close 5077 (time 1782222884.53 total 70.04 last 62.95) - open/close 5751 (time 1782222894.54 total 80.05 last 67.33) - open/close 6592 (time 1782222904.55 total 90.05 last 84.04) - open/close 7270 (time 1782222914.56 total 100.06 last 67.73) - open/close 8045 (time 1782222924.56 total 110.07 last 77.47) - open/close 8831 (time 1782222934.56 total 120.07 last 78.58) - open/close 9537 (time 1782222944.58 total 130.09 last 70.47) total: 10000 open/close in 135.40 seconds: 73.86 ops/second - unlinked 0 (time 1782222954 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 655 (time 1782223055.42 total 10.03 last 65.33) - open/close 1366 (time 1782223065.43 total 20.03 last 71.07) - open/close 1970 (time 1782223075.43 total 30.03 last 60.39) - open/close 2648 (time 1782223085.44 total 40.04 last 67.74) - open/close 3504 (time 1782223095.44 total 50.05 last 85.55) - open/close 4306 (time 1782223105.45 total 60.05 last 80.15) - open/close 5032 (time 1782223115.46 total 70.06 last 72.50) - open/close 5858 (time 1782223125.47 total 80.07 last 82.54) - open/close 6577 (time 1782223135.47 total 90.08 last 71.87) - open/close 7282 (time 1782223145.47 total 100.08 last 70.47) - open/close 8083 (time 1782223155.48 total 110.09 last 80.02) - open/close 8822 (time 1782223165.49 total 120.09 last 73.88) - open/close 9407 (time 1782223175.50 total 130.10 last 58.43) total: 10000 open/close in 139.74 seconds: 71.56 ops/second - unlinked 0 (time 1782223191 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 619 (time 1782223295.71 total 10.02 last 61.79) - open/close 1178 (time 1782223305.72 total 20.03 last 55.86) - open/close 1802 (time 1782223315.75 total 30.05 last 62.22) - open/close 2435 (time 1782223325.76 total 40.07 last 63.23) - open/close 3206 (time 1782223335.77 total 50.08 last 77.03) - open/close 3907 (time 1782223345.78 total 60.08 last 70.04) - open/close 4685 (time 1782223355.78 total 70.09 last 77.78) - open/close 5371 (time 1782223365.78 total 80.09 last 68.56) - open/close 6049 (time 1782223375.80 total 90.11 last 67.70) - open/close 6671 (time 1782223385.81 total 100.12 last 62.14) - open/close 7279 (time 1782223395.81 total 110.12 last 60.77) - open/close 8003 (time 1782223405.82 total 120.12 last 72.37) - open/close 8759 (time 1782223415.82 total 130.13 last 75.56) - open/close 9458 (time 1782223425.83 total 140.14 last 69.87) total: 10000 open/close in 148.02 seconds: 67.56 ops/second - unlinked 0 (time 1782223439 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.114@tcp sys.timeout=20 start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: modules unloaded. pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 PASS 69 (2842s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 10:08:54 (1782223734) oleg114-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg114-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg114-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 oleg114-server: oleg114-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg114-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg114-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg114-client.virtnet /mnt/lustre2 (opts:-f) oleg114-server: oleg114-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg114-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg114-server: oleg114-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg114-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server SKIP 111 (300s) Stopping clients: oleg114-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg114-client.virtnet /mnt/lustre2 (opts:-f) oleg114-server: oleg114-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3829 sec ========== 10:15:14 (1782224114) === conf-sanity: start cleanup 10:15:16 (1782224116) === === conf-sanity: finish cleanup 10:15:21 (1782224121) ===