-----============= acceptance-small: conf-sanity ============----- Wed Aug 26 18:32:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg451-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34683191 > 34553653) for LU-16938, go 81 - see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683191 >= 34554143) for LU-18348, go 32b 32c excepting tests: 32b 32c 32newtarball Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg451-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg451-server oleg451-server: oleg451-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server == conf-sanity test 45: long unlink handling in ptlrpcd == 18:36:35 (1787783795) start mds service on oleg451-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg451-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: modules unloaded. pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 45 (298s) == conf-sanity test 69: replace an OST with the same index ========================================================== 18:41:33 (1787784093) start mds service on oleg451-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg451-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 615 (time 1787784229.27 total 10.00 last 61.48) - open/close 1214 (time 1787784239.27 total 20.01 last 59.88) - open/close 1853 (time 1787784249.27 total 30.01 last 63.88) - open/close 2449 (time 1787784259.28 total 40.02 last 59.53) - open/close 3184 (time 1787784269.30 total 50.03 last 73.40) - open/close 3871 (time 1787784279.32 total 60.06 last 68.52) - open/close 4527 (time 1787784289.32 total 70.06 last 65.59) - open/close 5163 (time 1787784299.33 total 80.06 last 63.59) - open/close 5837 (time 1787784309.33 total 90.07 last 67.35) - open/close 6382 (time 1787784319.35 total 100.09 last 54.40) - open/close 7069 (time 1787784329.37 total 110.10 last 68.59) - open/close 7839 (time 1787784339.38 total 120.12 last 76.90) - open/close 8454 (time 1787784349.38 total 130.12 last 61.50) - open/close 9078 (time 1787784359.39 total 140.13 last 62.33) - open/close 9684 (time 1787784369.39 total 150.13 last 60.59) total: 10000 open/close in 156.12 seconds: 64.06 ops/second - unlinked 0 (time 1787784380 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 695 (time 1787784481.14 total 10.01 last 69.44) - open/close 1346 (time 1787784491.15 total 20.02 last 65.05) - open/close 1999 (time 1787784501.16 total 30.03 last 65.22) - open/close 2734 (time 1787784511.17 total 40.03 last 73.46) - open/close 3519 (time 1787784521.17 total 50.04 last 78.45) - open/close 4154 (time 1787784531.18 total 60.05 last 63.45) - open/close 4899 (time 1787784541.18 total 70.05 last 74.49) - open/close 5605 (time 1787784551.20 total 80.06 last 70.52) - open/close 6262 (time 1787784561.21 total 90.08 last 65.58) - open/close 6893 (time 1787784571.22 total 100.09 last 63.05) - open/close 7433 (time 1787784581.25 total 110.11 last 53.86) - open/close 8054 (time 1787784591.27 total 120.13 last 61.99) - open/close 8715 (time 1787784601.28 total 130.14 last 66.02) - open/close 9369 (time 1787784611.28 total 140.15 last 65.37) - open/close 9917 (time 1787784621.28 total 150.15 last 54.80) total: 10000 open/close in 151.44 seconds: 66.03 ops/second - unlinked 0 (time 1787784627 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 666 (time 1787784729.48 total 10.01 last 66.55) - open/close 1419 (time 1787784739.49 total 20.02 last 75.22) - open/close 2178 (time 1787784749.49 total 30.02 last 75.89) - open/close 2894 (time 1787784759.49 total 40.02 last 71.60) - open/close 3633 (time 1787784769.51 total 50.04 last 73.74) - open/close 4361 (time 1787784779.52 total 60.05 last 72.75) - open/close 5094 (time 1787784789.52 total 70.05 last 73.30) - open/close 5655 (time 1787784799.54 total 80.07 last 55.99) - open/close 6273 (time 1787784809.54 total 90.07 last 61.76) - open/close 6975 (time 1787784819.55 total 100.08 last 70.14) - open/close 7667 (time 1787784829.57 total 110.10 last 69.09) - open/close 8383 (time 1787784839.59 total 120.11 last 71.48) - open/close 9097 (time 1787784849.59 total 130.12 last 71.37) - open/close 9778 (time 1787784859.61 total 140.14 last 67.98) total: 10000 open/close in 143.83 seconds: 69.53 ops/second - unlinked 0 (time 1787784868 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 703 (time 1787784967.73 total 10.01 last 70.20) - open/close 1313 (time 1787784977.76 total 20.05 last 60.79) - open/close 1968 (time 1787784987.76 total 30.05 last 65.49) - open/close 2729 (time 1787784997.77 total 40.06 last 76.07) - open/close 3422 (time 1787785007.78 total 50.07 last 69.23) - open/close 4158 (time 1787785017.78 total 60.07 last 73.60) - open/close 4951 (time 1787785027.78 total 70.07 last 79.26) - open/close 5761 (time 1787785037.79 total 80.08 last 80.92) - open/close 6538 (time 1787785047.80 total 90.09 last 77.65) - open/close 7254 (time 1787785057.80 total 100.09 last 71.59) - open/close 7868 (time 1787785067.80 total 110.09 last 61.40) - open/close 8469 (time 1787785077.81 total 120.10 last 60.03) - open/close 9126 (time 1787785087.82 total 130.11 last 65.62) - open/close 9692 (time 1787785097.83 total 140.12 last 56.57) total: 10000 open/close in 144.53 seconds: 69.19 ops/second - unlinked 0 (time 1787785108 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 706 (time 1787785207.29 total 10.02 last 70.49) - open/close 1451 (time 1787785217.29 total 20.02 last 74.47) - open/close 2207 (time 1787785227.30 total 30.03 last 75.51) - open/close 2993 (time 1787785237.33 total 40.05 last 78.42) - open/close 3748 (time 1787785247.33 total 50.06 last 75.47) - open/close 4383 (time 1787785257.36 total 60.08 last 63.34) - open/close 4944 (time 1787785267.36 total 70.09 last 56.06) - open/close 5549 (time 1787785277.39 total 80.12 last 60.33) - open/close 6160 (time 1787785287.41 total 90.14 last 60.97) - open/close 6799 (time 1787785297.42 total 100.15 last 63.86) - open/close 7560 (time 1787785307.45 total 110.18 last 75.88) - open/close 8282 (time 1787785317.47 total 120.20 last 72.03) - open/close 9013 (time 1787785327.47 total 130.20 last 73.10) - open/close 9717 (time 1787785337.48 total 140.21 last 70.32) total: 10000 open/close in 144.66 seconds: 69.13 ops/second - unlinked 0 (time 1787785346 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 623 (time 1787785453.26 total 10.02 last 62.18) - open/close 1323 (time 1787785463.27 total 20.03 last 69.93) - open/close 1933 (time 1787785473.27 total 30.03 last 60.99) - open/close 2622 (time 1787785483.28 total 40.04 last 68.87) - open/close 3278 (time 1787785493.28 total 50.04 last 65.55) - open/close 4016 (time 1787785503.29 total 60.05 last 73.79) - open/close 4729 (time 1787785513.29 total 70.05 last 71.29) - open/close 5362 (time 1787785523.29 total 80.05 last 63.28) - open/close 6088 (time 1787785533.30 total 90.06 last 72.54) - open/close 6736 (time 1787785543.31 total 100.07 last 64.76) - open/close 7382 (time 1787785553.31 total 110.07 last 64.59) - open/close 8040 (time 1787785563.31 total 120.07 last 65.75) - open/close 8736 (time 1787785573.33 total 130.09 last 69.51) - open/close 9437 (time 1787785583.34 total 140.10 last 69.99) total: 10000 open/close in 148.09 seconds: 67.52 ops/second - unlinked 0 (time 1787785595 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 651 (time 1787785691.23 total 10.00 last 65.07) - open/close 1375 (time 1787785701.24 total 20.01 last 72.36) - open/close 2026 (time 1787785711.24 total 30.01 last 65.10) - open/close 2629 (time 1787785721.24 total 40.01 last 60.28) - open/close 3372 (time 1787785731.24 total 50.01 last 74.29) - open/close 4171 (time 1787785741.25 total 60.02 last 79.86) - open/close 4946 (time 1787785751.26 total 70.03 last 77.45) - open/close 5751 (time 1787785761.26 total 80.03 last 80.44) - open/close 6460 (time 1787785771.27 total 90.04 last 70.88) - open/close 7191 (time 1787785781.28 total 100.05 last 73.02) - open/close 7860 (time 1787785791.28 total 110.05 last 66.88) - open/close 8516 (time 1787785801.31 total 120.08 last 65.41) - open/close 9120 (time 1787785811.33 total 130.10 last 60.29) - open/close 9669 (time 1787785821.35 total 140.12 last 54.79) total: 10000 open/close in 145.57 seconds: 68.69 ops/second - unlinked 0 (time 1787785830 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 695 (time 1787785922.92 total 10.02 last 69.38) - open/close 1448 (time 1787785932.93 total 20.03 last 75.18) - open/close 2196 (time 1787785942.96 total 30.06 last 74.62) - open/close 2927 (time 1787785952.97 total 40.07 last 73.01) - open/close 3646 (time 1787785962.97 total 50.07 last 71.88) - open/close 4361 (time 1787785972.97 total 60.07 last 71.49) - open/close 5073 (time 1787785982.98 total 70.08 last 71.15) - open/close 5786 (time 1787785992.99 total 80.09 last 71.20) - open/close 6469 (time 1787786003.00 total 90.10 last 68.25) - open/close 7208 (time 1787786013.01 total 100.11 last 73.86) - open/close 8023 (time 1787786023.01 total 110.11 last 81.46) - open/close 8791 (time 1787786033.03 total 120.13 last 76.63) - open/close 9532 (time 1787786043.04 total 130.14 last 74.02) total: 10000 open/close in 136.35 seconds: 73.34 ops/second - unlinked 0 (time 1787786052 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 632 (time 1787786140.98 total 10.01 last 63.16) - open/close 1355 (time 1787786150.99 total 20.02 last 72.21) - open/close 2118 (time 1787786161.00 total 30.03 last 76.22) - open/close 2857 (time 1787786171.01 total 40.04 last 73.85) - open/close 3562 (time 1787786181.01 total 50.04 last 70.50) - open/close 4311 (time 1787786191.02 total 60.05 last 74.82) - open/close 4968 (time 1787786201.02 total 70.05 last 65.69) - open/close 5626 (time 1787786211.03 total 80.06 last 65.75) - open/close 6226 (time 1787786221.03 total 90.06 last 59.96) - open/close 6861 (time 1787786231.03 total 100.06 last 63.50) - open/close 7367 (time 1787786241.04 total 110.07 last 50.56) - open/close 8066 (time 1787786251.05 total 120.08 last 69.84) - open/close 8706 (time 1787786261.07 total 130.10 last 63.89) - open/close 9332 (time 1787786271.08 total 140.11 last 62.51) total: 10000 open/close in 149.65 seconds: 66.82 ops/second - unlinked 0 (time 1787786285 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 792 (time 1787786373.57 total 10.01 last 79.15) - open/close 1510 (time 1787786383.58 total 20.01 last 71.78) - open/close 2278 (time 1787786393.59 total 30.03 last 76.67) - open/close 3036 (time 1787786403.60 total 40.03 last 75.75) - open/close 3796 (time 1787786413.60 total 50.03 last 75.99) - open/close 4506 (time 1787786423.62 total 60.06 last 70.85) - open/close 5142 (time 1787786433.65 total 70.09 last 63.39) - open/close 5929 (time 1787786443.66 total 80.10 last 78.62) - open/close 6630 (time 1787786453.67 total 90.10 last 70.09) - open/close 7382 (time 1787786463.67 total 100.10 last 75.18) - open/close 8132 (time 1787786473.67 total 110.11 last 74.98) - open/close 8877 (time 1787786483.68 total 120.11 last 74.45) - open/close 9589 (time 1787786493.68 total 130.12 last 71.17) total: 10000 open/close in 136.00 seconds: 73.53 ops/second - unlinked 0 (time 1787786504 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.151@tcp sys.timeout=20 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg451-client.virtnet: -o user_xattr,flock 192.168.204.151@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg451-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server unloading modules via unload_modules_local on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing unload_modules_local oleg451-server: modules unloaded. pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-client: ssh exited with exit code 2 pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 PASS 69 (2650s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 19:25:44 (1787786744) oleg451-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg451-client: oleg451-server: ssh exited with exit code 2 oleg451-server: oleg451-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions oleg451-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg451-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg451-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) oleg451-server: oleg451-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg451-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg451-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg451-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg451-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg451-server: oleg451-server.virtnet: executing set_default_debug -1 all pdsh@oleg451-client: oleg451-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg451-server: oleg451-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg451-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg451-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg451-server stop mds service on oleg451-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg451-server SKIP 111 (221s) Stopping clients: oleg451-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg451-client.virtnet /mnt/lustre2 (opts:-f) oleg451-server: oleg451-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg451-server' oleg451-server: oleg451-server.virtnet: executing load_modules_local oleg451-server: Loading modules from /home/green/git/lustre-release/lustre oleg451-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg451-server: detected 4 online CPUs by sysfs oleg451-server: MODOPTS_LIBCFS= oleg451-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3471 sec ========== 19:30:30 (1787787030) === conf-sanity: start cleanup 19:30:32 (1787787032) === === conf-sanity: finish cleanup 19:30:35 (1787787035) ===