-----============= acceptance-small: conf-sanity ============----- Tue Apr 14 13:47:52 EDT 2026 oleg224-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg224-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg224-server oleg224-server: oleg224-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg224-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg224-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server == conf-sanity test 45: long unlink handling in ptlrpcd == 13:51:05 (1776189065) start mds service on oleg224-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg224-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg224-server: LNET ready to unload oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 PASS 45 (248s) == conf-sanity test 69: replace an OST with the same index ========================================================== 13:55:13 (1776189313) start mds service on oleg224-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 929 (time 1776189412.20 total 10.00 last 92.88) - open/close 1857 (time 1776189422.20 total 20.01 last 92.75) - open/close 2667 (time 1776189432.21 total 30.01 last 80.96) - open/close 3537 (time 1776189442.21 total 40.02 last 86.94) - open/close 4396 (time 1776189452.22 total 50.02 last 85.88) - open/close 5211 (time 1776189462.22 total 60.02 last 81.50) - open/close 6053 (time 1776189472.23 total 70.04 last 84.05) - open/close 6838 (time 1776189482.24 total 80.04 last 78.47) - open/close 7697 (time 1776189492.25 total 90.06 last 85.80) - open/close 8362 (time 1776189502.26 total 100.06 last 66.46) - open/close 9279 (time 1776189512.26 total 110.07 last 91.67) - open/close 9919 (time 1776189522.28 total 120.09 last 63.87) total: 10000 open/close in 121.29 seconds: 82.44 ops/second - unlinked 0 (time 1776189529 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 809 (time 1776189619.93 total 10.00 last 80.90) - open/close 1668 (time 1776189629.93 total 20.00 last 85.89) - open/close 2607 (time 1776189639.93 total 30.00 last 93.88) - open/close 3597 (time 1776189649.94 total 40.01 last 98.96) - open/close 4511 (time 1776189659.95 total 50.02 last 91.32) - open/close 5395 (time 1776189669.96 total 60.03 last 88.32) - open/close 6379 (time 1776189679.96 total 70.03 last 98.38) - open/close 7318 (time 1776189689.96 total 80.03 last 93.85) - open/close 8299 (time 1776189699.97 total 90.04 last 98.05) - open/close 9250 (time 1776189709.98 total 100.05 last 95.01) total: 10000 open/close in 108.86 seconds: 91.86 ops/second - unlinked 0 (time 1776189723 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 626 (time 1776189806.43 total 10.01 last 62.54) - open/close 1494 (time 1776189816.44 total 20.01 last 86.77) - open/close 2233 (time 1776189826.45 total 30.03 last 73.80) - open/close 3016 (time 1776189836.47 total 40.05 last 78.13) - open/close 3850 (time 1776189846.48 total 50.06 last 83.34) - open/close 4373 (time 1776189856.52 total 60.09 last 52.10) - open/close 5013 (time 1776189866.52 total 70.10 last 63.97) - open/close 5844 (time 1776189876.55 total 80.12 last 82.91) - open/close 6655 (time 1776189886.55 total 90.13 last 81.07) - open/close 7457 (time 1776189896.56 total 100.13 last 80.12) - open/close 8134 (time 1776189906.56 total 110.14 last 67.67) - open/close 8844 (time 1776189916.57 total 120.14 last 70.98) - open/close 9595 (time 1776189926.57 total 130.15 last 75.05) total: 10000 open/close in 135.32 seconds: 73.90 ops/second - unlinked 0 (time 1776189936 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 724 (time 1776190018.96 total 10.01 last 72.36) - open/close 1516 (time 1776190028.96 total 20.01 last 79.18) - open/close 2315 (time 1776190038.97 total 30.01 last 79.87) - open/close 3230 (time 1776190048.98 total 40.02 last 91.38) - open/close 4060 (time 1776190058.98 total 50.03 last 82.96) - open/close 4880 (time 1776190069.00 total 60.04 last 81.87) - open/close 5564 (time 1776190079.01 total 70.05 last 68.34) - open/close 6252 (time 1776190089.01 total 80.06 last 68.78) - open/close 6936 (time 1776190099.02 total 90.06 last 68.36) - open/close 7730 (time 1776190109.02 total 100.07 last 79.36) - open/close 8556 (time 1776190119.03 total 110.07 last 82.57) - open/close 9255 (time 1776190129.04 total 120.08 last 69.81) total: 10000 open/close in 129.21 seconds: 77.39 ops/second - unlinked 0 (time 1776190142 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 689 (time 1776190245.49 total 10.02 last 68.79) - open/close 1421 (time 1776190255.50 total 20.02 last 73.15) - open/close 2082 (time 1776190265.51 total 30.03 last 66.04) - open/close 2829 (time 1776190275.52 total 40.04 last 74.64) - open/close 3549 (time 1776190285.52 total 50.04 last 72.00) - open/close 4337 (time 1776190295.53 total 60.06 last 78.68) - open/close 4999 (time 1776190305.54 total 70.07 last 66.14) - open/close 5656 (time 1776190315.55 total 80.07 last 65.66) - open/close 6406 (time 1776190325.55 total 90.07 last 74.99) - open/close 7312 (time 1776190335.56 total 100.08 last 90.54) - open/close 8152 (time 1776190345.57 total 110.09 last 83.91) - open/close 8944 (time 1776190355.57 total 120.09 last 79.18) - open/close 9546 (time 1776190365.58 total 130.11 last 60.12) total: 10000 open/close in 138.15 seconds: 72.39 ops/second - unlinked 0 (time 1776190377 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 765 (time 1776190472.82 total 10.03 last 76.30) - open/close 1466 (time 1776190482.82 total 20.03 last 70.06) - open/close 2173 (time 1776190492.84 total 30.05 last 70.59) - open/close 2727 (time 1776190502.88 total 40.09 last 55.17) - open/close 3398 (time 1776190512.89 total 50.10 last 67.04) - open/close 3946 (time 1776190522.90 total 60.11 last 54.75) - open/close 4749 (time 1776190532.90 total 70.11 last 80.25) - open/close 5437 (time 1776190542.91 total 80.12 last 68.74) - open/close 6116 (time 1776190552.91 total 90.12 last 67.89) - open/close 6929 (time 1776190562.92 total 100.13 last 81.26) - open/close 7651 (time 1776190572.93 total 110.14 last 72.13) - open/close 8276 (time 1776190582.93 total 120.14 last 62.46) - open/close 8867 (time 1776190592.94 total 130.15 last 59.06) - open/close 9627 (time 1776190602.95 total 140.16 last 75.94) total: 10000 open/close in 145.26 seconds: 68.84 ops/second - unlinked 0 (time 1776190612 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 607 (time 1776190721.41 total 10.01 last 60.66) - open/close 1252 (time 1776190731.42 total 20.02 last 64.43) - open/close 1879 (time 1776190741.44 total 30.04 last 62.56) - open/close 2422 (time 1776190751.44 total 40.04 last 54.29) - open/close 3114 (time 1776190761.45 total 50.05 last 69.16) - open/close 3730 (time 1776190771.45 total 60.05 last 61.57) - open/close 4430 (time 1776190781.46 total 70.06 last 69.92) - open/close 5159 (time 1776190791.48 total 80.08 last 72.78) - open/close 5930 (time 1776190801.49 total 90.09 last 76.99) - open/close 6500 (time 1776190811.51 total 100.11 last 56.93) - open/close 7187 (time 1776190821.51 total 110.11 last 68.68) - open/close 7941 (time 1776190831.52 total 120.12 last 75.33) - open/close 8745 (time 1776190841.52 total 130.12 last 80.36) - open/close 9440 (time 1776190851.53 total 140.13 last 69.45) total: 10000 open/close in 148.17 seconds: 67.49 ops/second - unlinked 0 (time 1776190863 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 634 (time 1776190966.88 total 10.00 last 63.39) - open/close 1125 (time 1776190976.89 total 20.01 last 49.04) - open/close 1841 (time 1776190986.89 total 30.02 last 71.56) - open/close 2459 (time 1776190996.89 total 40.02 last 61.80) - open/close 3072 (time 1776191006.91 total 50.04 last 61.19) - open/close 3684 (time 1776191016.92 total 60.04 last 61.17) - open/close 4307 (time 1776191026.92 total 70.04 last 62.29) - open/close 5004 (time 1776191036.93 total 80.05 last 69.65) - open/close 5726 (time 1776191046.93 total 90.06 last 72.15) - open/close 6559 (time 1776191056.94 total 100.06 last 83.28) - open/close 7185 (time 1776191066.95 total 110.07 last 62.54) - open/close 7763 (time 1776191076.95 total 120.07 last 57.80) - open/close 8582 (time 1776191086.95 total 130.07 last 81.89) - open/close 9315 (time 1776191096.95 total 140.08 last 73.29) total: 10000 open/close in 149.81 seconds: 66.75 ops/second - unlinked 0 (time 1776191111 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 665 (time 1776191206.84 total 10.00 last 66.47) - open/close 1272 (time 1776191216.85 total 20.01 last 60.66) - open/close 2018 (time 1776191226.86 total 30.02 last 74.52) - open/close 2799 (time 1776191236.87 total 40.03 last 78.06) - open/close 3517 (time 1776191246.87 total 50.03 last 71.77) - open/close 4145 (time 1776191256.88 total 60.04 last 62.73) - open/close 4905 (time 1776191266.89 total 70.05 last 75.92) - open/close 5752 (time 1776191276.90 total 80.06 last 84.65) - open/close 6555 (time 1776191286.90 total 90.06 last 80.25) - open/close 7417 (time 1776191296.91 total 100.07 last 86.16) - open/close 8241 (time 1776191306.91 total 110.07 last 82.37) - open/close 8965 (time 1776191316.92 total 120.08 last 72.37) - open/close 9839 (time 1776191326.93 total 130.09 last 87.27) total: 10000 open/close in 132.10 seconds: 75.70 ops/second - unlinked 0 (time 1776191333 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 686 (time 1776191420.50 total 10.01 last 68.55) - open/close 1370 (time 1776191430.54 total 20.05 last 68.09) - open/close 2116 (time 1776191440.55 total 30.06 last 74.54) - open/close 2992 (time 1776191450.55 total 40.06 last 87.59) - open/close 3754 (time 1776191460.55 total 50.06 last 76.20) - open/close 4580 (time 1776191470.56 total 60.07 last 82.55) - open/close 5252 (time 1776191480.56 total 70.07 last 67.16) - open/close 5914 (time 1776191490.59 total 80.10 last 66.03) - open/close 6660 (time 1776191500.59 total 90.10 last 74.60) - open/close 7576 (time 1776191510.60 total 100.11 last 91.49) - open/close 8458 (time 1776191520.61 total 110.13 last 88.06) - open/close 9203 (time 1776191530.63 total 120.14 last 74.41) - open/close 9807 (time 1776191540.63 total 130.14 last 60.38) total: 10000 open/close in 133.03 seconds: 75.17 ops/second - unlinked 0 (time 1776191549 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg224-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg224-server: LNET ready to unload oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 PASS 69 (2421s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 14:35:35 (1776191735) oleg224-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present oleg224-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present umount lustre on /mnt/lustre..... stop ost1 service on oleg224-server stop mds service on oleg224-server stop mds service on oleg224-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O ea_inode,large_dir -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg224-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre Starting client oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre Started clients oleg224-client.virtnet: 192.168.202.124@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) mount lustre on /mnt/lustre2..... Starting client: oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre2 Starting client oleg224-client.virtnet: -o user_xattr,flock oleg224-server@tcp:/lustre /mnt/lustre2 Started clients oleg224-client.virtnet: 192.168.202.124@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1326500 1696 1206392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1252 126964 1% /mnt/lustre[OST:0] filesystem_summary: 142216 1252 126964 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 960000 272 959728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 268 49732 1% /mnt/lustre[OST:0] filesystem_summary: 50004 272 49732 1% /mnt/lustre creating 60000 hardlinks to oleg224-client.virtnet-0 creating 60000 hardlinks to oleg224-client.virtnet-1 waiting for PIDs 27039 27054 to complete - link 642 (time 1776191846.44 total 10.00 last 64.18) - link 1377 (time 1776191856.44 total 20.00 last 73.50) - link 2024 (time 1776191866.44 total 30.01 last 64.65) - link 2660 (time 1776191876.46 total 40.02 last 63.51) - link 3171 (time 1776191886.46 total 50.03 last 51.06) - link 3794 (time 1776191896.47 total 60.04 last 62.28) - link 4448 (time 1776191906.49 total 70.05 last 65.27) - link 4907 (time 1776191916.52 total 80.08 last 45.76) - link 5436 (time 1776191926.54 total 90.11 last 52.77) - link 6013 (time 1776191936.55 total 100.12 last 57.63) - link 6624 (time 1776191946.57 total 110.13 last 61.02) - link 7240 (time 1776191956.57 total 120.14 last 61.59) - link 7832 (time 1776191966.57 total 130.14 last 59.18) - link 8432 (time 1776191976.58 total 140.15 last 59.94) - link 9103 (time 1776191986.59 total 150.16 last 67.03) - link 9693 (time 1776191996.59 total 160.16 last 59.00) - link 10000 (time 1776192001.50 total 165.07 last 62.54) - link 10644 (time 1776192011.51 total 175.08 last 64.35) - link 11327 (time 1776192021.53 total 185.09 last 68.18) - link 11910 (time 1776192031.54 total 195.11 last 58.21) - link 12448 (time 1776192041.54 total 205.11 last 53.79) - link 13043 (time 1776192051.54 total 215.11 last 59.49) - link 13664 (time 1776192061.56 total 225.12 last 62.02) - link 14342 (time 1776192071.57 total 235.14 last 67.70) - link 15020 (time 1776192081.58 total 245.15 last 67.75) - link 15707 (time 1776192091.59 total 255.16 last 68.60) - link 16413 (time 1776192101.60 total 265.17 last 70.52) - link 17114 (time 1776192111.61 total 275.18 last 70.03) - link 17886 (time 1776192121.62 total 285.19 last 77.12) - link 18792 (time 1776192131.63 total 295.20 last 90.57) - link 19581 (time 1776192141.63 total 305.20 last 78.88) - link 20000 (time 1776192147.77 total 311.34 last 68.21) - link 20732 (time 1776192157.79 total 321.35 last 73.11) - link 21459 (time 1776192167.80 total 331.37 last 72.61) - link 22106 (time 1776192177.80 total 341.37 last 64.69) - link 22783 (time 1776192187.81 total 351.38 last 67.61) - link 23403 (time 1776192197.84 total 361.40 last 61.85) - link 24039 (time 1776192207.84 total 371.41 last 63.59) - link 24752 (time 1776192217.84 total 381.41 last 71.26) - link 25441 (time 1776192227.86 total 391.43 last 68.80) - link 25979 (time 1776192237.87 total 401.44 last 53.72) - link 26362 (time 1776192247.89 total 411.46 last 38.23) - link 26761 (time 1776192257.91 total 421.47 last 39.85) - link 27180 (time 1776192267.91 total 431.48 last 41.87) - link 27771 (time 1776192277.92 total 441.49 last 59.04) - link 28331 (time 1776192287.93 total 451.50 last 55.95) - link 28935 (time 1776192297.95 total 461.52 last 60.27) - link 29453 (time 1776192307.97 total 471.54 last 51.71) - link 29950 (time 1776192317.99 total 481.55 last 49.63) - link 30499 (time 1776192327.99 total 491.55 last 54.89) - link 30982 (time 1776192338.00 total 501.56 last 48.26) - link 31573 (time 1776192348.01 total 511.58 last 59.02) - link 32190 (time 1776192358.02 total 521.58 last 61.66) - link 32808 (time 1776192368.03 total 531.59 last 61.74) - link 33379 (time 1776192378.03 total 541.59 last 57.09) - link 33868 (time 1776192388.03 total 551.60 last 48.88) - link 34514 (time 1776192398.03 total 561.60 last 64.60) - link 35108 (time 1776192408.04 total 571.60 last 59.37) - link 35762 (time 1776192418.04 total 581.61 last 65.36) - link 36447 (time 1776192428.05 total 591.61 last 68.47) - link 37179 (time 1776192438.05 total 601.61 last 73.20) - link 37945 (time 1776192448.06 total 611.62 last 76.53) - link 38689 (time 1776192458.06 total 621.63 last 74.35) - link 39527 (time 1776192468.07 total 631.64 last 83.70) - link 40000 (time 1776192474.85 total 638.42 last 69.79) - link 40782 (time 1776192484.86 total 648.43 last 78.13) - link 41521 (time 1776192494.88 total 658.45 last 73.75) - link 42210 (time 1776192504.89 total 668.46 last 68.83) - link 42857 (time 1776192514.90 total 678.47 last 64.66) - link 43533 (time 1776192524.90 total 688.47 last 67.59) - link 44236 (time 1776192534.90 total 698.47 last 70.30) - link 45016 (time 1776192544.90 total 708.47 last 77.97) - link 45599 (time 1776192554.91 total 718.47 last 58.29) - link 46046 (time 1776192564.93 total 728.50 last 44.60) - link 46659 (time 1776192574.93 total 738.50 last 61.27) - link 47465 (time 1776192584.94 total 748.51 last 80.52) - link 48127 (time 1776192594.96 total 758.53 last 66.10) - link 48924 (time 1776192604.96 total 768.53 last 79.68) - link 49609 (time 1776192614.96 total 778.53 last 68.50) - link 50000 (time 1776192619.32 total 782.89 last 89.71) - link 50764 (time 1776192629.33 total 792.89 last 76.36) - link 51429 (time 1776192639.36 total 802.93 last 66.25) - link 52070 (time 1776192649.37 total 812.93 last 64.08) - link 52761 (time 1776192659.38 total 822.95 last 69.01) - link 53393 (time 1776192669.38 total 832.95 last 63.19) - link 53997 (time 1776192679.41 total 842.98 last 60.23) - link 54515 (time 1776192689.42 total 852.98 last 51.76) - link 55128 (time 1776192699.42 total 862.99 last 61.29) - link 55906 (time 1776192709.42 total 872.99 last 77.77) - link 56492 (time 1776192719.43 total 882.99 last 58.58) - link 57038 (time 1776192729.44 total 893.01 last 54.50) - link 57591 (time 1776192739.46 total 903.03 last 55.19) - link 58208 (time 1776192749.48 total 913.05 last 61.59) - link 58823 (time 1776192759.49 total 923.05 last 61.48) - link 59420 (time 1776192769.49 total 933.06 last 59.67) total: 60000 link in 941.39 seconds: 63.74 ops/second - link 630 (time 1776191845.37 total 10.00 last 62.99) - link 1355 (time 1776191855.39 total 20.02 last 72.34) - link 1999 (time 1776191865.39 total 30.02 last 64.39) - link 2618 (time 1776191875.40 total 40.03 last 61.89) - link 3114 (time 1776191885.41 total 50.04 last 49.55) - link 3725 (time 1776191895.41 total 60.04 last 61.05) - link 4399 (time 1776191905.42 total 70.05 last 67.38) - link 4894 (time 1776191915.43 total 80.06 last 49.43) - link 5397 (time 1776191925.44 total 90.07 last 50.26) - link 5976 (time 1776191935.46 total 100.09 last 57.77) - link 6585 (time 1776191945.47 total 110.10 last 60.89) - link 7186 (time 1776191955.47 total 120.10 last 60.07) - link 7789 (time 1776191965.48 total 130.11 last 60.27) - link 8382 (time 1776191975.50 total 140.13 last 59.15) - link 9050 (time 1776191985.50 total 150.13 last 66.77) - link 9647 (time 1776191995.51 total 160.14 last 59.69) - link 10000 (time 1776192001.52 total 166.15 last 58.72) - link 10636 (time 1776192011.53 total 176.16 last 63.51) - link 11307 (time 1776192021.54 total 186.17 last 67.04) - link 11901 (time 1776192031.55 total 196.18 last 59.35) - link 12431 (time 1776192041.57 total 206.20 last 52.90) - link 13019 (time 1776192051.58 total 216.21 last 58.74) - link 13656 (time 1776192061.59 total 226.22 last 63.62) - link 14338 (time 1776192071.60 total 236.23 last 68.09) - link 15022 (time 1776192081.61 total 246.24 last 68.36) - link 15719 (time 1776192091.62 total 256.25 last 69.61) - link 16425 (time 1776192101.65 total 266.28 last 70.40) - link 17128 (time 1776192111.66 total 276.29 last 70.27) - link 17908 (time 1776192121.66 total 286.29 last 77.94) - link 18817 (time 1776192131.69 total 296.32 last 90.66) - link 19614 (time 1776192141.69 total 306.32 last 79.67) - link 20000 (time 1776192147.28 total 311.91 last 69.12) - link 20721 (time 1776192157.28 total 321.91 last 72.05) - link 21429 (time 1776192167.29 total 331.92 last 70.77) - link 22087 (time 1776192177.29 total 341.92 last 65.78) - link 22768 (time 1776192187.30 total 351.93 last 68.04) - link 23393 (time 1776192197.31 total 361.94 last 62.43) - link 23994 (time 1776192207.31 total 371.94 last 60.09) - link 24714 (time 1776192217.31 total 381.94 last 72.00) - link 25398 (time 1776192227.32 total 391.95 last 68.36) - link 25949 (time 1776192237.33 total 401.97 last 55.01) - link 26278 (time 1776192247.36 total 411.99 last 32.83) - link 26656 (time 1776192257.38 total 422.01 last 37.71) - link 27066 (time 1776192267.40 total 432.03 last 40.92) - link 27633 (time 1776192277.40 total 442.03 last 56.69) - link 28184 (time 1776192287.41 total 452.04 last 55.07) - link 28798 (time 1776192297.42 total 462.05 last 61.32) - link 29312 (time 1776192307.43 total 472.06 last 51.37) - link 29790 (time 1776192317.44 total 482.07 last 47.75) - link 30000 (time 1776192321.33 total 485.96 last 53.98) - link 30526 (time 1776192331.34 total 495.97 last 52.56) - link 31039 (time 1776192341.34 total 505.97 last 51.29) - link 31664 (time 1776192351.36 total 515.99 last 62.39) - link 32265 (time 1776192361.36 total 525.99 last 60.09) - link 32924 (time 1776192371.37 total 536.00 last 65.82) - link 33440 (time 1776192381.37 total 546.00 last 51.59) - link 33985 (time 1776192391.39 total 556.02 last 54.41) - link 34596 (time 1776192401.41 total 566.04 last 60.94) - link 35210 (time 1776192411.42 total 576.05 last 61.38) - link 35860 (time 1776192421.42 total 586.05 last 64.98) - link 36596 (time 1776192431.42 total 596.05 last 73.58) - link 37382 (time 1776192441.42 total 606.05 last 78.58) - link 38137 (time 1776192451.43 total 616.06 last 75.49) - link 38924 (time 1776192461.43 total 626.06 last 78.69) - link 39685 (time 1776192471.44 total 636.07 last 76.03) - link 40000 (time 1776192475.55 total 640.18 last 76.57) - link 40775 (time 1776192485.56 total 650.19 last 77.41) - link 41490 (time 1776192495.56 total 660.19 last 71.48) - link 42218 (time 1776192505.57 total 670.20 last 72.79) - link 42856 (time 1776192515.57 total 680.20 last 63.75) - link 43536 (time 1776192525.58 total 690.21 last 67.97) - link 44240 (time 1776192535.58 total 700.21 last 70.36) - link 45026 (time 1776192545.59 total 710.22 last 78.52) - link 45583 (time 1776192555.60 total 720.23 last 55.69) - link 46020 (time 1776192565.63 total 730.26 last 43.54) - link 46706 (time 1776192575.64 total 740.27 last 68.58) - link 47483 (time 1776192585.65 total 750.28 last 77.55) - link 48145 (time 1776192595.67 total 760.30 last 66.09) - link 48893 (time 1776192605.68 total 770.31 last 74.72) - link 49588 (time 1776192615.69 total 780.32 last 69.44) - link 50000 (time 1776192620.22 total 784.85 last 91.04) - link 50761 (time 1776192630.22 total 794.85 last 76.08) - link 51442 (time 1776192640.22 total 804.85 last 68.08) - link 52060 (time 1776192650.23 total 814.86 last 61.77) - link 52740 (time 1776192660.23 total 824.86 last 68.00) - link 53385 (time 1776192670.23 total 834.86 last 64.50) - link 53992 (time 1776192680.23 total 844.86 last 60.66) - link 54494 (time 1776192690.25 total 854.88 last 50.13) - link 55128 (time 1776192700.26 total 864.89 last 63.35) - link 55835 (time 1776192710.29 total 874.92 last 70.45) - link 56415 (time 1776192720.30 total 884.93 last 57.96) - link 56946 (time 1776192730.31 total 894.94 last 53.04) - link 57511 (time 1776192740.32 total 904.95 last 56.45) - link 58138 (time 1776192750.32 total 914.95 last 62.68) - link 58724 (time 1776192760.32 total 924.95 last 58.58) - link 59340 (time 1776192770.32 total 934.95 last 61.60) total: 60000 link in 944.52 seconds: 63.52 ops/second estimate 39686s left after 120000 files / 945s umount lustre on /mnt/lustre2..... Stopping client oleg224-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg224-server: LNET ready to unload oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. ETA 39686s after 120000 files / 945s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg224-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg224-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 29], used inodes 276/960000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 193 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 194 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 195 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 196 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 197 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 200 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 201 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 202 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 203 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 204 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 205 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 206 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 207 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 208 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 209 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 210 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 211 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 212 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 213 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 214 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 215 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 216 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 217 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 218 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 219 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 220 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 222 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 223 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 224 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 225 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 226 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 227 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 228 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 229 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 230 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 231 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 232 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 233 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 234 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 235 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 236 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 237 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 238 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 239 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 240 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 241 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 242 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 243 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 244 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 245 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 246 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 247 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 248 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 249 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 250 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 251 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 252 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 253 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 254 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 255 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 256 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 257 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 258 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 259 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 260 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 261 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 262 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 263 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 264 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 265 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 266 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 267 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 268 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 269 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 270 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 271 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 277 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2660: increase inode 278 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 278 badness 1 to 3 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] group 20 finished [Thread 0] group 21 finished [Thread 0] group 22 finished [Thread 0] group 23 finished [Thread 0] group 24 finished [Thread 0] group 25 finished [Thread 0] group 26 finished [Thread 0] group 27 finished [Thread 0] group 28 finished [Thread 0] group 29 finished [Thread 0] Pass 1: Memory used: 352k/1212k (126k/227k), time: 0.11/ 0.01/ 0.01 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 17.82MB/s [Thread 0] Scanned group range [0, 29], used inodes 276/282 Pass 2: Checking directory structure Pass 2: Memory used: 352k/304k (82k/271k), time: 1.40/ 0.30/ 0.09 Pass 2: I/O read: 49MB, write: 0MB, rate: 35.08MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 352k/304k (82k/271k), time: 1.64/ 0.34/ 0.10 Pass 3: Memory used: 352k/304k (81k/272k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 352k/0k (79k/274k), time: 0.09/ 0.03/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 11.51MB/s Pass 5: Checking group summary information Pass 5: Memory used: 492k/0k (78k/415k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 134.35MB/s 276 inodes used (0.03%, out of 960000) 3 non-contiguous files (1.1%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 278394 blocks used (46.40%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 492k/0k (80k/413k), time: 1.74/ 0.37/ 0.11 I/O read: 49MB, write: 0MB, rate: 28.17MB/s pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 oleg224-server: oleg224-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (1192s) Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f) oleg224-server: oleg224-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4119 sec ========== 14:56:33 (1776192993)