-----============= acceptance-small: conf-sanity ============----- Tue Apr 14 14:30:07 EDT 2026 oleg308-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg308-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg308-server oleg308-server: oleg308-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg308-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server == conf-sanity test 45: long unlink handling in ptlrpcd == 14:33:11 (1776191591) start mds service on oleg308-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg308-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg308-server: LNET ready to unload oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 PASS 45 (248s) == conf-sanity test 69: replace an OST with the same index ========================================================== 14:37:20 (1776191840) start mds service on oleg308-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 916 (time 1776191937.56 total 10.01 last 91.49) - open/close 1823 (time 1776191947.56 total 20.02 last 90.64) - open/close 2689 (time 1776191957.57 total 30.02 last 86.55) - open/close 3576 (time 1776191967.58 total 40.03 last 88.65) - open/close 4400 (time 1776191977.58 total 50.03 last 82.38) - open/close 5276 (time 1776191987.58 total 60.04 last 87.58) - open/close 6154 (time 1776191997.58 total 70.04 last 87.80) - open/close 6893 (time 1776192007.59 total 80.04 last 73.87) - open/close 7639 (time 1776192017.61 total 90.07 last 74.39) - open/close 8517 (time 1776192027.62 total 100.07 last 87.78) - open/close 9400 (time 1776192037.62 total 110.07 last 88.27) total: 10000 open/close in 119.20 seconds: 83.89 ops/second - unlinked 0 (time 1776192050 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 687 (time 1776192137.98 total 10.01 last 68.66) - open/close 1509 (time 1776192147.98 total 20.01 last 82.19) - open/close 2367 (time 1776192157.99 total 30.02 last 85.71) - open/close 3217 (time 1776192168.00 total 40.03 last 84.93) - open/close 4072 (time 1776192178.00 total 50.03 last 85.48) - open/close 4795 (time 1776192188.00 total 60.03 last 72.28) - open/close 5553 (time 1776192198.01 total 70.04 last 75.76) - open/close 6462 (time 1776192208.01 total 80.04 last 90.86) - open/close 7335 (time 1776192218.02 total 90.05 last 87.25) - open/close 8143 (time 1776192228.02 total 100.05 last 80.77) - open/close 8932 (time 1776192238.03 total 110.06 last 78.84) - open/close 9749 (time 1776192248.03 total 120.06 last 81.67) total: 10000 open/close in 123.58 seconds: 80.92 ops/second - unlinked 0 (time 1776192255 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 706 (time 1776192341.15 total 10.01 last 70.52) - open/close 1207 (time 1776192351.15 total 20.02 last 50.08) - open/close 1943 (time 1776192361.16 total 30.02 last 73.57) - open/close 2686 (time 1776192371.16 total 40.02 last 74.29) - open/close 3455 (time 1776192381.17 total 50.03 last 76.86) - open/close 4308 (time 1776192391.17 total 60.03 last 85.24) - open/close 5161 (time 1776192401.19 total 70.05 last 85.17) - open/close 5861 (time 1776192411.20 total 80.06 last 69.92) - open/close 6510 (time 1776192421.20 total 90.06 last 64.90) - open/close 7269 (time 1776192431.21 total 100.07 last 75.83) - open/close 8123 (time 1776192441.21 total 110.08 last 85.35) - open/close 8985 (time 1776192451.22 total 120.08 last 86.12) - open/close 9838 (time 1776192461.23 total 130.09 last 85.27) total: 10000 open/close in 132.01 seconds: 75.75 ops/second - unlinked 0 (time 1776192466 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 542 (time 1776192552.27 total 10.01 last 54.14) - open/close 1268 (time 1776192562.28 total 20.02 last 72.57) - open/close 2057 (time 1776192572.28 total 30.02 last 78.85) - open/close 2779 (time 1776192582.29 total 40.03 last 72.15) - open/close 3584 (time 1776192592.29 total 50.03 last 80.48) - open/close 4389 (time 1776192602.30 total 60.04 last 80.45) - open/close 5121 (time 1776192612.31 total 70.05 last 73.13) - open/close 5633 (time 1776192622.33 total 80.07 last 51.10) - open/close 6364 (time 1776192632.33 total 90.07 last 73.06) - open/close 7068 (time 1776192642.33 total 100.07 last 70.40) - open/close 7802 (time 1776192652.35 total 110.09 last 73.29) - open/close 8579 (time 1776192662.35 total 120.09 last 77.65) - open/close 9349 (time 1776192672.36 total 130.10 last 76.95) total: 10000 open/close in 139.56 seconds: 71.65 ops/second - unlinked 0 (time 1776192686 ; total 1 ; last 1) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 822 (time 1776192776.07 total 10.02 last 82.06) - open/close 1687 (time 1776192786.08 total 20.03 last 86.43) - open/close 2559 (time 1776192796.08 total 30.03 last 87.14) - open/close 3440 (time 1776192806.10 total 40.05 last 87.96) - open/close 4317 (time 1776192816.10 total 50.05 last 87.69) - open/close 5113 (time 1776192826.11 total 60.06 last 79.54) - open/close 5828 (time 1776192836.11 total 70.06 last 71.48) - open/close 6565 (time 1776192846.13 total 80.07 last 73.59) - open/close 7285 (time 1776192856.13 total 90.08 last 71.97) - open/close 8029 (time 1776192866.14 total 100.09 last 74.35) - open/close 8832 (time 1776192876.15 total 110.09 last 80.23) - open/close 9662 (time 1776192886.15 total 120.10 last 82.98) total: 10000 open/close in 124.12 seconds: 80.57 ops/second - unlinked 0 (time 1776192893 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 778 (time 1776192982.51 total 10.01 last 77.70) - open/close 1579 (time 1776192992.51 total 20.02 last 80.07) - open/close 2266 (time 1776193002.53 total 30.03 last 68.59) - open/close 2942 (time 1776193012.54 total 40.05 last 67.51) - open/close 3636 (time 1776193022.56 total 50.06 last 69.31) - open/close 4288 (time 1776193032.57 total 60.07 last 65.13) - open/close 4920 (time 1776193042.57 total 70.08 last 63.15) - open/close 5676 (time 1776193052.58 total 80.08 last 75.59) - open/close 6479 (time 1776193062.58 total 90.08 last 80.29) - open/close 7316 (time 1776193072.58 total 100.09 last 83.63) - open/close 8126 (time 1776193082.59 total 110.09 last 80.96) - open/close 8952 (time 1776193092.60 total 120.11 last 82.49) - open/close 9663 (time 1776193102.60 total 130.11 last 71.09) total: 10000 open/close in 133.90 seconds: 74.68 ops/second - unlinked 0 (time 1776193110 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 682 (time 1776193206.67 total 10.01 last 68.13) - open/close 1496 (time 1776193216.67 total 20.01 last 81.37) - open/close 2357 (time 1776193226.69 total 30.03 last 85.96) - open/close 3194 (time 1776193236.71 total 40.05 last 83.55) - open/close 3936 (time 1776193246.72 total 50.06 last 74.13) - open/close 4765 (time 1776193256.72 total 60.06 last 82.85) - open/close 5482 (time 1776193266.73 total 70.07 last 71.67) - open/close 6120 (time 1776193276.73 total 80.07 last 63.77) - open/close 6759 (time 1776193286.74 total 90.08 last 63.87) - open/close 7501 (time 1776193296.74 total 100.08 last 74.13) - open/close 8262 (time 1776193306.77 total 110.11 last 75.93) - open/close 8763 (time 1776193316.77 total 120.11 last 50.07) - open/close 9654 (time 1776193326.78 total 130.12 last 89.01) total: 10000 open/close in 134.62 seconds: 74.28 ops/second - unlinked 0 (time 1776193335 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 810 (time 1776193423.46 total 10.01 last 80.93) - open/close 1460 (time 1776193433.47 total 20.01 last 64.97) - open/close 2189 (time 1776193443.49 total 30.03 last 72.77) - open/close 2964 (time 1776193453.49 total 40.03 last 77.48) - open/close 3622 (time 1776193463.50 total 50.04 last 65.73) - open/close 4421 (time 1776193473.51 total 60.05 last 79.86) - open/close 5232 (time 1776193483.51 total 70.05 last 81.08) - open/close 6098 (time 1776193493.51 total 80.06 last 86.54) - open/close 6899 (time 1776193503.52 total 90.06 last 80.06) - open/close 7726 (time 1776193513.52 total 100.07 last 82.67) - open/close 8557 (time 1776193523.52 total 110.07 last 83.09) - open/close 9393 (time 1776193533.53 total 120.07 last 83.56) total: 10000 open/close in 127.62 seconds: 78.36 ops/second - unlinked 0 (time 1776193545 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 840 (time 1776193629.79 total 10.00 last 83.97) - open/close 1705 (time 1776193639.80 total 20.01 last 86.48) - open/close 2510 (time 1776193649.80 total 30.01 last 80.45) - open/close 3240 (time 1776193659.81 total 40.02 last 72.94) - open/close 3948 (time 1776193669.81 total 50.02 last 70.79) - open/close 4817 (time 1776193679.82 total 60.03 last 86.83) - open/close 5556 (time 1776193689.84 total 70.05 last 73.77) - open/close 6369 (time 1776193699.85 total 80.06 last 81.21) - open/close 7108 (time 1776193709.85 total 90.06 last 73.89) - open/close 7616 (time 1776193719.86 total 100.07 last 50.74) - open/close 8461 (time 1776193729.87 total 110.08 last 84.47) - open/close 9384 (time 1776193739.87 total 120.08 last 92.23) total: 10000 open/close in 127.50 seconds: 78.43 ops/second - unlinked 0 (time 1776193751 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 778 (time 1776193835.01 total 10.01 last 77.75) - open/close 1661 (time 1776193845.01 total 20.01 last 88.25) - open/close 2512 (time 1776193855.03 total 30.03 last 84.97) - open/close 3298 (time 1776193865.04 total 40.04 last 78.49) - open/close 4131 (time 1776193875.05 total 50.05 last 83.24) - open/close 5010 (time 1776193885.05 total 60.05 last 87.89) - open/close 5852 (time 1776193895.06 total 70.05 last 84.18) - open/close 6683 (time 1776193905.06 total 80.06 last 83.07) - open/close 7592 (time 1776193915.07 total 90.06 last 90.83) - open/close 8483 (time 1776193925.07 total 100.07 last 89.06) - open/close 9377 (time 1776193935.08 total 110.07 last 89.35) total: 10000 open/close in 117.45 seconds: 85.14 ops/second - unlinked 0 (time 1776193945 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg308-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg308-server: LNET ready to unload oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 PASS 69 (2278s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 15:15:19 (1776194119) oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present oleg308-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present umount lustre on /mnt/lustre..... stop ost1 service on oleg308-server stop mds service on oleg308-server stop mds service on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O ea_inode,large_dir -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg308-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre Starting client oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre Started clients oleg308-client.virtnet: 192.168.203.108@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) mount lustre on /mnt/lustre2..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre2 Starting client oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre2 Started clients oleg308-client.virtnet: 192.168.203.108@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1326500 1696 1206392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1252 126964 1% /mnt/lustre[OST:0] filesystem_summary: 142216 1252 126964 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 960000 272 959728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 268 49732 1% /mnt/lustre[OST:0] filesystem_summary: 50004 272 49732 1% /mnt/lustre creating 60000 hardlinks to oleg308-client.virtnet-0 creating 60000 hardlinks to oleg308-client.virtnet-1 waiting for PIDs 27009 27023 to complete - link 716 (time 1776194217.57 total 10.01 last 71.55) - link 1493 (time 1776194227.58 total 20.01 last 77.64) - link 2302 (time 1776194237.58 total 30.02 last 80.88) - link 3106 (time 1776194247.58 total 40.02 last 80.36) - link 3970 (time 1776194257.59 total 50.03 last 86.30) - link 4752 (time 1776194267.61 total 60.05 last 78.11) - link 5512 (time 1776194277.61 total 70.05 last 75.96) - link 6256 (time 1776194287.61 total 80.05 last 74.39) - link 6939 (time 1776194297.63 total 90.07 last 68.20) - link 7579 (time 1776194307.64 total 100.08 last 63.94) - link 8152 (time 1776194317.64 total 110.08 last 57.26) - link 8813 (time 1776194327.68 total 120.12 last 65.88) - link 9523 (time 1776194337.68 total 130.12 last 70.95) - link 10000 (time 1776194344.23 total 136.67 last 72.86) - link 10756 (time 1776194354.25 total 146.69 last 75.46) - link 11378 (time 1776194364.26 total 156.70 last 62.15) - link 12141 (time 1776194374.26 total 166.70 last 76.24) - link 12846 (time 1776194384.27 total 176.71 last 70.48) - link 13497 (time 1776194394.27 total 186.71 last 65.08) - link 14115 (time 1776194404.27 total 196.71 last 61.78) - link 14774 (time 1776194414.28 total 206.72 last 65.87) - link 15592 (time 1776194424.29 total 216.72 last 81.76) - link 16232 (time 1776194434.29 total 226.73 last 63.97) - link 16932 (time 1776194444.30 total 236.74 last 69.95) - link 17649 (time 1776194454.30 total 246.74 last 71.69) - link 18420 (time 1776194464.31 total 256.75 last 77.00) - link 19066 (time 1776194474.32 total 266.76 last 64.57) - link 19774 (time 1776194484.33 total 276.77 last 70.73) - link 20000 (time 1776194487.46 total 279.90 last 72.05) - link 20804 (time 1776194497.47 total 289.90 last 80.38) - link 21576 (time 1776194507.48 total 299.92 last 77.07) - link 22395 (time 1776194517.49 total 309.93 last 81.82) - link 23229 (time 1776194527.50 total 319.94 last 83.32) - link 23987 (time 1776194537.51 total 329.95 last 75.77) - link 24747 (time 1776194547.52 total 339.96 last 75.92) - link 25548 (time 1776194557.53 total 349.97 last 80.04) - link 26341 (time 1776194567.53 total 359.97 last 79.24) - link 27122 (time 1776194577.54 total 369.98 last 78.05) - link 27916 (time 1776194587.54 total 379.98 last 79.39) - link 28626 (time 1776194597.55 total 389.99 last 70.95) - link 29349 (time 1776194607.55 total 399.99 last 72.29) - link 30000 (time 1776194616.36 total 408.80 last 73.84) - link 30734 (time 1776194626.38 total 418.82 last 73.28) - link 31559 (time 1776194636.38 total 428.82 last 82.49) - link 32370 (time 1776194646.39 total 438.83 last 81.01) - link 33056 (time 1776194656.39 total 448.83 last 68.59) - link 33777 (time 1776194666.39 total 458.83 last 72.10) - link 34538 (time 1776194676.40 total 468.84 last 76.03) - link 35182 (time 1776194686.41 total 478.85 last 64.34) - link 35771 (time 1776194696.43 total 488.87 last 58.81) - link 36365 (time 1776194706.44 total 498.88 last 59.35) - link 36991 (time 1776194716.45 total 508.89 last 62.54) - link 37638 (time 1776194726.46 total 518.90 last 64.62) - link 38283 (time 1776194736.46 total 528.90 last 64.48) - link 39034 (time 1776194746.47 total 538.91 last 75.08) - link 39794 (time 1776194756.47 total 548.91 last 76.00) - link 40000 (time 1776194758.79 total 551.23 last 88.75) - link 40721 (time 1776194768.80 total 561.24 last 72.03) - link 41328 (time 1776194778.81 total 571.25 last 60.64) - link 41951 (time 1776194788.81 total 581.25 last 62.29) - link 42610 (time 1776194798.81 total 591.25 last 65.89) - link 43353 (time 1776194808.83 total 601.27 last 74.15) - link 44082 (time 1776194818.85 total 611.29 last 72.76) - link 44881 (time 1776194828.86 total 621.30 last 79.78) - link 45657 (time 1776194838.88 total 631.32 last 77.49) - link 46430 (time 1776194848.88 total 641.32 last 77.26) - link 47145 (time 1776194858.89 total 651.33 last 71.44) - link 47873 (time 1776194868.89 total 661.33 last 72.79) - link 48501 (time 1776194878.90 total 671.34 last 62.76) - link 49112 (time 1776194888.91 total 681.35 last 61.04) - link 49662 (time 1776194898.91 total 691.35 last 55.00) - link 50000 (time 1776194903.95 total 696.39 last 67.02) - link 50623 (time 1776194913.96 total 706.40 last 62.27) - link 51305 (time 1776194923.96 total 716.40 last 68.20) - link 52021 (time 1776194933.96 total 726.40 last 71.55) - link 52778 (time 1776194943.97 total 736.41 last 75.63) - link 53488 (time 1776194953.97 total 746.41 last 70.98) - link 54294 (time 1776194963.99 total 756.43 last 80.48) - link 55092 (time 1776194973.99 total 766.43 last 79.79) - link 55834 (time 1776194983.99 total 776.43 last 74.19) - link 56412 (time 1776194994.00 total 786.44 last 57.77) - link 57036 (time 1776195004.01 total 796.45 last 62.34) - link 57754 (time 1776195014.03 total 806.47 last 71.66) - link 58377 (time 1776195024.03 total 816.47 last 62.28) - link 59036 (time 1776195034.03 total 826.47 last 65.90) - link 59773 (time 1776195044.04 total 836.48 last 73.63) total: 60000 link in 839.38 seconds: 71.48 ops/second - link 729 (time 1776194218.57 total 10.01 last 72.83) - link 1493 (time 1776194228.58 total 20.01 last 76.37) - link 2331 (time 1776194238.59 total 30.02 last 83.71) - link 3118 (time 1776194248.59 total 40.03 last 78.69) - link 3996 (time 1776194258.59 total 50.03 last 87.76) - link 4749 (time 1776194268.60 total 60.04 last 75.26) - link 5543 (time 1776194278.61 total 70.04 last 79.35) - link 6280 (time 1776194288.61 total 80.04 last 73.70) - link 6942 (time 1776194298.61 total 90.04 last 66.20) - link 7563 (time 1776194308.62 total 100.06 last 62.02) - link 8128 (time 1776194318.63 total 110.07 last 56.42) - link 8799 (time 1776194328.64 total 120.07 last 67.07) - link 9526 (time 1776194338.64 total 130.08 last 72.64) - link 10000 (time 1776194345.33 total 136.77 last 70.89) - link 10709 (time 1776194355.34 total 146.77 last 70.85) - link 11327 (time 1776194365.35 total 156.79 last 61.72) - link 12080 (time 1776194375.36 total 166.79 last 75.24) - link 12826 (time 1776194385.37 total 176.81 last 74.52) - link 13418 (time 1776194395.37 total 186.81 last 59.19) - link 14053 (time 1776194405.38 total 196.82 last 63.45) - link 14723 (time 1776194415.40 total 206.84 last 66.87) - link 15532 (time 1776194425.41 total 216.84 last 80.85) - link 16163 (time 1776194435.41 total 226.85 last 63.08) - link 16871 (time 1776194445.41 total 236.85 last 70.77) - link 17607 (time 1776194455.42 total 246.86 last 73.56) - link 18380 (time 1776194465.43 total 256.87 last 77.22) - link 19025 (time 1776194475.44 total 266.88 last 64.44) - link 19732 (time 1776194485.45 total 276.88 last 70.65) - link 20000 (time 1776194489.00 total 280.43 last 75.48) - link 20811 (time 1776194499.01 total 290.45 last 80.97) - link 21591 (time 1776194509.02 total 300.45 last 77.96) - link 22411 (time 1776194519.03 total 310.47 last 81.88) - link 23268 (time 1776194529.05 total 320.48 last 85.58) - link 23998 (time 1776194539.05 total 330.48 last 72.99) - link 24804 (time 1776194549.05 total 340.48 last 80.60) - link 25596 (time 1776194559.05 total 350.49 last 79.15) - link 26373 (time 1776194569.06 total 360.49 last 77.67) - link 27160 (time 1776194579.06 total 370.50 last 78.66) - link 27952 (time 1776194589.07 total 380.51 last 79.11) - link 28675 (time 1776194599.08 total 390.52 last 72.26) - link 29412 (time 1776194609.08 total 400.52 last 73.66) - link 30000 (time 1776194617.14 total 408.58 last 72.99) - link 30744 (time 1776194627.14 total 418.58 last 74.39) - link 31602 (time 1776194637.15 total 428.59 last 85.72) - link 32407 (time 1776194647.15 total 438.59 last 80.50) - link 33073 (time 1776194657.17 total 448.60 last 66.49) - link 33790 (time 1776194667.18 total 458.62 last 71.62) - link 34543 (time 1776194677.21 total 468.65 last 75.05) - link 35183 (time 1776194687.21 total 478.65 last 63.99) - link 35772 (time 1776194697.23 total 488.66 last 58.83) - link 36319 (time 1776194707.23 total 498.67 last 54.67) - link 36968 (time 1776194717.24 total 508.67 last 64.86) - link 37600 (time 1776194727.25 total 518.69 last 63.13) - link 38232 (time 1776194737.27 total 528.71 last 63.07) - link 38997 (time 1776194747.27 total 538.71 last 76.46) - link 39746 (time 1776194757.28 total 548.71 last 74.88) - link 40000 (time 1776194760.60 total 552.03 last 76.54) - link 40673 (time 1776194770.62 total 562.05 last 67.17) - link 41315 (time 1776194780.62 total 572.06 last 64.18) - link 41946 (time 1776194790.63 total 582.07 last 63.03) - link 42591 (time 1776194800.64 total 592.07 last 64.46) - link 43344 (time 1776194810.66 total 602.09 last 75.16) - link 44059 (time 1776194820.67 total 612.11 last 71.38) - link 44864 (time 1776194830.68 total 622.12 last 80.44) - link 45614 (time 1776194840.68 total 632.12 last 74.97) - link 46374 (time 1776194850.69 total 642.13 last 75.93) - link 47094 (time 1776194860.70 total 652.13 last 71.97) - link 47837 (time 1776194870.70 total 662.13 last 74.29) - link 48444 (time 1776194880.70 total 672.14 last 60.68) - link 49058 (time 1776194890.72 total 682.16 last 61.28) - link 49601 (time 1776194900.74 total 692.17 last 54.21) - link 50000 (time 1776194907.05 total 698.49 last 63.18) - link 50649 (time 1776194917.05 total 708.49 last 64.90) - link 51324 (time 1776194927.06 total 718.50 last 67.47) - link 52018 (time 1776194937.06 total 728.50 last 69.36) - link 52756 (time 1776194947.07 total 738.51 last 73.75) - link 53464 (time 1776194957.08 total 748.52 last 70.73) - link 54274 (time 1776194967.09 total 758.53 last 80.91) - link 55053 (time 1776194977.11 total 768.55 last 77.76) - link 55747 (time 1776194987.13 total 778.56 last 69.28) - link 56356 (time 1776194997.14 total 788.57 last 60.85) - link 56982 (time 1776195007.14 total 798.58 last 62.56) - link 57673 (time 1776195017.17 total 808.60 last 68.94) - link 58253 (time 1776195027.17 total 818.60 last 57.99) - link 58947 (time 1776195037.18 total 828.62 last 69.33) - link 59713 (time 1776195047.18 total 838.62 last 76.58) total: 60000 link in 841.65 seconds: 71.29 ops/second estimate 35387s left after 120000 files / 843s umount lustre on /mnt/lustre2..... Stopping client oleg308-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg308-server: LNET ready to unload oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. ETA 35387s after 120000 files / 843s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg308-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg308-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 29], used inodes 276/960000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 193 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 194 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 195 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 196 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 197 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 200 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 201 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 202 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 203 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 204 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 205 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 206 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 207 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 208 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 209 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 210 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 211 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 212 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 213 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 214 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 215 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 216 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 217 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 218 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 219 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 220 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 221 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 223 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 224 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 225 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 226 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 227 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 228 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 229 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 230 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 231 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 232 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 233 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 234 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 235 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 236 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 237 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 238 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 239 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 240 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 241 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 242 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 243 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 244 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 245 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 246 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 247 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 248 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 249 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 250 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 251 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 252 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 253 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 254 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 255 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 256 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 257 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 258 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 259 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 260 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 261 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 262 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 263 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 264 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 265 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 266 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 267 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 268 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 269 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 270 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 271 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2660: increase inode 277 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 277 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 278 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] group 20 finished [Thread 0] group 21 finished [Thread 0] group 22 finished [Thread 0] group 23 finished [Thread 0] group 24 finished [Thread 0] group 25 finished [Thread 0] group 26 finished [Thread 0] group 27 finished [Thread 0] group 28 finished [Thread 0] group 29 finished [Thread 0] Pass 1: Memory used: 352k/1212k (126k/227k), time: 0.10/ 0.01/ 0.00 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 20.09MB/s [Thread 0] Scanned group range [0, 29], used inodes 276/283 Pass 2: Checking directory structure Pass 2: Memory used: 352k/304k (82k/271k), time: 1.65/ 0.31/ 0.08 Pass 2: I/O read: 49MB, write: 0MB, rate: 29.70MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 352k/304k (82k/271k), time: 1.93/ 0.34/ 0.09 Pass 3: Memory used: 352k/304k (80k/273k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 352k/0k (79k/274k), time: 0.09/ 0.02/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 10.79MB/s Pass 5: Checking group summary information Pass 5: Memory used: 492k/0k (77k/416k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 76.17MB/s 276 inodes used (0.03%, out of 960000) 4 non-contiguous files (1.4%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 278399 blocks used (46.40%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 492k/0k (80k/413k), time: 2.06/ 0.37/ 0.09 I/O read: 49MB, write: 0MB, rate: 23.81MB/s pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 oleg308-server: oleg308-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (1067s) Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) oleg308-server: oleg308-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3831 sec ========== 15:33:59 (1776195239)