-----============= acceptance-small: conf-sanity ============----- Fri Dec 5 07:13:43 EST 2025 oleg617-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg617-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg617-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg617-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg617-server oleg617-server: oleg617-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing load_modules_local oleg617-server: Loading modules from /home/green/git/lustre-release/lustre oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg617-server: detected 4 online CPUs by sysfs oleg617-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg617-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg617-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg617-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg617-server: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg617-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg617-server: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg617-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server == conf-sanity test 45: long unlink handling in ptlrpcd == 07:16:37 (1764936997) start mds service on oleg617-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg617-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg617-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg617-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg617-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg617-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg617-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing unload_modules_local oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg617-server: LNET ready to unload oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 PASS 45 (242s) == conf-sanity test 69: replace an OST with the same index ========================================================== 07:20:39 (1764937239) start mds service on oleg617-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing load_modules_local oleg617-server: Loading modules from /home/green/git/lustre-release/lustre oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg617-server: detected 4 online CPUs by sysfs oleg617-server: Force libcfs to create 2 CPU partitions oleg617-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg617-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg617-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg617-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 750 (time 1764937320.65 total 10.00 last 74.97) - open/close 1635 (time 1764937330.66 total 20.02 last 88.39) - open/close 2457 (time 1764937340.67 total 30.02 last 82.14) - open/close 3318 (time 1764937350.67 total 40.03 last 86.05) - open/close 4134 (time 1764937360.68 total 50.04 last 81.52) - open/close 4953 (time 1764937370.69 total 60.04 last 81.86) - open/close 5744 (time 1764937380.69 total 70.05 last 79.06) - open/close 6589 (time 1764937390.69 total 80.05 last 84.49) - open/close 7370 (time 1764937400.70 total 90.05 last 78.07) - open/close 8152 (time 1764937410.71 total 100.07 last 78.11) - open/close 8891 (time 1764937420.72 total 110.07 last 73.85) - open/close 9757 (time 1764937430.72 total 120.08 last 86.55) total: 10000 open/close in 122.85 seconds: 81.40 ops/second - unlinked 0 (time 1764937436 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 833 (time 1764937517.71 total 10.00 last 83.30) - open/close 1652 (time 1764937527.71 total 20.00 last 81.90) - open/close 2528 (time 1764937537.71 total 30.00 last 87.60) - open/close 3334 (time 1764937547.71 total 40.00 last 80.59) - open/close 4068 (time 1764937557.72 total 50.01 last 73.36) - open/close 4806 (time 1764937567.72 total 60.01 last 73.75) - open/close 5563 (time 1764937577.72 total 70.02 last 75.70) - open/close 6438 (time 1764937587.73 total 80.02 last 87.43) - open/close 7307 (time 1764937597.74 total 90.03 last 86.87) - open/close 8144 (time 1764937607.75 total 100.04 last 83.62) - open/close 8817 (time 1764937617.75 total 110.04 last 67.27) - open/close 9475 (time 1764937627.76 total 120.05 last 65.72) total: 10000 open/close in 125.27 seconds: 79.83 ops/second - unlinked 0 (time 1764937636 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 827 (time 1764937722.36 total 10.00 last 82.67) - open/close 1699 (time 1764937732.36 total 20.01 last 87.17) - open/close 2504 (time 1764937742.37 total 30.01 last 80.46) - open/close 3357 (time 1764937752.37 total 40.02 last 85.25) - open/close 4194 (time 1764937762.38 total 50.03 last 83.61) - open/close 5068 (time 1764937772.39 total 60.04 last 87.32) - open/close 5913 (time 1764937782.40 total 70.05 last 84.45) - open/close 6797 (time 1764937792.40 total 80.05 last 88.37) - open/close 7616 (time 1764937802.41 total 90.05 last 81.85) - open/close 8517 (time 1764937812.43 total 100.08 last 89.89) - open/close 9301 (time 1764937822.44 total 110.09 last 78.34) - open/close 9992 (time 1764937832.46 total 120.10 last 68.97) total: 10000 open/close in 120.24 seconds: 83.16 ops/second - unlinked 0 (time 1764937836 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 868 (time 1764937921.36 total 10.01 last 86.75) - open/close 1723 (time 1764937931.37 total 20.01 last 85.48) - open/close 2572 (time 1764937941.38 total 30.02 last 84.80) - open/close 3310 (time 1764937951.38 total 40.02 last 73.80) - open/close 4168 (time 1764937961.38 total 50.02 last 85.79) - open/close 4938 (time 1764937971.40 total 60.04 last 76.83) - open/close 5786 (time 1764937981.41 total 70.05 last 84.73) - open/close 6621 (time 1764937991.43 total 80.07 last 83.32) - open/close 7485 (time 1764938001.44 total 90.08 last 86.31) - open/close 8361 (time 1764938011.44 total 100.08 last 87.59) - open/close 9171 (time 1764938021.44 total 110.09 last 80.99) - open/close 9929 (time 1764938031.45 total 120.09 last 75.78) total: 10000 open/close in 121.08 seconds: 82.59 ops/second - unlinked 0 (time 1764938036 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 838 (time 1764938122.48 total 10.01 last 83.72) - open/close 1649 (time 1764938132.49 total 20.03 last 80.96) - open/close 2388 (time 1764938142.50 total 30.03 last 73.88) - open/close 3184 (time 1764938152.50 total 40.04 last 79.55) - open/close 3830 (time 1764938162.51 total 50.04 last 64.56) - open/close 4626 (time 1764938172.51 total 60.05 last 79.56) - open/close 5455 (time 1764938182.53 total 70.06 last 82.80) - open/close 6283 (time 1764938192.54 total 80.07 last 82.70) - open/close 7034 (time 1764938202.55 total 90.08 last 75.04) - open/close 7595 (time 1764938212.55 total 100.09 last 56.06) - open/close 8178 (time 1764938222.56 total 110.10 last 58.22) - open/close 8811 (time 1764938232.59 total 120.12 last 63.17) - open/close 9566 (time 1764938242.59 total 130.12 last 75.48) total: 10000 open/close in 135.55 seconds: 73.77 ops/second - unlinked 0 (time 1764938252 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 904 (time 1764938340.68 total 10.00 last 90.37) - open/close 1650 (time 1764938350.69 total 20.01 last 74.54) - open/close 2479 (time 1764938360.69 total 30.01 last 82.89) - open/close 3264 (time 1764938370.69 total 40.02 last 78.45) - open/close 4181 (time 1764938380.70 total 50.03 last 91.62) - open/close 4956 (time 1764938390.70 total 60.03 last 77.49) - open/close 5817 (time 1764938400.70 total 70.03 last 86.08) - open/close 6666 (time 1764938410.71 total 80.04 last 84.85) - open/close 7525 (time 1764938420.72 total 90.04 last 85.85) - open/close 8390 (time 1764938430.72 total 100.05 last 86.46) - open/close 9164 (time 1764938440.73 total 110.05 last 77.35) - open/close 9929 (time 1764938450.73 total 120.05 last 76.50) total: 10000 open/close in 120.91 seconds: 82.71 ops/second - unlinked 0 (time 1764938455 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 913 (time 1764938535.40 total 10.01 last 91.20) - open/close 1812 (time 1764938545.40 total 20.01 last 89.89) - open/close 2698 (time 1764938555.40 total 30.01 last 88.57) - open/close 3524 (time 1764938565.40 total 40.02 last 82.59) - open/close 4359 (time 1764938575.41 total 50.02 last 83.45) - open/close 5254 (time 1764938585.41 total 60.02 last 89.48) - open/close 6117 (time 1764938595.41 total 70.03 last 86.29) - open/close 7007 (time 1764938605.41 total 80.03 last 88.97) - open/close 7721 (time 1764938615.42 total 90.03 last 71.37) - open/close 8624 (time 1764938625.43 total 100.04 last 90.23) - open/close 9554 (time 1764938635.43 total 110.04 last 93.00) total: 10000 open/close in 115.04 seconds: 86.93 ops/second - unlinked 0 (time 1764938644 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 911 (time 1764938728.09 total 10.00 last 91.10) - open/close 1775 (time 1764938738.11 total 20.02 last 86.21) - open/close 2666 (time 1764938748.12 total 30.03 last 89.03) - open/close 3518 (time 1764938758.12 total 40.03 last 85.19) - open/close 4396 (time 1764938768.13 total 50.04 last 87.73) - open/close 5231 (time 1764938778.13 total 60.04 last 83.49) - open/close 5924 (time 1764938788.14 total 70.04 last 69.28) - open/close 6646 (time 1764938798.16 total 80.07 last 72.00) - open/close 7343 (time 1764938808.17 total 90.08 last 69.65) - open/close 8154 (time 1764938818.18 total 100.09 last 81.03) - open/close 8950 (time 1764938828.19 total 110.09 last 79.56) - open/close 9825 (time 1764938838.19 total 120.09 last 87.49) total: 10000 open/close in 122.14 seconds: 81.87 ops/second - unlinked 0 (time 1764938843 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 794 (time 1764938933.23 total 10.02 last 79.28) - open/close 1618 (time 1764938943.23 total 20.02 last 82.37) - open/close 2387 (time 1764938953.24 total 30.02 last 76.89) - open/close 3275 (time 1764938963.25 total 40.03 last 88.71) - open/close 4178 (time 1764938973.25 total 50.03 last 90.27) - open/close 5049 (time 1764938983.27 total 60.05 last 86.93) - open/close 5874 (time 1764938993.27 total 70.06 last 82.48) - open/close 6836 (time 1764939003.27 total 80.06 last 96.17) - open/close 7707 (time 1764939013.28 total 90.06 last 87.05) - open/close 8613 (time 1764939023.29 total 100.07 last 90.52) - open/close 9362 (time 1764939033.30 total 110.09 last 74.78) total: 10000 open/close in 118.02 seconds: 84.73 ops/second - unlinked 0 (time 1764939044 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 825 (time 1764939127.27 total 10.01 last 82.42) - open/close 1556 (time 1764939137.27 total 20.01 last 73.08) - open/close 2421 (time 1764939147.27 total 30.02 last 86.47) - open/close 3188 (time 1764939157.29 total 40.03 last 76.60) - open/close 3935 (time 1764939167.29 total 50.03 last 74.69) - open/close 4747 (time 1764939177.30 total 60.04 last 81.13) - open/close 5512 (time 1764939187.30 total 70.04 last 76.48) - open/close 6343 (time 1764939197.31 total 80.05 last 83.03) - open/close 7186 (time 1764939207.32 total 90.06 last 84.22) - open/close 8065 (time 1764939217.33 total 100.07 last 87.79) - open/close 8940 (time 1764939227.34 total 110.08 last 87.45) - open/close 9586 (time 1764939237.34 total 120.08 last 64.57) total: 10000 open/close in 125.29 seconds: 79.81 ops/second - unlinked 0 (time 1764939246 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg617-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.206.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg617-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Started lustre-OST0000 oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg617-server: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg617-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg617-server: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg617-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg617-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg617-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing unload_modules_local oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg617-server: LNET ready to unload oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 PASS 69 (2170s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 07:56:50 (1764939410) oleg617-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present oleg617-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present umount lustre on /mnt/lustre..... stop ost1 service on oleg617-server stop mds service on oleg617-server stop mds service on oleg617-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing unload_modules_local oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O ea_inode,large_dir -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing load_modules_local oleg617-server: Loading modules from /home/green/git/lustre-release/lustre oleg617-server: detected 4 online CPUs by sysfs oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg617-server: Force libcfs to create 2 CPU partitions oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg617-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg617-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg617-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.206.117@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg617-server: oleg617-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre Starting client oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre Started clients oleg617-client.virtnet: 192.168.206.117@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) mount lustre on /mnt/lustre2..... Starting client: oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre2 Starting client oleg617-client.virtnet: -o user_xattr,flock oleg617-server@tcp:/lustre /mnt/lustre2 Started clients oleg617-client.virtnet: 192.168.206.117@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1326500 1696 1206392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1252 126964 1% /mnt/lustre[OST:0] filesystem_summary: 142216 1252 126964 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 960000 272 959728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 268 49732 1% /mnt/lustre[OST:0] filesystem_summary: 50004 272 49732 1% /mnt/lustre creating 60000 hardlinks to oleg617-client.virtnet-0 creating 60000 hardlinks to oleg617-client.virtnet-1 waiting for PIDs 26993 27008 to complete - link 752 (time 1764939508.50 total 10.01 last 75.09) - link 1429 (time 1764939518.51 total 20.02 last 67.63) - link 2121 (time 1764939528.52 total 30.04 last 69.11) - link 2902 (time 1764939538.54 total 40.06 last 77.95) - link 3691 (time 1764939548.54 total 50.06 last 78.88) - link 4431 (time 1764939558.54 total 60.06 last 73.98) - link 5236 (time 1764939568.55 total 70.07 last 80.45) - link 5982 (time 1764939578.56 total 80.08 last 74.55) - link 6793 (time 1764939588.57 total 90.08 last 81.02) - link 7518 (time 1764939598.57 total 100.09 last 72.48) - link 8216 (time 1764939608.59 total 110.11 last 69.65) - link 8951 (time 1764939618.60 total 120.11 last 73.47) - link 9685 (time 1764939628.60 total 130.12 last 73.37) - link 10000 (time 1764939632.83 total 134.35 last 74.45) - link 10751 (time 1764939642.83 total 144.35 last 75.09) - link 11560 (time 1764939652.84 total 154.36 last 80.81) - link 12512 (time 1764939662.84 total 164.36 last 95.20) - link 13385 (time 1764939672.85 total 174.37 last 87.24) - link 14352 (time 1764939682.86 total 184.38 last 96.61) - link 15355 (time 1764939692.88 total 194.40 last 100.12) - link 16386 (time 1764939702.89 total 204.41 last 102.97) - link 17455 (time 1764939712.90 total 214.42 last 106.82) - link 18567 (time 1764939722.90 total 224.42 last 111.15) - link 19748 (time 1764939732.91 total 234.43 last 118.06) - link 20000 (time 1764939735.05 total 236.56 last 117.83) - link 21110 (time 1764939745.06 total 246.57 last 110.88) - link 22103 (time 1764939755.07 total 256.58 last 99.20) - link 23038 (time 1764939765.07 total 266.59 last 93.49) - link 23943 (time 1764939775.07 total 276.59 last 90.48) - link 24874 (time 1764939785.08 total 286.60 last 93.04) - link 25810 (time 1764939795.08 total 296.60 last 93.57) - link 26917 (time 1764939805.09 total 306.61 last 110.61) - link 27973 (time 1764939815.09 total 316.61 last 105.53) - link 29106 (time 1764939825.10 total 326.62 last 113.21) - link 30000 (time 1764939833.79 total 335.31 last 102.87) - link 31025 (time 1764939843.80 total 345.32 last 102.42) - link 31967 (time 1764939853.80 total 355.32 last 94.19) - link 32923 (time 1764939863.81 total 365.32 last 95.57) - link 33962 (time 1764939873.81 total 375.33 last 103.84) - link 35186 (time 1764939883.82 total 385.34 last 122.32) - link 36270 (time 1764939893.82 total 395.34 last 108.36) - link 37258 (time 1764939903.82 total 405.34 last 98.80) - link 38293 (time 1764939913.83 total 415.34 last 103.44) - link 39324 (time 1764939923.83 total 425.35 last 103.04) - link 40000 (time 1764939930.10 total 431.62 last 107.81) - link 41177 (time 1764939940.11 total 441.63 last 117.58) - link 42098 (time 1764939950.12 total 451.64 last 92.05) - link 42934 (time 1764939960.12 total 461.64 last 83.55) - link 43815 (time 1764939970.13 total 471.65 last 88.05) - link 44754 (time 1764939980.13 total 481.65 last 93.85) - link 45838 (time 1764939990.13 total 491.65 last 108.40) - link 46834 (time 1764940000.14 total 501.65 last 99.58) - link 47985 (time 1764940010.14 total 511.66 last 115.01) - link 49033 (time 1764940020.15 total 521.67 last 104.75) - link 50000 (time 1764940029.49 total 531.01 last 103.49) - link 51147 (time 1764940039.49 total 541.01 last 114.69) - link 52347 (time 1764940049.50 total 551.01 last 119.97) - link 53437 (time 1764940059.50 total 561.02 last 108.94) - link 54539 (time 1764940069.51 total 571.03 last 110.12) - link 55666 (time 1764940079.51 total 581.03 last 112.63) - link 57024 (time 1764940089.52 total 591.04 last 135.69) - link 58235 (time 1764940099.53 total 601.05 last 120.99) - link 59548 (time 1764940109.54 total 611.06 last 131.21) total: 60000 link in 615.57 seconds: 97.47 ops/second - link 765 (time 1764939509.49 total 10.01 last 76.42) - link 1432 (time 1764939519.50 total 20.02 last 66.66) - link 2153 (time 1764939529.50 total 30.02 last 72.09) - link 2935 (time 1764939539.50 total 40.02 last 78.17) - link 3721 (time 1764939549.50 total 50.02 last 78.59) - link 4464 (time 1764939559.51 total 60.03 last 74.24) - link 5267 (time 1764939569.51 total 70.03 last 80.29) - link 6017 (time 1764939579.52 total 80.04 last 74.97) - link 6804 (time 1764939589.53 total 90.04 last 78.63) - link 7507 (time 1764939599.53 total 100.05 last 70.26) - link 8209 (time 1764939609.55 total 110.07 last 70.09) - link 8925 (time 1764939619.55 total 120.07 last 71.58) - link 9655 (time 1764939629.56 total 130.08 last 72.92) - link 10000 (time 1764939634.27 total 134.78 last 73.36) - link 10761 (time 1764939644.27 total 144.79 last 76.06) - link 11584 (time 1764939654.28 total 154.79 last 82.27) - link 12505 (time 1764939664.28 total 164.80 last 92.08) - link 13375 (time 1764939674.29 total 174.81 last 86.90) - link 14375 (time 1764939684.29 total 184.81 last 100.00) - link 15348 (time 1764939694.30 total 194.82 last 97.18) - link 16396 (time 1764939704.31 total 204.83 last 104.70) - link 17494 (time 1764939714.34 total 214.85 last 109.52) - link 18574 (time 1764939724.36 total 224.87 last 107.79) - link 19760 (time 1764939734.36 total 234.87 last 118.60) - link 20000 (time 1764939736.77 total 237.28 last 99.59) - link 21116 (time 1764939746.77 total 247.29 last 111.53) - link 22077 (time 1764939756.78 total 257.30 last 96.04) - link 23019 (time 1764939766.78 total 267.30 last 94.19) - link 23899 (time 1764939776.79 total 277.30 last 87.93) - link 24837 (time 1764939786.79 total 287.31 last 93.78) - link 25792 (time 1764939796.79 total 297.31 last 95.48) - link 26889 (time 1764939806.80 total 307.31 last 109.64) - link 27965 (time 1764939816.81 total 317.32 last 107.49) - link 29049 (time 1764939826.82 total 327.34 last 108.23) - link 30000 (time 1764939836.08 total 336.60 last 102.73) - link 31020 (time 1764939846.09 total 346.61 last 101.89) - link 31973 (time 1764939856.10 total 356.61 last 95.25) - link 32957 (time 1764939866.10 total 366.62 last 98.37) - link 34034 (time 1764939876.10 total 376.62 last 107.64) - link 35252 (time 1764939886.11 total 386.62 last 121.79) - link 36311 (time 1764939896.11 total 396.63 last 105.86) - link 37259 (time 1764939906.11 total 406.63 last 94.79) - link 38319 (time 1764939916.11 total 416.63 last 105.96) - link 39365 (time 1764939926.13 total 426.64 last 104.46) - link 40000 (time 1764939931.58 total 432.10 last 116.49) - link 41144 (time 1764939941.58 total 442.10 last 114.33) - link 42065 (time 1764939951.59 total 452.10 last 92.08) - link 42907 (time 1764939961.59 total 462.11 last 84.15) - link 43802 (time 1764939971.59 total 472.11 last 89.49) - link 44776 (time 1764939981.60 total 482.12 last 97.35) - link 45872 (time 1764939991.61 total 492.12 last 109.50) - link 46881 (time 1764940001.62 total 502.13 last 100.81) - link 48006 (time 1764940011.62 total 512.14 last 112.48) - link 49061 (time 1764940021.63 total 522.14 last 105.41) - link 50000 (time 1764940030.71 total 531.23 last 103.38) - link 51145 (time 1764940040.71 total 541.23 last 114.45) - link 52321 (time 1764940050.72 total 551.23 last 117.56) - link 53404 (time 1764940060.72 total 561.24 last 108.24) - link 54501 (time 1764940070.73 total 571.25 last 109.57) - link 55665 (time 1764940080.74 total 581.26 last 116.31) - link 57032 (time 1764940090.74 total 591.26 last 136.65) - link 58233 (time 1764940100.75 total 601.27 last 120.05) - link 59517 (time 1764940110.75 total 611.27 last 128.32) total: 60000 link in 616.13 seconds: 97.38 ops/second estimate 25939s left after 120000 files / 617s umount lustre on /mnt/lustre2..... Stopping client oleg617-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg617-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg617-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg617-server stop mds service on oleg617-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg617-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing unload_modules_local oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg617-server: LNET ready to unload oleg617-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. ETA 25939s after 120000 files / 617s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg617-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg617-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 29], used inodes 276/960000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 193 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 194 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 195 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 196 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 197 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 200 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 201 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 202 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 203 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 204 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 205 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 206 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 207 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 208 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 209 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 210 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 211 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 212 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 213 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 214 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 215 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 216 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 217 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 218 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 219 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 220 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 221 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 223 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 224 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 225 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 226 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 227 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 228 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 229 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 230 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 231 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 232 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 233 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 234 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 235 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 236 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 237 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 238 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 239 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 240 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 241 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 242 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 243 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 244 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 245 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 246 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 247 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 248 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 249 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 250 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 251 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 252 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 253 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 254 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 255 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 256 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 257 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 258 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 259 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 260 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 261 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 262 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 263 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 264 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 265 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 266 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 267 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 268 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 269 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 270 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 271 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 277 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2660: increase inode 278 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 278 badness 1 to 3 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] group 20 finished [Thread 0] group 21 finished [Thread 0] group 22 finished [Thread 0] group 23 finished [Thread 0] group 24 finished [Thread 0] group 25 finished [Thread 0] group 26 finished [Thread 0] group 27 finished [Thread 0] group 28 finished [Thread 0] group 29 finished [Thread 0] Pass 1: Memory used: 352k/1212k (126k/227k), time: 0.03/ 0.01/ 0.01 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 79.97MB/s [Thread 0] Scanned group range [0, 29], used inodes 276/281 Pass 2: Checking directory structure Pass 2: Memory used: 352k/304k (82k/271k), time: 0.54/ 0.30/ 0.09 Pass 2: I/O read: 49MB, write: 0MB, rate: 91.31MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 352k/304k (82k/271k), time: 0.62/ 0.33/ 0.09 Pass 3: Memory used: 352k/304k (80k/273k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 352k/0k (79k/274k), time: 0.03/ 0.02/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 35.28MB/s Pass 5: Checking group summary information Pass 5: Memory used: 492k/0k (77k/416k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 266.88MB/s 276 inodes used (0.03%, out of 960000) 4 non-contiguous files (1.4%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 278381 blocks used (46.40%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 492k/0k (80k/413k), time: 0.66/ 0.36/ 0.10 I/O read: 49MB, write: 0MB, rate: 74.69MB/s pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-client: ssh exited with exit code 2 pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 Stopping clients: oleg617-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg617-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg617-client: oleg617-server: ssh exited with exit code 2 oleg617-server: oleg617-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing load_modules_local oleg617-server: Loading modules from /home/green/git/lustre-release/lustre oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg617-server: detected 4 online CPUs by sysfs oleg617-server: Force libcfs to create 2 CPU partitions oleg617-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg617-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (796s) Stopping clients: oleg617-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg617-client.virtnet /mnt/lustre2 (opts:-f) oleg617-server: oleg617-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg617-server' oleg617-server: oleg617-server.virtnet: executing load_modules_local oleg617-server: Loading modules from /home/green/git/lustre-release/lustre oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg617-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg617-server: detected 4 online CPUs by sysfs oleg617-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3412 sec ========== 08:10:36 (1764940236)