-----============= acceptance-small: conf-sanity ============----- Tue Apr 14 15:56:03 EDT 2026 oleg356-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg356-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg356-server oleg356-server: oleg356-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg356-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:59:09 (1776196749) start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg356-server: LNET ready to unload oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 PASS 45 (253s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:03:22 (1776197002) start mds service on oleg356-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 871 (time 1776197102.46 total 10.00 last 87.06) - open/close 1666 (time 1776197112.47 total 20.02 last 79.40) - open/close 2524 (time 1776197122.47 total 30.02 last 85.77) - open/close 3333 (time 1776197132.47 total 40.02 last 80.88) - open/close 3990 (time 1776197142.48 total 50.03 last 65.67) - open/close 4704 (time 1776197152.50 total 60.05 last 71.24) - open/close 5519 (time 1776197162.52 total 70.06 last 81.39) - open/close 6405 (time 1776197172.52 total 80.07 last 88.52) - open/close 7343 (time 1776197182.53 total 90.08 last 93.73) - open/close 8151 (time 1776197192.55 total 100.09 last 80.69) - open/close 8933 (time 1776197202.56 total 110.10 last 78.12) - open/close 9788 (time 1776197212.56 total 120.11 last 85.48) total: 10000 open/close in 122.53 seconds: 81.61 ops/second - unlinked 0 (time 1776197218 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 830 (time 1776197307.24 total 10.00 last 82.99) - open/close 1680 (time 1776197317.25 total 20.01 last 84.94) - open/close 2451 (time 1776197327.25 total 30.01 last 77.08) - open/close 3217 (time 1776197337.26 total 40.02 last 76.53) - open/close 4044 (time 1776197347.27 total 50.03 last 82.62) - open/close 4819 (time 1776197357.27 total 60.03 last 77.46) - open/close 5645 (time 1776197367.29 total 70.05 last 82.49) - open/close 6451 (time 1776197377.29 total 80.05 last 80.55) - open/close 7187 (time 1776197387.30 total 90.06 last 73.54) - open/close 7862 (time 1776197397.30 total 100.06 last 67.49) - open/close 8624 (time 1776197407.31 total 110.07 last 76.17) - open/close 9206 (time 1776197417.32 total 120.08 last 58.14) - open/close 9913 (time 1776197427.32 total 130.08 last 70.66) total: 10000 open/close in 131.25 seconds: 76.19 ops/second - unlinked 0 (time 1776197433 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 779 (time 1776197523.08 total 10.02 last 77.71) - open/close 1592 (time 1776197533.08 total 20.03 last 81.25) - open/close 2349 (time 1776197543.09 total 30.03 last 75.68) - open/close 3191 (time 1776197553.09 total 40.04 last 84.16) - open/close 3981 (time 1776197563.09 total 50.04 last 78.97) - open/close 4788 (time 1776197573.10 total 60.04 last 80.69) - open/close 5613 (time 1776197583.10 total 70.05 last 82.47) - open/close 6458 (time 1776197593.11 total 80.05 last 84.45) - open/close 7346 (time 1776197603.11 total 90.06 last 88.77) - open/close 8160 (time 1776197613.11 total 100.06 last 81.39) - open/close 9020 (time 1776197623.12 total 110.07 last 85.92) - open/close 9805 (time 1776197633.12 total 120.07 last 78.48) total: 10000 open/close in 122.37 seconds: 81.72 ops/second - unlinked 0 (time 1776197638 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 800 (time 1776197729.28 total 10.00 last 79.98) - open/close 1578 (time 1776197739.28 total 20.01 last 77.77) - open/close 2399 (time 1776197749.28 total 30.01 last 82.07) - open/close 3119 (time 1776197759.29 total 40.01 last 71.97) - open/close 3846 (time 1776197769.29 total 50.02 last 72.68) - open/close 4540 (time 1776197779.30 total 60.02 last 69.35) - open/close 5361 (time 1776197789.30 total 70.03 last 82.07) - open/close 6146 (time 1776197799.32 total 80.04 last 78.37) - open/close 6987 (time 1776197809.34 total 90.07 last 83.93) - open/close 7847 (time 1776197819.35 total 100.07 last 85.92) - open/close 8684 (time 1776197829.35 total 110.08 last 83.67) - open/close 9473 (time 1776197839.37 total 120.09 last 78.79) total: 10000 open/close in 126.11 seconds: 79.30 ops/second - unlinked 0 (time 1776197849 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 829 (time 1776197934.31 total 10.02 last 82.76) - open/close 1719 (time 1776197944.31 total 20.02 last 88.99) - open/close 2547 (time 1776197954.32 total 30.03 last 82.73) - open/close 3413 (time 1776197964.33 total 40.03 last 86.58) - open/close 4260 (time 1776197974.33 total 50.03 last 84.68) - open/close 5145 (time 1776197984.33 total 60.04 last 88.46) - open/close 5947 (time 1776197994.34 total 70.05 last 80.12) - open/close 6799 (time 1776198004.35 total 80.06 last 85.12) - open/close 7641 (time 1776198014.36 total 90.06 last 84.14) - open/close 8449 (time 1776198024.37 total 100.07 last 80.72) - open/close 9218 (time 1776198034.37 total 110.08 last 76.87) - open/close 9993 (time 1776198044.39 total 120.10 last 77.36) total: 10000 open/close in 120.22 seconds: 83.18 ops/second - unlinked 0 (time 1776198048 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 837 (time 1776198140.20 total 10.01 last 83.65) - open/close 1675 (time 1776198150.20 total 20.01 last 83.76) - open/close 2472 (time 1776198160.20 total 30.01 last 79.68) - open/close 3248 (time 1776198170.22 total 40.03 last 77.50) - open/close 4043 (time 1776198180.23 total 50.04 last 79.41) - open/close 4827 (time 1776198190.23 total 60.04 last 78.38) - open/close 5514 (time 1776198200.23 total 70.04 last 68.69) - open/close 6082 (time 1776198210.28 total 80.09 last 56.53) - open/close 6603 (time 1776198220.29 total 90.10 last 52.07) - open/close 7336 (time 1776198230.29 total 100.10 last 73.24) - open/close 8109 (time 1776198240.30 total 110.11 last 77.22) - open/close 8882 (time 1776198250.30 total 120.11 last 77.30) - open/close 9709 (time 1776198260.31 total 130.12 last 82.68) total: 10000 open/close in 133.41 seconds: 74.96 ops/second - unlinked 0 (time 1776198267 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 748 (time 1776198363.94 total 10.01 last 74.74) - open/close 1514 (time 1776198373.95 total 20.01 last 76.57) - open/close 2263 (time 1776198383.95 total 30.01 last 74.88) - open/close 3071 (time 1776198393.95 total 40.01 last 80.80) - open/close 3875 (time 1776198403.96 total 50.02 last 80.33) - open/close 4654 (time 1776198413.96 total 60.03 last 77.86) - open/close 5457 (time 1776198423.96 total 70.03 last 80.30) - open/close 6290 (time 1776198433.98 total 80.05 last 83.14) - open/close 7170 (time 1776198443.99 total 90.05 last 87.94) - open/close 7998 (time 1776198454.01 total 100.07 last 82.66) - open/close 8807 (time 1776198464.01 total 110.08 last 80.84) - open/close 9590 (time 1776198474.02 total 120.08 last 78.26) total: 10000 open/close in 124.86 seconds: 80.09 ops/second - unlinked 0 (time 1776198482 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 900 (time 1776198576.05 total 10.01 last 89.87) - open/close 1723 (time 1776198586.05 total 20.02 last 82.29) - open/close 2564 (time 1776198596.05 total 30.02 last 84.09) - open/close 3437 (time 1776198606.06 total 40.02 last 87.24) - open/close 4250 (time 1776198616.07 total 50.04 last 81.19) - open/close 5135 (time 1776198626.08 total 60.04 last 88.47) - open/close 5995 (time 1776198636.09 total 70.06 last 85.84) - open/close 6875 (time 1776198646.11 total 80.07 last 87.90) - open/close 7631 (time 1776198656.11 total 90.07 last 75.58) - open/close 8347 (time 1776198666.11 total 100.08 last 71.58) - open/close 9120 (time 1776198676.11 total 110.08 last 77.27) - open/close 9992 (time 1776198686.12 total 120.09 last 87.13) total: 10000 open/close in 120.18 seconds: 83.21 ops/second - unlinked 0 (time 1776198690 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 838 (time 1776198774.99 total 10.00 last 83.79) - open/close 1674 (time 1776198785.00 total 20.02 last 83.49) - open/close 2415 (time 1776198795.01 total 30.02 last 74.06) - open/close 3239 (time 1776198805.02 total 40.04 last 82.27) - open/close 4103 (time 1776198815.03 total 50.04 last 86.35) - open/close 4959 (time 1776198825.03 total 60.05 last 85.57) - open/close 5833 (time 1776198835.04 total 70.06 last 87.30) - open/close 6695 (time 1776198845.05 total 80.06 last 86.13) - open/close 7604 (time 1776198855.06 total 90.07 last 90.84) - open/close 8425 (time 1776198865.06 total 100.08 last 82.06) - open/close 9361 (time 1776198875.07 total 110.08 last 93.53) total: 10000 open/close in 117.94 seconds: 84.79 ops/second - unlinked 0 (time 1776198885 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 755 (time 1776198977.38 total 10.00 last 75.47) - open/close 1528 (time 1776198987.38 total 20.01 last 77.26) - open/close 2367 (time 1776198997.41 total 30.03 last 83.70) - open/close 3135 (time 1776199007.43 total 40.05 last 76.65) - open/close 3933 (time 1776199017.44 total 50.06 last 79.71) - open/close 4746 (time 1776199027.45 total 60.08 last 81.20) - open/close 5589 (time 1776199037.46 total 70.08 last 84.24) - open/close 6367 (time 1776199047.46 total 80.08 last 77.79) - open/close 7160 (time 1776199057.47 total 90.10 last 79.20) - open/close 7804 (time 1776199067.47 total 100.10 last 64.39) - open/close 8469 (time 1776199077.48 total 110.10 last 66.48) - open/close 9119 (time 1776199087.48 total 120.11 last 64.97) - open/close 9809 (time 1776199097.48 total 130.11 last 69.00) total: 10000 open/close in 133.28 seconds: 75.03 ops/second - unlinked 0 (time 1776199104 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg356-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg356-server: LNET ready to unload oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 PASS 69 (2273s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:41:16 (1776199276) oleg356-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present oleg356-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present umount lustre on /mnt/lustre..... stop ost1 service on oleg356-server stop mds service on oleg356-server stop mds service on oleg356-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O ea_inode,large_dir -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg356-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre Starting client oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre Started clients oleg356-client.virtnet: 192.168.203.156@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) mount lustre on /mnt/lustre2..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre2 Starting client oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre2 Started clients oleg356-client.virtnet: 192.168.203.156@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1326500 1696 1206392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1252 126964 1% /mnt/lustre[OST:0] filesystem_summary: 142216 1252 126964 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 960000 272 959728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 268 49732 1% /mnt/lustre[OST:0] filesystem_summary: 50004 272 49732 1% /mnt/lustre creating 60000 hardlinks to oleg356-client.virtnet-0 creating 60000 hardlinks to oleg356-client.virtnet-1 waiting for PIDs 27038 27052 to complete - link 520 (time 1776199367.66 total 10.01 last 51.96) - link 1093 (time 1776199377.66 total 20.01 last 57.29) - link 1711 (time 1776199387.67 total 30.02 last 61.75) - link 2394 (time 1776199397.67 total 40.02 last 68.28) - link 3147 (time 1776199407.68 total 50.03 last 75.25) - link 3949 (time 1776199417.68 total 60.03 last 80.20) - link 4726 (time 1776199427.68 total 70.03 last 77.70) - link 5465 (time 1776199437.70 total 80.05 last 73.73) - link 6035 (time 1776199447.70 total 90.05 last 56.99) - link 6773 (time 1776199457.71 total 100.06 last 73.75) - link 7540 (time 1776199467.72 total 110.07 last 76.64) - link 8368 (time 1776199477.72 total 120.07 last 82.77) - link 9200 (time 1776199487.73 total 130.08 last 83.14) - link 9965 (time 1776199497.74 total 140.08 last 76.45) - link 10701 (time 1776199507.75 total 150.10 last 73.49) - link 11481 (time 1776199517.75 total 160.10 last 77.99) - link 12240 (time 1776199527.75 total 170.10 last 75.89) - link 13005 (time 1776199537.77 total 180.12 last 76.36) - link 13762 (time 1776199547.78 total 190.13 last 75.65) - link 14582 (time 1776199557.79 total 200.14 last 81.92) - link 15387 (time 1776199567.80 total 210.15 last 80.43) - link 16114 (time 1776199577.81 total 220.16 last 72.62) - link 16788 (time 1776199587.83 total 230.18 last 67.24) - link 17557 (time 1776199597.84 total 240.19 last 76.81) - link 18283 (time 1776199607.85 total 250.20 last 72.57) - link 19088 (time 1776199617.86 total 260.21 last 80.41) - link 19859 (time 1776199627.88 total 270.23 last 76.93) - link 20687 (time 1776199637.92 total 280.27 last 82.44) - link 21392 (time 1776199647.95 total 290.30 last 70.30) - link 22059 (time 1776199657.96 total 300.31 last 66.66) - link 22603 (time 1776199667.99 total 310.34 last 54.22) - link 23247 (time 1776199678.01 total 320.36 last 64.24) - link 23886 (time 1776199688.02 total 330.37 last 63.84) - link 24639 (time 1776199698.04 total 340.39 last 75.15) - link 25500 (time 1776199708.07 total 350.42 last 85.86) - link 26272 (time 1776199718.13 total 360.48 last 76.78) - link 26950 (time 1776199728.13 total 370.48 last 67.74) - link 27607 (time 1776199738.17 total 380.52 last 65.46) - link 28193 (time 1776199748.18 total 390.53 last 58.53) - link 28878 (time 1776199758.19 total 400.54 last 68.42) - link 29600 (time 1776199768.22 total 410.57 last 72.05) - link 30000 (time 1776199773.36 total 415.71 last 77.72) - link 30721 (time 1776199783.37 total 425.72 last 72.02) - link 31471 (time 1776199793.38 total 435.73 last 74.97) - link 32144 (time 1776199803.38 total 445.73 last 67.30) - link 32914 (time 1776199813.39 total 455.74 last 76.94) - link 33659 (time 1776199823.39 total 465.74 last 74.47) - link 34437 (time 1776199833.40 total 475.75 last 77.70) - link 35108 (time 1776199843.40 total 485.75 last 67.09) - link 35846 (time 1776199853.42 total 495.77 last 73.72) - link 36581 (time 1776199863.43 total 505.78 last 73.42) - link 37266 (time 1776199873.44 total 515.79 last 68.41) - link 38116 (time 1776199883.45 total 525.80 last 84.89) - link 38883 (time 1776199893.46 total 535.81 last 76.66) - link 39489 (time 1776199903.46 total 545.81 last 60.55) - link 40000 (time 1776199912.25 total 554.60 last 58.13) - link 40656 (time 1776199922.26 total 564.61 last 65.58) - link 41379 (time 1776199932.26 total 574.61 last 72.29) - link 42128 (time 1776199942.26 total 584.61 last 74.89) - link 42933 (time 1776199952.27 total 594.62 last 80.42) - link 43777 (time 1776199962.28 total 604.63 last 84.30) - link 44533 (time 1776199972.29 total 614.64 last 75.57) - link 45252 (time 1776199982.30 total 624.65 last 71.81) - link 46017 (time 1776199992.30 total 634.65 last 76.50) - link 46780 (time 1776200002.30 total 644.65 last 76.27) - link 47546 (time 1776200012.31 total 654.66 last 76.55) - link 48307 (time 1776200022.32 total 664.67 last 76.04) - link 49041 (time 1776200032.32 total 674.67 last 73.39) - link 49722 (time 1776200042.32 total 684.67 last 68.07) - link 50000 (time 1776200045.73 total 688.08 last 81.55) - link 50693 (time 1776200055.74 total 698.09 last 69.26) - link 51358 (time 1776200065.75 total 708.10 last 66.42) - link 52078 (time 1776200075.75 total 718.10 last 71.99) - link 52752 (time 1776200085.77 total 728.12 last 67.30) - link 53430 (time 1776200095.77 total 738.12 last 67.79) - link 54077 (time 1776200105.78 total 748.13 last 64.63) - link 54633 (time 1776200115.78 total 758.13 last 55.58) - link 55203 (time 1776200125.79 total 768.14 last 56.96) - link 55761 (time 1776200135.79 total 778.14 last 55.78) - link 56327 (time 1776200145.80 total 788.15 last 56.58) - link 56874 (time 1776200155.81 total 798.16 last 54.63) - link 57503 (time 1776200165.81 total 808.16 last 62.88) - link 58187 (time 1776200175.82 total 818.17 last 68.36) - link 58852 (time 1776200185.83 total 828.18 last 66.43) - link 59519 (time 1776200195.84 total 838.19 last 66.62) total: 60000 link in 845.80 seconds: 70.94 ops/second - link 514 (time 1776199369.18 total 10.01 last 51.36) - link 1055 (time 1776199379.19 total 20.02 last 54.03) - link 1651 (time 1776199389.20 total 30.03 last 59.55) - link 2341 (time 1776199399.20 total 40.04 last 68.95) - link 3085 (time 1776199409.21 total 50.04 last 74.36) - link 3875 (time 1776199419.22 total 60.05 last 78.94) - link 4683 (time 1776199429.22 total 70.05 last 80.78) - link 5374 (time 1776199439.24 total 80.07 last 68.96) - link 5989 (time 1776199449.25 total 90.08 last 61.44) - link 6716 (time 1776199459.25 total 100.09 last 72.68) - link 7502 (time 1776199469.26 total 110.09 last 78.59) - link 8344 (time 1776199479.26 total 120.09 last 84.19) - link 9146 (time 1776199489.27 total 130.10 last 80.12) - link 9905 (time 1776199499.27 total 140.10 last 75.89) - link 10628 (time 1776199509.29 total 150.12 last 72.13) - link 11432 (time 1776199519.30 total 160.13 last 80.34) - link 12189 (time 1776199529.31 total 170.14 last 75.65) - link 12948 (time 1776199539.32 total 180.15 last 75.79) - link 13717 (time 1776199549.32 total 190.16 last 76.88) - link 14533 (time 1776199559.33 total 200.16 last 81.55) - link 15298 (time 1776199569.34 total 210.17 last 76.44) - link 16053 (time 1776199579.35 total 220.18 last 75.43) - link 16741 (time 1776199589.36 total 230.20 last 68.68) - link 17457 (time 1776199599.39 total 240.22 last 71.41) - link 18243 (time 1776199609.40 total 250.23 last 78.53) - link 19040 (time 1776199619.41 total 260.24 last 79.62) - link 19811 (time 1776199629.41 total 270.24 last 77.08) - link 20000 (time 1776199631.74 total 272.58 last 81.09) - link 20772 (time 1776199641.75 total 282.58 last 77.16) - link 21476 (time 1776199651.76 total 292.60 last 70.29) - link 22099 (time 1776199661.78 total 302.61 last 62.19) - link 22666 (time 1776199671.82 total 312.65 last 56.49) - link 23302 (time 1776199681.83 total 322.66 last 63.54) - link 24021 (time 1776199691.83 total 332.67 last 71.85) - link 24783 (time 1776199701.84 total 342.67 last 76.19) - link 25620 (time 1776199711.84 total 352.67 last 83.67) - link 26369 (time 1776199721.84 total 362.67 last 74.90) - link 27032 (time 1776199731.84 total 372.68 last 66.27) - link 27652 (time 1776199741.86 total 382.70 last 61.89) - link 28324 (time 1776199751.87 total 392.70 last 67.16) - link 29002 (time 1776199761.87 total 402.70 last 67.79) - link 29731 (time 1776199771.87 total 412.71 last 72.87) - link 30000 (time 1776199775.81 total 416.65 last 68.25) - link 30748 (time 1776199785.82 total 426.65 last 74.77) - link 31485 (time 1776199795.83 total 436.66 last 73.63) - link 32200 (time 1776199805.83 total 446.66 last 71.49) - link 32994 (time 1776199815.84 total 456.67 last 79.33) - link 33740 (time 1776199825.85 total 466.68 last 74.51) - link 34515 (time 1776199835.85 total 476.69 last 77.48) - link 35191 (time 1776199845.86 total 486.69 last 67.56) - link 35916 (time 1776199855.88 total 496.71 last 72.39) - link 36657 (time 1776199865.88 total 506.71 last 74.07) - link 37374 (time 1776199875.89 total 516.72 last 71.65) - link 38204 (time 1776199885.89 total 526.72 last 82.97) - link 38955 (time 1776199895.90 total 536.74 last 74.99) - link 39552 (time 1776199905.91 total 546.74 last 59.66) - link 40000 (time 1776199913.84 total 554.68 last 56.48) - link 40626 (time 1776199923.86 total 564.69 last 62.51) - link 41326 (time 1776199933.86 total 574.69 last 69.99) - link 42078 (time 1776199943.86 total 584.70 last 75.17) - link 42939 (time 1776199953.87 total 594.70 last 86.07) - link 43734 (time 1776199963.87 total 604.70 last 79.47) - link 44484 (time 1776199973.88 total 614.71 last 74.93) - link 45190 (time 1776199983.90 total 624.73 last 70.48) - link 45942 (time 1776199993.91 total 634.75 last 75.08) - link 46722 (time 1776200003.94 total 644.77 last 77.79) - link 47474 (time 1776200013.94 total 654.77 last 75.19) - link 48247 (time 1776200023.95 total 664.78 last 77.23) - link 48941 (time 1776200033.95 total 674.79 last 69.38) - link 49664 (time 1776200043.96 total 684.79 last 72.27) - link 50000 (time 1776200048.32 total 689.16 last 76.95) - link 50636 (time 1776200058.33 total 699.16 last 63.56) - link 51361 (time 1776200068.34 total 709.17 last 72.43) - link 52056 (time 1776200078.35 total 719.18 last 69.46) - link 52716 (time 1776200088.35 total 729.18 last 65.99) - link 53411 (time 1776200098.35 total 739.18 last 69.48) - link 53999 (time 1776200108.37 total 749.21 last 58.68) - link 54557 (time 1776200118.39 total 759.22 last 55.72) - link 55122 (time 1776200128.40 total 769.23 last 56.43) - link 55688 (time 1776200138.40 total 779.23 last 56.59) - link 56239 (time 1776200148.42 total 789.25 last 54.98) - link 56822 (time 1776200158.43 total 799.26 last 58.28) - link 57481 (time 1776200168.44 total 809.27 last 65.84) - link 58153 (time 1776200178.46 total 819.29 last 67.03) - link 58870 (time 1776200188.47 total 829.30 last 71.65) - link 59509 (time 1776200198.48 total 839.31 last 63.81) total: 60000 link in 846.50 seconds: 70.88 ops/second estimate 35657s left after 120000 files / 849s umount lustre on /mnt/lustre2..... Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg356-server: LNET ready to unload oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. ETA 35657s after 120000 files / 849s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg356-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg356-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 29], used inodes 276/960000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 193 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 194 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 195 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 196 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 197 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 200 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 201 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 202 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 203 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 204 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 205 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 206 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 207 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 208 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 209 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 210 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 211 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 212 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 213 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 214 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 215 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 216 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 217 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 218 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 219 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 220 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 221 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 223 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 224 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 225 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 226 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 227 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 228 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 229 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 230 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 231 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 232 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 233 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 234 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 235 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 236 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 237 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 238 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 239 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 240 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 241 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 242 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 243 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 244 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 245 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 246 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 247 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 248 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 249 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 250 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 251 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 252 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 253 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 254 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 255 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 256 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 257 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 258 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 259 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 260 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 261 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 262 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 263 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 264 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 265 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 266 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 267 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 268 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 269 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 270 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 271 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2660: increase inode 276 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 276 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2660: increase inode 277 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 277 badness 1 to 3 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] group 20 finished [Thread 0] group 21 finished [Thread 0] group 22 finished [Thread 0] group 23 finished [Thread 0] group 24 finished [Thread 0] group 25 finished [Thread 0] group 26 finished [Thread 0] group 27 finished [Thread 0] group 28 finished [Thread 0] group 29 finished [Thread 0] Pass 1: Memory used: 352k/1212k (126k/227k), time: 0.04/ 0.01/ 0.01 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 49.27MB/s [Thread 0] Scanned group range [0, 29], used inodes 276/283 Pass 2: Checking directory structure Pass 2: Memory used: 352k/304k (82k/271k), time: 0.93/ 0.35/ 0.08 Pass 2: I/O read: 49MB, write: 0MB, rate: 52.55MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 352k/304k (82k/271k), time: 1.04/ 0.40/ 0.09 Pass 3: Memory used: 352k/304k (80k/272k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 352k/0k (79k/274k), time: 0.07/ 0.02/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 14.10MB/s Pass 5: Checking group summary information Pass 5: Memory used: 492k/0k (77k/416k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 137.65MB/s 276 inodes used (0.03%, out of 960000) 3 non-contiguous files (1.1%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 278393 blocks used (46.40%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 492k/0k (80k/413k), time: 1.13/ 0.42/ 0.09 I/O read: 49MB, write: 0MB, rate: 43.44MB/s pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: oleg356-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (1072s) Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) oleg356-server: oleg356-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3857 sec ========== 17:00:22 (1776200422)