-----============= acceptance-small: conf-sanity ============----- Wed Apr 15 11:15:13 EDT 2026 oleg307-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg307-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg307-server oleg307-server: oleg307-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg307-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg307-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg307-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg307-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server == conf-sanity test 45: long unlink handling in ptlrpcd == 11:18:23 (1776266303) start mds service on oleg307-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg307-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg307-server: LNET ready to unload oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 PASS 45 (278s) == conf-sanity test 69: replace an OST with the same index ========================================================== 11:23:02 (1776266582) start mds service on oleg307-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 845 (time 1776266679.39 total 10.01 last 84.40) - open/close 1631 (time 1776266689.39 total 20.02 last 78.57) - open/close 2414 (time 1776266699.42 total 30.04 last 78.10) - open/close 3249 (time 1776266709.42 total 40.05 last 83.46) - open/close 4086 (time 1776266719.45 total 50.07 last 83.51) - open/close 4899 (time 1776266729.45 total 60.07 last 81.29) - open/close 5711 (time 1776266739.45 total 70.07 last 81.19) - open/close 6571 (time 1776266749.46 total 80.08 last 85.93) - open/close 7407 (time 1776266759.46 total 90.08 last 83.57) - open/close 8169 (time 1776266769.47 total 100.09 last 76.16) - open/close 9010 (time 1776266779.47 total 110.10 last 84.05) - open/close 9792 (time 1776266789.48 total 120.10 last 78.19) total: 10000 open/close in 122.76 seconds: 81.46 ops/second - unlinked 0 (time 1776266796 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 718 (time 1776266888.24 total 10.01 last 71.71) - open/close 1408 (time 1776266898.26 total 20.03 last 68.91) - open/close 2120 (time 1776266908.27 total 30.03 last 71.15) - open/close 2867 (time 1776266918.27 total 40.04 last 74.68) - open/close 3611 (time 1776266928.27 total 50.04 last 74.40) - open/close 4302 (time 1776266938.27 total 60.04 last 69.07) - open/close 5058 (time 1776266948.27 total 70.04 last 75.59) - open/close 5852 (time 1776266958.28 total 80.05 last 79.35) - open/close 6595 (time 1776266968.29 total 90.06 last 74.24) - open/close 7441 (time 1776266978.29 total 100.06 last 84.58) - open/close 8248 (time 1776266988.30 total 110.06 last 80.66) - open/close 9052 (time 1776266998.30 total 120.07 last 80.37) - open/close 9806 (time 1776267008.33 total 130.10 last 75.19) total: 10000 open/close in 132.93 seconds: 75.23 ops/second - unlinked 0 (time 1776267016 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 824 (time 1776267108.70 total 10.01 last 82.33) - open/close 1460 (time 1776267118.70 total 20.01 last 63.59) - open/close 2284 (time 1776267128.71 total 30.02 last 82.35) - open/close 3134 (time 1776267138.73 total 40.04 last 84.78) - open/close 3887 (time 1776267148.74 total 50.05 last 75.27) - open/close 4717 (time 1776267158.75 total 60.05 last 82.93) - open/close 5546 (time 1776267168.75 total 70.06 last 82.85) - open/close 6215 (time 1776267178.78 total 80.09 last 66.73) - open/close 6865 (time 1776267188.79 total 90.10 last 64.92) - open/close 7416 (time 1776267198.79 total 100.10 last 55.08) - open/close 8017 (time 1776267208.81 total 110.12 last 59.99) - open/close 8692 (time 1776267218.82 total 120.13 last 67.46) - open/close 9391 (time 1776267228.82 total 130.13 last 69.87) total: 10000 open/close in 138.34 seconds: 72.28 ops/second - unlinked 0 (time 1776267241 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 833 (time 1776267328.05 total 10.00 last 83.28) - open/close 1664 (time 1776267338.06 total 20.01 last 83.02) - open/close 2476 (time 1776267348.06 total 30.02 last 81.16) - open/close 3292 (time 1776267358.07 total 40.02 last 81.58) - open/close 4102 (time 1776267368.07 total 50.02 last 81.00) - open/close 4970 (time 1776267378.08 total 60.03 last 86.69) - open/close 5747 (time 1776267388.10 total 70.05 last 77.56) - open/close 6522 (time 1776267398.10 total 80.05 last 77.47) - open/close 7036 (time 1776267408.12 total 90.07 last 51.32) - open/close 7781 (time 1776267418.12 total 100.07 last 74.49) - open/close 8595 (time 1776267428.12 total 110.07 last 81.40) - open/close 9400 (time 1776267438.13 total 120.08 last 80.42) total: 10000 open/close in 127.40 seconds: 78.49 ops/second - unlinked 0 (time 1776267449 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 837 (time 1776267536.53 total 10.00 last 83.69) - open/close 1646 (time 1776267546.54 total 20.01 last 80.84) - open/close 2456 (time 1776267556.55 total 30.02 last 80.89) - open/close 3236 (time 1776267566.57 total 40.03 last 77.92) - open/close 4039 (time 1776267576.57 total 50.03 last 80.29) - open/close 4877 (time 1776267586.57 total 60.04 last 83.78) - open/close 5703 (time 1776267596.57 total 70.04 last 82.59) - open/close 6440 (time 1776267606.58 total 80.04 last 73.66) - open/close 7282 (time 1776267616.58 total 90.05 last 84.18) - open/close 8128 (time 1776267626.59 total 100.05 last 84.54) - open/close 8904 (time 1776267636.60 total 110.06 last 77.51) - open/close 9695 (time 1776267646.61 total 120.07 last 79.03) total: 10000 open/close in 123.55 seconds: 80.94 ops/second - unlinked 0 (time 1776267654 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 857 (time 1776267739.47 total 10.00 last 85.69) - open/close 1665 (time 1776267749.48 total 20.00 last 80.78) - open/close 2520 (time 1776267759.48 total 30.00 last 85.49) - open/close 3387 (time 1776267769.49 total 40.02 last 86.61) - open/close 4168 (time 1776267779.50 total 50.03 last 78.00) - open/close 4969 (time 1776267789.50 total 60.03 last 80.08) - open/close 5800 (time 1776267799.50 total 70.03 last 83.10) - open/close 6605 (time 1776267809.51 total 80.03 last 80.48) - open/close 7361 (time 1776267819.51 total 90.03 last 75.58) - open/close 7992 (time 1776267829.53 total 100.06 last 62.96) - open/close 8697 (time 1776267839.55 total 110.08 last 70.37) - open/close 9475 (time 1776267849.56 total 120.09 last 77.72) total: 10000 open/close in 126.29 seconds: 79.18 ops/second - unlinked 0 (time 1776267859 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 854 (time 1776267948.62 total 10.01 last 85.33) - open/close 1552 (time 1776267958.62 total 20.02 last 69.75) - open/close 2411 (time 1776267968.63 total 30.02 last 85.85) - open/close 3209 (time 1776267978.63 total 40.02 last 79.80) - open/close 4074 (time 1776267988.64 total 50.03 last 86.44) - open/close 4965 (time 1776267998.64 total 60.04 last 89.05) - open/close 5786 (time 1776268008.65 total 70.05 last 82.02) - open/close 6615 (time 1776268018.67 total 80.06 last 82.77) - open/close 7432 (time 1776268028.67 total 90.06 last 81.69) - open/close 8217 (time 1776268038.67 total 100.07 last 78.46) - open/close 9124 (time 1776268048.68 total 110.08 last 90.62) total: 10000 open/close in 119.81 seconds: 83.47 ops/second - unlinked 0 (time 1776268061 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 815 (time 1776268144.92 total 10.01 last 81.45) - open/close 1559 (time 1776268154.93 total 20.01 last 74.36) - open/close 2263 (time 1776268164.94 total 30.02 last 70.36) - open/close 2976 (time 1776268174.95 total 40.03 last 71.22) - open/close 3752 (time 1776268184.96 total 50.04 last 77.54) - open/close 4565 (time 1776268194.97 total 60.05 last 81.22) - open/close 5249 (time 1776268204.97 total 70.05 last 68.36) - open/close 5968 (time 1776268214.99 total 80.07 last 71.79) - open/close 6661 (time 1776268225.01 total 90.09 last 69.11) - open/close 7243 (time 1776268235.02 total 100.10 last 58.15) - open/close 7945 (time 1776268245.02 total 110.10 last 70.19) - open/close 8649 (time 1776268255.04 total 120.12 last 70.27) - open/close 9489 (time 1776268265.04 total 130.12 last 83.97) total: 10000 open/close in 135.74 seconds: 73.67 ops/second - unlinked 0 (time 1776268275 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 822 (time 1776268371.52 total 10.00 last 82.20) - open/close 1652 (time 1776268381.53 total 20.01 last 82.91) - open/close 2516 (time 1776268391.54 total 30.03 last 86.27) - open/close 3353 (time 1776268401.55 total 40.03 last 83.63) - open/close 4206 (time 1776268411.56 total 50.04 last 85.22) - open/close 4873 (time 1776268421.57 total 60.05 last 66.66) - open/close 5755 (time 1776268431.58 total 70.07 last 88.04) - open/close 6570 (time 1776268441.58 total 80.07 last 81.50) - open/close 7323 (time 1776268451.61 total 90.09 last 75.11) - open/close 8171 (time 1776268461.61 total 100.09 last 84.80) - open/close 9033 (time 1776268471.61 total 110.10 last 86.16) - open/close 9827 (time 1776268481.62 total 120.10 last 79.37) total: 10000 open/close in 122.57 seconds: 81.58 ops/second - unlinked 0 (time 1776268487 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 776 (time 1776268582.31 total 10.01 last 77.55) - open/close 1520 (time 1776268592.33 total 20.03 last 74.26) - open/close 2216 (time 1776268602.33 total 30.03 last 69.58) - open/close 2962 (time 1776268612.35 total 40.05 last 74.47) - open/close 3771 (time 1776268622.35 total 50.05 last 80.87) - open/close 4535 (time 1776268632.35 total 60.05 last 76.37) - open/close 5293 (time 1776268642.36 total 70.06 last 75.72) - open/close 6056 (time 1776268652.37 total 80.07 last 76.25) - open/close 6903 (time 1776268662.37 total 90.07 last 84.67) - open/close 7621 (time 1776268672.38 total 100.08 last 71.80) - open/close 8295 (time 1776268682.38 total 110.08 last 67.36) - open/close 9178 (time 1776268692.39 total 120.09 last 88.24) total: 10000 open/close in 130.08 seconds: 76.88 ops/second - unlinked 0 (time 1776268706 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg307-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg307-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg307-server: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg307-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg307-server: LNET ready to unload oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 PASS 69 (2303s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 12:01:25 (1776268885) oleg307-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present oleg307-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present umount lustre on /mnt/lustre..... stop ost1 service on oleg307-server stop mds service on oleg307-server stop mds service on oleg307-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O ea_inode,large_dir -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg307-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.107@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre Starting client oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre Started clients oleg307-client.virtnet: 192.168.203.107@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) mount lustre on /mnt/lustre2..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre2 Starting client oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre2 Started clients oleg307-client.virtnet: 192.168.203.107@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1326500 1696 1206392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1252 126964 1% /mnt/lustre[OST:0] filesystem_summary: 142216 1252 126964 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 960000 272 959728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 268 49732 1% /mnt/lustre[OST:0] filesystem_summary: 50004 272 49732 1% /mnt/lustre creating 60000 hardlinks to oleg307-client.virtnet-0 creating 60000 hardlinks to oleg307-client.virtnet-1 waiting for PIDs 27035 27049 to complete - link 669 (time 1776268983.53 total 10.01 last 66.81) - link 1368 (time 1776268993.54 total 20.02 last 69.85) - link 1956 (time 1776269003.55 total 30.03 last 58.73) - link 2540 (time 1776269013.56 total 40.05 last 58.32) - link 3195 (time 1776269023.57 total 50.06 last 65.44) - link 3844 (time 1776269033.57 total 60.06 last 64.88) - link 4611 (time 1776269043.58 total 70.07 last 76.64) - link 5219 (time 1776269053.61 total 80.09 last 60.65) - link 5918 (time 1776269063.61 total 90.09 last 69.88) - link 6637 (time 1776269073.62 total 100.11 last 71.81) - link 7328 (time 1776269083.63 total 110.11 last 69.05) - link 7946 (time 1776269093.64 total 120.13 last 61.72) - link 8599 (time 1776269103.64 total 130.13 last 65.29) - link 9268 (time 1776269113.65 total 140.13 last 66.87) - link 10000 (time 1776269123.47 total 149.95 last 74.56) - link 10567 (time 1776269133.48 total 159.96 last 56.63) - link 11083 (time 1776269143.48 total 169.97 last 51.57) - link 11617 (time 1776269153.49 total 179.98 last 53.37) - link 12163 (time 1776269163.49 total 189.98 last 54.59) - link 12841 (time 1776269173.50 total 199.98 last 67.77) - link 13457 (time 1776269183.51 total 209.99 last 61.53) - link 14133 (time 1776269193.52 total 220.00 last 67.51) - link 14810 (time 1776269203.54 total 230.02 last 67.59) - link 15493 (time 1776269213.54 total 240.02 last 68.28) - link 16125 (time 1776269223.55 total 250.04 last 63.12) - link 16823 (time 1776269233.55 total 260.04 last 69.79) - link 17531 (time 1776269243.56 total 270.05 last 70.72) - link 18352 (time 1776269253.57 total 280.06 last 82.05) - link 19088 (time 1776269263.57 total 290.06 last 73.59) - link 19837 (time 1776269273.59 total 300.07 last 74.79) - link 20000 (time 1776269276.22 total 302.70 last 61.94) - link 20688 (time 1776269286.24 total 312.72 last 68.66) - link 21391 (time 1776269296.24 total 322.73 last 70.27) - link 22124 (time 1776269306.24 total 332.73 last 73.30) - link 22866 (time 1776269316.24 total 342.73 last 74.19) - link 23610 (time 1776269326.26 total 352.74 last 74.31) - link 24248 (time 1776269336.27 total 362.75 last 63.75) - link 24962 (time 1776269346.28 total 372.76 last 71.33) - link 25609 (time 1776269356.28 total 382.76 last 64.69) - link 26370 (time 1776269366.29 total 392.77 last 76.03) - link 27030 (time 1776269376.31 total 402.79 last 65.88) - link 27638 (time 1776269386.33 total 412.82 last 60.64) - link 28209 (time 1776269396.35 total 422.83 last 57.02) - link 28885 (time 1776269406.35 total 432.83 last 67.58) - link 29562 (time 1776269416.36 total 442.84 last 67.65) - link 30000 (time 1776269422.27 total 448.75 last 74.11) - link 30790 (time 1776269432.28 total 458.76 last 78.90) - link 31567 (time 1776269442.28 total 468.77 last 77.68) - link 32362 (time 1776269452.29 total 478.77 last 79.47) - link 33136 (time 1776269462.30 total 488.78 last 77.31) - link 33940 (time 1776269472.30 total 498.78 last 80.39) - link 34619 (time 1776269482.33 total 508.81 last 67.72) - link 35260 (time 1776269492.33 total 518.81 last 64.09) - link 35775 (time 1776269502.33 total 528.81 last 51.49) - link 36399 (time 1776269512.34 total 538.83 last 62.32) - link 37080 (time 1776269522.35 total 548.84 last 68.02) - link 37796 (time 1776269532.35 total 558.84 last 71.60) - link 38493 (time 1776269542.36 total 568.84 last 69.68) - link 39161 (time 1776269552.39 total 578.87 last 66.59) - link 39835 (time 1776269562.40 total 588.88 last 67.35) - link 40000 (time 1776269564.47 total 590.95 last 79.56) - link 40666 (time 1776269574.47 total 600.96 last 66.56) - link 41359 (time 1776269584.48 total 610.97 last 69.24) - link 41992 (time 1776269594.49 total 620.98 last 63.23) - link 42691 (time 1776269604.50 total 630.99 last 69.85) - link 43411 (time 1776269614.52 total 641.00 last 71.89) - link 44024 (time 1776269624.54 total 651.03 last 61.12) - link 44607 (time 1776269634.55 total 661.04 last 58.25) - link 45296 (time 1776269644.56 total 671.05 last 68.84) - link 46015 (time 1776269654.56 total 681.05 last 71.90) - link 46617 (time 1776269664.58 total 691.06 last 60.11) - link 47285 (time 1776269674.58 total 701.06 last 66.79) - link 47941 (time 1776269684.58 total 711.07 last 65.59) - link 48634 (time 1776269694.58 total 721.07 last 69.28) - link 49344 (time 1776269704.59 total 731.07 last 70.98) - link 50000 (time 1776269713.98 total 740.46 last 69.85) - link 50607 (time 1776269723.98 total 750.46 last 60.69) - link 51090 (time 1776269733.99 total 760.48 last 48.23) - link 51494 (time 1776269744.00 total 770.48 last 40.38) - link 52018 (time 1776269754.01 total 780.49 last 52.36) - link 52559 (time 1776269764.01 total 790.50 last 54.06) - link 53197 (time 1776269774.02 total 800.50 last 63.76) - link 53831 (time 1776269784.03 total 810.51 last 63.35) - link 54541 (time 1776269794.03 total 820.52 last 70.97) - link 55271 (time 1776269804.03 total 830.52 last 73.00) - link 55967 (time 1776269814.03 total 840.52 last 69.58) - link 56766 (time 1776269824.04 total 850.53 last 79.83) - link 57521 (time 1776269834.06 total 860.55 last 75.35) - link 58202 (time 1776269844.07 total 870.56 last 68.05) - link 58893 (time 1776269854.07 total 880.56 last 69.08) - link 59555 (time 1776269864.08 total 890.56 last 66.18) total: 60000 link in 896.89 seconds: 66.90 ops/second - link 660 (time 1776268984.59 total 10.01 last 65.95) - link 1360 (time 1776268994.59 total 20.01 last 69.95) - link 1931 (time 1776269004.60 total 30.03 last 57.04) - link 2511 (time 1776269014.62 total 40.04 last 57.90) - link 3181 (time 1776269024.64 total 50.06 last 66.88) - link 3855 (time 1776269034.64 total 60.06 last 67.38) - link 4598 (time 1776269044.65 total 70.07 last 74.23) - link 5193 (time 1776269054.67 total 80.10 last 59.37) - link 5885 (time 1776269064.69 total 90.11 last 69.11) - link 6600 (time 1776269074.70 total 100.12 last 71.41) - link 7276 (time 1776269084.73 total 110.15 last 67.40) - link 7908 (time 1776269094.74 total 120.16 last 63.12) - link 8567 (time 1776269104.75 total 130.17 last 65.86) - link 9249 (time 1776269114.76 total 140.18 last 68.09) - link 9982 (time 1776269124.77 total 150.19 last 73.26) - link 10522 (time 1776269134.80 total 160.22 last 53.83) - link 11032 (time 1776269144.81 total 170.24 last 50.93) - link 11557 (time 1776269154.82 total 180.24 last 52.49) - link 12104 (time 1776269164.82 total 190.24 last 54.70) - link 12783 (time 1776269174.82 total 200.24 last 67.89) - link 13357 (time 1776269184.83 total 210.25 last 57.34) - link 14026 (time 1776269194.84 total 220.26 last 66.83) - link 14705 (time 1776269204.87 total 230.29 last 67.68) - link 15402 (time 1776269214.88 total 240.30 last 69.66) - link 16044 (time 1776269224.88 total 250.30 last 64.17) - link 16759 (time 1776269234.89 total 260.31 last 71.47) - link 17477 (time 1776269244.90 total 270.32 last 71.72) - link 18293 (time 1776269254.91 total 280.33 last 81.48) - link 19024 (time 1776269264.92 total 290.35 last 73.01) - link 19764 (time 1776269274.93 total 300.35 last 73.96) - link 20000 (time 1776269278.65 total 304.07 last 63.46) - link 20710 (time 1776269288.65 total 314.07 last 70.99) - link 21397 (time 1776269298.66 total 324.08 last 68.63) - link 22129 (time 1776269308.67 total 334.09 last 73.14) - link 22929 (time 1776269318.68 total 344.10 last 79.89) - link 23652 (time 1776269328.69 total 354.11 last 72.25) - link 24258 (time 1776269338.69 total 364.11 last 60.59) - link 24959 (time 1776269348.70 total 374.12 last 70.01) - link 25610 (time 1776269358.71 total 384.13 last 65.05) - link 26351 (time 1776269368.72 total 394.14 last 74.00) - link 26986 (time 1776269378.73 total 404.15 last 63.44) - link 27611 (time 1776269388.75 total 414.17 last 62.39) - link 28204 (time 1776269398.77 total 424.19 last 59.20) - link 28881 (time 1776269408.77 total 434.19 last 67.66) - link 29564 (time 1776269418.78 total 444.20 last 68.26) - link 30000 (time 1776269424.82 total 450.25 last 72.13) - link 30778 (time 1776269434.83 total 460.25 last 77.73) - link 31569 (time 1776269444.85 total 470.27 last 78.98) - link 32373 (time 1776269454.85 total 480.27 last 80.35) - link 33171 (time 1776269464.86 total 490.28 last 79.78) - link 33944 (time 1776269474.87 total 500.29 last 77.21) - link 34613 (time 1776269484.87 total 510.30 last 66.85) - link 35231 (time 1776269494.91 total 520.33 last 61.61) - link 35744 (time 1776269504.91 total 530.33 last 51.29) - link 36408 (time 1776269514.92 total 540.34 last 66.31) - link 37072 (time 1776269524.93 total 550.35 last 66.36) - link 37798 (time 1776269534.93 total 560.35 last 72.56) - link 38502 (time 1776269544.94 total 570.36 last 70.35) - link 39131 (time 1776269554.95 total 580.37 last 62.83) - link 39843 (time 1776269564.96 total 590.38 last 71.12) - link 40000 (time 1776269567.64 total 593.06 last 58.67) - link 40713 (time 1776269577.64 total 603.06 last 71.27) - link 41409 (time 1776269587.65 total 613.08 last 69.52) - link 42073 (time 1776269597.65 total 623.08 last 66.38) - link 42798 (time 1776269607.66 total 633.08 last 72.46) - link 43504 (time 1776269617.66 total 643.08 last 70.59) - link 44091 (time 1776269627.68 total 653.10 last 58.62) - link 44711 (time 1776269637.69 total 663.11 last 61.94) - link 45390 (time 1776269647.69 total 673.11 last 67.87) - link 46103 (time 1776269657.70 total 683.12 last 71.22) - link 46704 (time 1776269667.71 total 693.13 last 60.05) - link 47364 (time 1776269677.71 total 703.13 last 65.99) - link 48053 (time 1776269687.72 total 713.14 last 68.87) - link 48716 (time 1776269697.72 total 723.14 last 66.30) - link 49447 (time 1776269707.73 total 733.16 last 72.97) - link 50000 (time 1776269715.86 total 741.28 last 68.06) - link 50578 (time 1776269725.89 total 751.31 last 57.64) - link 51049 (time 1776269735.90 total 761.32 last 47.05) - link 51440 (time 1776269745.90 total 771.32 last 39.08) - link 51981 (time 1776269755.91 total 781.33 last 54.07) - link 52505 (time 1776269765.93 total 791.35 last 52.30) - link 53177 (time 1776269775.94 total 801.36 last 67.12) - link 53834 (time 1776269785.94 total 811.37 last 65.67) - link 54528 (time 1776269795.97 total 821.39 last 69.24) - link 55281 (time 1776269805.98 total 831.40 last 75.18) - link 55992 (time 1776269816.01 total 841.43 last 70.91) - link 56809 (time 1776269826.02 total 851.44 last 81.63) - link 57542 (time 1776269836.08 total 861.50 last 72.84) - link 58233 (time 1776269846.08 total 871.51 last 69.09) - link 58915 (time 1776269856.10 total 881.52 last 68.09) - link 59590 (time 1776269866.14 total 891.56 last 67.22) total: 60000 link in 897.26 seconds: 66.87 ops/second estimate 37992s left after 120000 files / 898s umount lustre on /mnt/lustre2..... Stopping client oleg307-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg307-server: LNET ready to unload oleg307-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. ETA 37992s after 120000 files / 898s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg307-server: e2fsck 1.47.3-wc2 (11-Nov-2025) oleg307-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 29], used inodes 276/960000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2680: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 193 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 194 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 195 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 196 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 197 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 200 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 201 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 202 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 203 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 204 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 205 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 206 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 207 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 208 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 209 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 210 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 211 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 212 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 213 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 214 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 215 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 216 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 217 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 218 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 219 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 220 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 221 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 223 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 224 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 225 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 226 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 227 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 228 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 229 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 230 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 231 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 232 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 233 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 234 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 235 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 236 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 237 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 238 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 239 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 240 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 241 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 242 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 243 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 244 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 245 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 246 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 247 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 248 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 249 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 250 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 251 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 252 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 253 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 254 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 255 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 256 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 257 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 258 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 259 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 260 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 261 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 262 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 263 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 264 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 265 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 266 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 267 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 268 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 269 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 270 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 271 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2660: increase inode 277 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 277 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2680: increase inode 278 badness 0 to 2 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] group 20 finished [Thread 0] group 21 finished [Thread 0] group 22 finished [Thread 0] group 23 finished [Thread 0] group 24 finished [Thread 0] group 25 finished [Thread 0] group 26 finished [Thread 0] group 27 finished [Thread 0] group 28 finished [Thread 0] group 29 finished [Thread 0] Pass 1: Memory used: 352k/1208k (126k/227k), time: 0.04/ 0.01/ 0.01 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 47.35MB/s [Thread 0] Scanned group range [0, 29], used inodes 276/282 Pass 2: Checking directory structure Pass 2: Memory used: 352k/304k (82k/271k), time: 1.05/ 0.33/ 0.09 Pass 2: I/O read: 48MB, write: 0MB, rate: 45.69MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 352k/304k (82k/271k), time: 1.17/ 0.36/ 0.11 Pass 3: Memory used: 352k/304k (80k/273k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 352k/0k (79k/274k), time: 0.06/ 0.02/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 18.13MB/s Pass 5: Checking group summary information Pass 5: Memory used: 492k/0k (77k/416k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 145.03MB/s 276 inodes used (0.03%, out of 960000) 3 non-contiguous files (1.1%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 278306 blocks used (46.38%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 492k/0k (80k/413k), time: 1.24/ 0.39/ 0.11 I/O read: 49MB, write: 0MB, rate: 39.44MB/s pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 oleg307-server: oleg307-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (1126s) Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) oleg307-server: oleg307-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3957 sec ========== 12:21:11 (1776270071)