== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:31:14 (1785396674) start mds service on oleg114-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg114-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 160 (time 1785396769.51 total 10.01 last 15.99) - open/close 305 (time 1785396779.55 total 20.04 last 14.45) - open/close 524 (time 1785396789.59 total 30.09 last 21.81) - open/close 681 (time 1785396800.09 total 40.59 last 14.95) - open/close 805 (time 1785396810.09 total 50.59 last 12.40) - open/close 927 (time 1785396821.11 total 61.61 last 11.07) - open/close 1064 (time 1785396831.21 total 71.71 last 13.56) - open/close 1193 (time 1785396841.40 total 81.90 last 12.67) - open/close 1352 (time 1785396852.27 total 92.76 last 14.63) - open/close 1512 (time 1785396862.28 total 102.78 last 15.97) - open/close 1675 (time 1785396872.30 total 112.79 last 16.28) - open/close 1797 (time 1785396882.41 total 122.91 last 12.06) - open/close 1959 (time 1785396892.42 total 132.92 last 16.18) - open/close 2109 (time 1785396902.44 total 142.94 last 14.97) - open/close 2242 (time 1785396912.62 total 153.12 last 13.07) - open/close 2302 (time 1785396922.67 total 163.17 last 5.97) - open/close 2408 (time 1785396932.69 total 173.19 last 10.57) - open/close 2477 (time 1785396942.73 total 183.22 last 6.88) - open/close 2560 (time 1785396953.54 total 194.04 last 7.67) - open/close 2657 (time 1785396963.85 total 204.35 last 9.41) - open/close 2799 (time 1785396973.87 total 214.37 last 14.17) - open/close 2897 (time 1785396984.61 total 225.10 last 9.13) - open/close 3018 (time 1785396994.98 total 235.48 last 11.66) - open/close 3144 (time 1785397005.01 total 245.50 last 12.57) - open/close 3309 (time 1785397015.01 total 255.51 last 16.50) - open/close 3444 (time 1785397025.51 total 266.01 last 12.86) - open/close 3557 (time 1785397036.20 total 276.70 last 10.57) - open/close 3699 (time 1785397047.01 total 287.50 last 13.14) - open/close 3818 (time 1785397057.10 total 297.60 last 11.78) - open/close 3970 (time 1785397067.50 total 307.99 last 14.63) - open/close 4146 (time 1785397078.48 total 318.98 last 16.02) - open/close 4251 (time 1785397091.63 total 332.13 last 7.98) - open/close 4392 (time 1785397101.75 total 342.25 last 13.94) - open/close 4510 (time 1785397112.03 total 352.53 last 11.48) - open/close 4678 (time 1785397122.04 total 362.54 last 16.78) - open/close 4744 (time 1785397132.73 total 373.23 last 6.18) - open/close 5021 (time 1785397142.75 total 383.25 last 27.63) - open/close 5228 (time 1785397153.45 total 393.95 last 19.36) - open/close 5309 (time 1785397164.44 total 404.94 last 7.37) - open/close 5407 (time 1785397175.03 total 415.53 last 9.25) - open/close 5505 (time 1785397185.67 total 426.17 last 9.21) - open/close 5627 (time 1785397195.68 total 436.18 last 12.18) - open/close 5811 (time 1785397205.81 total 446.31 last 18.16) - open/close 5943 (time 1785397216.72 total 457.22 last 12.11) - open/close 6026 (time 1785397226.72 total 467.22 last 8.30) - open/close 6153 (time 1785397237.08 total 477.58 last 12.26) - open/close 6405 (time 1785397247.14 total 487.64 last 25.06) - open/close 6654 (time 1785397257.14 total 497.64 last 24.90) - open/close 6812 (time 1785397267.17 total 507.67 last 15.75) - open/close 6999 (time 1785397278.19 total 518.68 last 16.98) - open/close 7185 (time 1785397288.19 total 528.69 last 18.59) - open/close 7355 (time 1785397298.91 total 539.40 last 15.87) - open/close 7454 (time 1785397308.91 total 549.41 last 9.90) - open/close 7585 (time 1785397319.37 total 559.87 last 12.52) - open/close 7718 (time 1785397329.92 total 570.42 last 12.61) - open/close 7827 (time 1785397339.96 total 580.46 last 10.85) - open/close 7950 (time 1785397350.24 total 590.74 last 11.97) - open/close 8107 (time 1785397360.49 total 600.99 last 15.32) - open/close 8219 (time 1785397373.05 total 613.54 last 8.92) - open/close 8327 (time 1785397383.40 total 623.89 last 10.43) - open/close 8432 (time 1785397393.96 total 634.46 last 9.94) - open/close 8644 (time 1785397404.72 total 645.22 last 19.69) - open/close 8794 (time 1785397415.25 total 655.75 last 14.25) - open/close 8998 (time 1785397425.27 total 665.77 last 20.36) - open/close 9087 (time 1785397435.68 total 676.18 last 8.55) - open/close 9253 (time 1785397446.56 total 687.06 last 15.25) - open/close 9308 (time 1785397457.18 total 697.68 last 5.18) - open/close 9466 (time 1785397467.20 total 707.69 last 15.78) - open/close 9676 (time 1785397478.10 total 718.60 last 19.25) - open/close 9805 (time 1785397488.31 total 728.80 last 12.64) - open/close 9959 (time 1785397498.93 total 739.42 last 14.50) - open/close 10121 (time 1785397508.94 total 749.43 last 16.18) - open/close 10235 (time 1785397519.91 total 760.40 last 10.39) - open/close 10313 (time 1785397530.33 total 770.82 last 7.49) - open/close 10441 (time 1785397540.95 total 781.45 last 12.04) - open/close 10588 (time 1785397551.00 total 791.50 last 14.63) - open/close 10715 (time 1785397561.40 total 801.89 last 12.22) - open/close 10818 (time 1785397574.02 total 814.51 last 8.16) - open/close 10887 (time 1785397584.16 total 824.66 last 6.80) - open/close 11023 (time 1785397594.28 total 834.78 last 13.44) - open/close 11206 (time 1785397605.02 total 845.52 last 17.04) - open/close 11346 (time 1785397615.72 total 856.22 last 13.08) - open/close 11457 (time 1785397626.67 total 867.17 last 10.14) - open/close 11611 (time 1785397637.41 total 877.91 last 14.34) - open/close 11742 (time 1785397648.47 total 888.97 last 11.84) - open/close 11827 (time 1785397658.87 total 899.37 last 8.17) - open/close 11956 (time 1785397668.90 total 909.40 last 12.87) - open/close 12066 (time 1785397679.56 total 920.06 last 10.32) - open/close 12205 (time 1785397690.44 total 930.94 last 12.77) - open/close 12359 (time 1785397701.45 total 941.94 last 13.99) - open/close 12471 (time 1785397711.45 total 951.95 last 11.19) - open/close 12631 (time 1785397721.45 total 961.95 last 15.99) - open/close 12773 (time 1785397733.75 total 974.25 last 11.54) - open/close 12952 (time 1785397743.79 total 984.29 last 17.84) - open/close 13120 (time 1785397754.35 total 994.85 last 15.90) - open/close 13229 (time 1785397764.37 total 1004.87 last 10.88) - open/close 13356 (time 1785397774.56 total 1015.06 last 12.47) - open/close 13445 (time 1785397784.61 total 1025.11 last 8.86) - open/close 13551 (time 1785397794.62 total 1035.12 last 10.58) - open/close 13744 (time 1785397804.94 total 1045.44 last 18.71) - open/close 13857 (time 1785397814.95 total 1055.45 last 11.29) - open/close 14024 (time 1785397824.97 total 1065.47 last 16.66) - open/close 14129 (time 1785397835.48 total 1075.98 last 9.99) - open/close 14218 (time 1785397845.49 total 1085.98 last 8.89) - open/close 14268 (time 1785397856.49 total 1096.99 last 4.54) - open/close 14396 (time 1785397866.83 total 1107.33 last 12.39) - open/close 14564 (time 1785397877.49 total 1117.99 last 15.76) - open/close 14647 (time 1785397887.99 total 1128.49 last 7.90) - open/close 14735 (time 1785397898.47 total 1138.97 last 8.40) - open/close 14850 (time 1785397908.51 total 1149.01 last 11.45) - open/close 14973 (time 1785397918.81 total 1159.31 last 11.94) - open/close 15087 (time 1785397928.86 total 1169.36 last 11.35) - open/close 15306 (time 1785397941.05 total 1181.55 last 17.96) - open/close 15580 (time 1785397951.06 total 1191.56 last 27.39) - open/close 16248 (time 1785397961.09 total 1201.58 last 66.62) - open/close 17057 (time 1785397971.09 total 1211.58 last 80.90) - open/close 17893 (time 1785397981.09 total 1221.59 last 83.58) - open/close 18444 (time 1785397991.11 total 1231.61 last 54.97) - open/close 18928 (time 1785398001.11 total 1241.61 last 48.39) - open/close 19385 (time 1785398011.12 total 1251.62 last 45.67) - open/close 20000 (time 1785398019.74 total 1260.24 last 71.32) - open/close 20780 (time 1785398029.76 total 1270.26 last 77.84) - open/close 21528 (time 1785398039.78 total 1280.28 last 74.66) - open/close 22252 (time 1785398049.78 total 1290.28 last 72.39) - open/close 22792 (time 1785398059.83 total 1300.32 last 53.77) - open/close 23486 (time 1785398069.84 total 1310.33 last 69.33) - open/close 24245 (time 1785398079.84 total 1320.34 last 75.89) - open/close 24783 (time 1785398089.84 total 1330.34 last 53.79) - open/close 25379 (time 1785398099.85 total 1340.34 last 59.56) - open/close 25920 (time 1785398109.86 total 1350.36 last 54.04) - open/close 26703 (time 1785398119.89 total 1360.39 last 78.05) - open/close 27191 (time 1785398129.90 total 1370.39 last 48.77) - open/close 27760 (time 1785398139.92 total 1380.42 last 56.74) - open/close 28511 (time 1785398149.92 total 1390.42 last 75.10) - open/close 29199 (time 1785398159.93 total 1400.43 last 68.77) - open/close 29901 (time 1785398169.93 total 1410.43 last 70.18) - open/close 30664 (time 1785398179.94 total 1420.43 last 76.26) - open/close 31389 (time 1785398189.94 total 1430.44 last 72.45) - open/close 32207 (time 1785398199.96 total 1440.46 last 81.65) - open/close 32790 (time 1785398209.98 total 1450.48 last 58.19) - open/close 33279 (time 1785398219.99 total 1460.49 last 48.86) - open/close 33852 (time 1785398230.01 total 1470.51 last 57.19) - open/close 34443 (time 1785398240.01 total 1480.51 last 59.06) - open/close 34876 (time 1785398250.02 total 1490.51 last 43.29) - open/close 35624 (time 1785398260.02 total 1500.52 last 74.80) - open/close 36187 (time 1785398270.03 total 1510.53 last 56.21) - open/close 36745 (time 1785398280.06 total 1520.56 last 55.65) - open/close 37352 (time 1785398290.07 total 1530.57 last 60.66) - open/close 37958 (time 1785398300.08 total 1540.57 last 60.55) - open/close 38718 (time 1785398310.08 total 1550.58 last 75.94) - open/close 39386 (time 1785398320.10 total 1560.59 last 66.72) - open/close 40000 (time 1785398328.08 total 1568.58 last 76.87) - open/close 40736 (time 1785398338.09 total 1578.59 last 73.55) - open/close 41669 (time 1785398348.09 total 1588.59 last 93.28) - open/close 42317 (time 1785398358.10 total 1598.60 last 64.75) - open/close 42707 (time 1785398368.12 total 1608.62 last 38.90) - open/close 43109 (time 1785398378.14 total 1618.64 last 40.14) - open/close 43867 (time 1785398388.16 total 1628.66 last 75.64) - open/close 44641 (time 1785398398.17 total 1638.67 last 77.35) - open/close 45442 (time 1785398408.18 total 1648.68 last 79.97) - open/close 46220 (time 1785398418.19 total 1658.69 last 77.74) - open/close 46902 (time 1785398428.20 total 1668.70 last 68.13) - open/close 47509 (time 1785398438.21 total 1678.71 last 60.66) - open/close 48059 (time 1785398448.21 total 1688.71 last 54.98) - open/close 48420 (time 1785398458.23 total 1698.73 last 36.03) - open/close 48911 (time 1785398468.24 total 1708.74 last 49.06) - open/close 49472 (time 1785398478.24 total 1718.74 last 56.09) - open/close 49972 (time 1785398488.25 total 1728.75 last 49.95) total: 50000 open/close in 1729.59 seconds: 28.91 ops/second - unlinked 0 (time 1785398495 ; total 0 ; last 0) - unlinked 10000 (time 1785398657 ; total 162 ; last 162) - unlinked 20000 (time 1785398819 ; total 324 ; last 162) - unlinked 30000 (time 1785398979 ; total 484 ; last 160) - unlinked 40000 (time 1785399132 ; total 637 ; last 153) total: 50000 unlinks in 794 seconds: 62.972294 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: modules unloaded. pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2