== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:21:41 (1785313301) start mds service on oleg120-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg120-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 120 (time 1785313409.97 total 10.45 last 11.48) - open/close 239 (time 1785313420.45 total 20.93 last 11.35) - open/close 472 (time 1785313430.47 total 30.96 last 23.25) - open/close 646 (time 1785313440.48 total 40.97 last 17.37) - open/close 799 (time 1785313450.49 total 50.98 last 15.28) - open/close 921 (time 1785313460.52 total 61.00 last 12.17) - open/close 1056 (time 1785313470.53 total 71.02 last 13.47) - open/close 1172 (time 1785313480.57 total 81.06 last 11.56) - open/close 1304 (time 1785313491.23 total 91.72 last 12.38) - open/close 1470 (time 1785313502.24 total 102.72 last 15.09) - open/close 1639 (time 1785313512.62 total 113.10 last 16.28) - open/close 1751 (time 1785313523.17 total 123.66 last 10.61) - open/close 1897 (time 1785313533.42 total 133.91 last 14.24) - open/close 2051 (time 1785313543.42 total 143.91 last 15.40) - open/close 2157 (time 1785313553.44 total 153.93 last 10.58) - open/close 2263 (time 1785313563.47 total 163.95 last 10.57) - open/close 2434 (time 1785313574.51 total 175.00 last 15.49) - open/close 2559 (time 1785313585.19 total 185.68 last 11.70) - open/close 2659 (time 1785313595.87 total 196.36 last 9.36) - open/close 2788 (time 1785313605.90 total 206.39 last 12.87) - open/close 2911 (time 1785313616.35 total 216.84 last 11.77) - open/close 3094 (time 1785313626.41 total 226.89 last 18.20) - open/close 3187 (time 1785313637.36 total 237.85 last 8.49) - open/close 3326 (time 1785313647.36 total 247.85 last 13.90) - open/close 3485 (time 1785313658.11 total 258.59 last 14.80) - open/close 3688 (time 1785313668.11 total 268.60 last 20.30) - open/close 3751 (time 1785313681.21 total 281.69 last 4.81) - open/close 3850 (time 1785313691.95 total 292.43 last 9.22) - open/close 3961 (time 1785313702.07 total 302.56 last 10.96) - open/close 4134 (time 1785313712.09 total 312.57 last 17.28) - open/close 4282 (time 1785313722.30 total 322.79 last 14.49) - open/close 4453 (time 1785313732.34 total 332.82 last 17.04) - open/close 4576 (time 1785313742.35 total 342.83 last 12.29) - open/close 4718 (time 1785313752.35 total 352.84 last 14.19) - open/close 4832 (time 1785313762.59 total 363.08 last 11.14) - open/close 4959 (time 1785313773.10 total 373.59 last 12.08) - open/close 5063 (time 1785313783.11 total 383.60 last 10.39) - open/close 5190 (time 1785313793.68 total 394.17 last 12.01) - open/close 5376 (time 1785313804.52 total 405.01 last 17.16) - open/close 5510 (time 1785313814.53 total 415.01 last 13.39) - open/close 5667 (time 1785313824.64 total 425.13 last 15.53) - open/close 5850 (time 1785313835.16 total 435.65 last 17.39) - open/close 5943 (time 1785313845.18 total 445.67 last 9.28) - open/close 6091 (time 1785313855.99 total 456.48 last 13.69) - open/close 6156 (time 1785313866.00 total 466.49 last 6.49) - open/close 6260 (time 1785313876.02 total 476.51 last 10.38) - open/close 6362 (time 1785313886.02 total 486.51 last 10.20) - open/close 6436 (time 1785313896.43 total 496.91 last 7.11) - open/close 6547 (time 1785313907.46 total 507.95 last 10.06) - open/close 6706 (time 1785313917.85 total 518.34 last 15.30) - open/close 6856 (time 1785313928.74 total 529.23 last 13.78) - open/close 7025 (time 1785313938.77 total 539.26 last 16.84) - open/close 7080 (time 1785313951.53 total 552.02 last 4.31) - open/close 7206 (time 1785313962.20 total 562.69 last 11.80) - open/close 7368 (time 1785313972.23 total 572.71 last 16.16) - open/close 7555 (time 1785313982.23 total 582.72 last 18.69) - open/close 7661 (time 1785313992.23 total 592.72 last 10.60) - open/close 7831 (time 1785314002.72 total 603.20 last 16.22) - open/close 7946 (time 1785314014.78 total 615.27 last 9.53) - open/close 8108 (time 1785314025.79 total 626.27 last 14.72) - open/close 8308 (time 1785314036.80 total 637.29 last 18.16) - open/close 8448 (time 1785314047.64 total 648.13 last 12.91) - open/close 8571 (time 1785314058.21 total 658.70 last 11.64) - open/close 8751 (time 1785314068.21 total 668.70 last 17.99) - open/close 8803 (time 1785314078.31 total 678.80 last 5.15) - open/close 8887 (time 1785314089.19 total 689.68 last 7.72) - open/close 8964 (time 1785314099.66 total 700.15 last 7.35) - open/close 9043 (time 1785314109.95 total 710.43 last 7.68) - open/close 9129 (time 1785314121.07 total 721.56 last 7.73) - open/close 9234 (time 1785314131.57 total 732.06 last 10.00) - open/close 9354 (time 1785314142.06 total 742.54 last 11.45) - open/close 9563 (time 1785314152.69 total 753.18 last 19.65) - open/close 9706 (time 1785314163.56 total 764.04 last 13.16) - open/close 9876 (time 1785314173.85 total 774.34 last 16.51) - open/close 9957 (time 1785314187.38 total 787.86 last 5.99) - open/close 10120 (time 1785314198.34 total 798.83 last 14.87) - open/close 10288 (time 1785314208.35 total 808.84 last 16.78) - open/close 10365 (time 1785314219.38 total 819.87 last 6.98) - open/close 10500 (time 1785314230.41 total 830.89 last 12.25) - open/close 10618 (time 1785314240.41 total 840.90 last 11.80) - open/close 10677 (time 1785314250.41 total 850.90 last 5.90) - open/close 10853 (time 1785314262.38 total 862.87 last 14.71) - open/close 10985 (time 1785314273.26 total 873.75 last 12.13) - open/close 11189 (time 1785314283.35 total 883.84 last 20.22) - open/close 11255 (time 1785314294.44 total 894.93 last 5.95) - open/close 11454 (time 1785314304.87 total 905.36 last 19.08) - open/close 11526 (time 1785314315.51 total 916.00 last 6.77) - open/close 11600 (time 1785314327.09 total 927.58 last 6.39) - open/close 11690 (time 1785314338.03 total 938.52 last 8.23) - open/close 11851 (time 1785314348.04 total 948.53 last 16.09) - open/close 11976 (time 1785314359.65 total 960.13 last 10.77) - open/close 12088 (time 1785314370.42 total 970.90 last 10.40) - open/close 12201 (time 1785314381.31 total 981.79 last 10.38) - open/close 12305 (time 1785314391.32 total 991.81 last 10.39) - open/close 12417 (time 1785314401.33 total 1001.81 last 11.19) - open/close 12636 (time 1785314412.34 total 1012.83 last 19.88) - open/close 12755 (time 1785314425.42 total 1025.91 last 9.10) - open/close 12908 (time 1785314435.43 total 1035.92 last 15.29) - open/close 13121 (time 1785314446.51 total 1047.00 last 19.22) - open/close 13203 (time 1785314457.19 total 1057.68 last 7.68) - open/close 13436 (time 1785314468.04 total 1068.52 last 21.49) - open/close 13592 (time 1785314478.64 total 1079.13 last 14.71) - open/close 13798 (time 1785314488.66 total 1089.14 last 20.57) - open/close 13968 (time 1785314498.68 total 1099.17 last 16.95) - open/close 14076 (time 1785314509.78 total 1110.27 last 9.73) - open/close 14126 (time 1785314520.50 total 1120.99 last 4.67) - open/close 14243 (time 1785314533.61 total 1134.10 last 8.92) - open/close 14329 (time 1785314543.63 total 1144.11 last 8.59) - open/close 14433 (time 1785314554.55 total 1155.04 last 9.52) - open/close 14651 (time 1785314564.59 total 1165.08 last 21.72) - open/close 14869 (time 1785314574.60 total 1175.09 last 21.78) - open/close 15041 (time 1785314586.66 total 1187.15 last 14.26) - open/close 15132 (time 1785314596.66 total 1197.15 last 9.10) - open/close 15271 (time 1785314607.40 total 1207.89 last 12.95) - open/close 15412 (time 1785314617.78 total 1218.27 last 13.58) - open/close 15484 (time 1785314628.67 total 1229.16 last 6.61) - open/close 15666 (time 1785314638.68 total 1239.17 last 18.18) - open/close 15770 (time 1785314649.39 total 1249.88 last 9.71) - open/close 15896 (time 1785314660.45 total 1260.94 last 11.39) - open/close 15993 (time 1785314670.46 total 1270.94 last 9.70) - open/close 16537 (time 1785314680.51 total 1281.00 last 54.11) - open/close 17033 (time 1785314690.52 total 1291.01 last 49.53) - open/close 17524 (time 1785314700.54 total 1301.02 last 49.03) - open/close 17814 (time 1785314710.60 total 1311.09 last 28.82) - open/close 18027 (time 1785314720.61 total 1321.10 last 21.28) - open/close 18579 (time 1785314730.61 total 1331.10 last 55.19) - open/close 19098 (time 1785314740.65 total 1341.14 last 51.68) - open/close 19550 (time 1785314750.67 total 1351.16 last 45.12) - open/close 20000 (time 1785314759.91 total 1360.40 last 48.72) - open/close 20701 (time 1785314769.92 total 1370.41 last 70.00) - open/close 21305 (time 1785314779.93 total 1380.41 last 60.38) - open/close 22016 (time 1785314789.96 total 1390.44 last 70.89) - open/close 22570 (time 1785314799.96 total 1400.45 last 55.37) - open/close 23541 (time 1785314809.97 total 1410.46 last 97.02) - open/close 24249 (time 1785314819.98 total 1420.47 last 70.71) - open/close 24712 (time 1785314830.02 total 1430.50 last 46.14) - open/close 25266 (time 1785314840.04 total 1440.52 last 55.29) - open/close 25777 (time 1785314850.07 total 1450.56 last 50.91) - open/close 26230 (time 1785314860.08 total 1460.56 last 45.28) - open/close 26617 (time 1785314870.10 total 1470.59 last 38.61) - open/close 27008 (time 1785314880.22 total 1480.71 last 38.64) - open/close 27275 (time 1785314890.24 total 1490.72 last 26.66) - open/close 27502 (time 1785314900.28 total 1500.76 last 22.61) - open/close 28102 (time 1785314910.28 total 1510.77 last 59.97) - open/close 28628 (time 1785314920.28 total 1520.77 last 52.60) - open/close 29072 (time 1785314930.31 total 1530.80 last 44.27) - open/close 29517 (time 1785314940.32 total 1540.80 last 44.48) - open/close 30000 (time 1785314949.70 total 1550.19 last 51.48) - open/close 30625 (time 1785314959.71 total 1560.19 last 62.45) - open/close 31251 (time 1785314969.71 total 1570.20 last 62.56) - open/close 31768 (time 1785314979.72 total 1580.21 last 51.65) - open/close 32268 (time 1785314989.73 total 1590.22 last 49.95) - open/close 32756 (time 1785314999.75 total 1600.23 last 48.73) - open/close 33337 (time 1785315009.81 total 1610.29 last 57.75) - open/close 33913 (time 1785315019.81 total 1620.30 last 57.59) - open/close 34672 (time 1785315029.82 total 1630.31 last 75.79) - open/close 35386 (time 1785315039.84 total 1640.33 last 71.25) - open/close 36143 (time 1785315049.85 total 1650.34 last 75.67) - open/close 36829 (time 1785315059.87 total 1660.35 last 68.48) - open/close 37452 (time 1785315069.88 total 1670.37 last 62.22) - open/close 38009 (time 1785315079.90 total 1680.39 last 55.59) - open/close 38577 (time 1785315089.90 total 1690.39 last 56.78) - open/close 39042 (time 1785315099.95 total 1700.44 last 46.27) - open/close 39578 (time 1785315109.96 total 1710.44 last 53.58) - open/close 40000 (time 1785315116.32 total 1716.80 last 66.36) - open/close 40814 (time 1785315126.35 total 1726.83 last 81.16) - open/close 41321 (time 1785315136.35 total 1736.84 last 50.67) - open/close 41872 (time 1785315146.36 total 1746.84 last 55.07) - open/close 42694 (time 1785315156.36 total 1756.85 last 82.16) - open/close 43528 (time 1785315166.37 total 1766.85 last 83.37) - open/close 44234 (time 1785315176.38 total 1776.87 last 70.51) - open/close 45008 (time 1785315186.39 total 1786.87 last 77.33) - open/close 45611 (time 1785315196.48 total 1796.97 last 59.72) - open/close 46302 (time 1785315206.49 total 1806.98 last 69.06) - open/close 46727 (time 1785315216.49 total 1816.98 last 42.49) - open/close 47340 (time 1785315226.51 total 1826.99 last 61.21) - open/close 47844 (time 1785315236.52 total 1837.01 last 50.31) - open/close 48438 (time 1785315246.53 total 1847.02 last 59.35) - open/close 49031 (time 1785315256.57 total 1857.06 last 59.08) - open/close 49702 (time 1785315266.58 total 1867.07 last 67.05) total: 50000 open/close in 1870.37 seconds: 26.73 ops/second - unlinked 0 (time 1785315274 ; total 0 ; last 0) - unlinked 10000 (time 1785315445 ; total 171 ; last 171) - unlinked 20000 (time 1785315648 ; total 374 ; last 203) - unlinked 30000 (time 1785315821 ; total 547 ; last 173) - unlinked 40000 (time 1785315992 ; total 718 ; last 171) total: 50000 unlinks in 894 seconds: 55.928413 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: modules unloaded. pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2