== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:09:56 (1786313396) start mds service on oleg445-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 94 (time 1786313495.44 total 10.26 last 9.16) - open/close 235 (time 1786313505.45 total 20.27 last 14.08) - open/close 364 (time 1786313517.40 total 32.21 last 10.80) - open/close 590 (time 1786313528.35 total 43.16 last 20.64) - open/close 768 (time 1786313541.26 total 56.08 last 13.79) - open/close 931 (time 1786313552.23 total 67.04 last 14.86) - open/close 1069 (time 1786313563.01 total 77.82 last 12.80) - open/close 1173 (time 1786313573.04 total 87.85 last 10.37) - open/close 1348 (time 1786313583.04 total 97.86 last 17.49) - open/close 1478 (time 1786313593.76 total 108.57 last 12.13) - open/close 1639 (time 1786313603.79 total 118.60 last 16.06) - open/close 1799 (time 1786313613.80 total 128.61 last 15.98) - open/close 1990 (time 1786313624.43 total 139.24 last 17.98) - open/close 2135 (time 1786313635.53 total 150.35 last 13.05) - open/close 2228 (time 1786313646.44 total 161.26 last 8.52) - open/close 2365 (time 1786313657.13 total 171.95 last 12.81) - open/close 2598 (time 1786313667.77 total 182.59 last 21.90) - open/close 2779 (time 1786313677.79 total 192.61 last 18.06) - open/close 2958 (time 1786313687.85 total 202.67 last 17.80) - open/close 3127 (time 1786313697.87 total 212.68 last 16.87) - open/close 3321 (time 1786313707.87 total 222.68 last 19.39) - open/close 3511 (time 1786313718.54 total 233.35 last 17.81) - open/close 3710 (time 1786313728.86 total 243.68 last 19.27) - open/close 3946 (time 1786313738.87 total 253.69 last 23.57) - open/close 4123 (time 1786313749.09 total 263.90 last 17.32) - open/close 4287 (time 1786313759.09 total 273.91 last 16.40) - open/close 4518 (time 1786313771.77 total 286.58 last 18.23) - open/close 4702 (time 1786313781.92 total 296.73 last 18.12) - open/close 4956 (time 1786313791.93 total 306.75 last 25.37) - open/close 5073 (time 1786313802.65 total 317.46 last 10.92) - open/close 5139 (time 1786313814.81 total 329.62 last 5.43) - open/close 5244 (time 1786313824.96 total 339.78 last 10.34) - open/close 5391 (time 1786313835.93 total 350.75 last 13.40) - open/close 5628 (time 1786313845.94 total 360.76 last 23.67) - open/close 5706 (time 1786313855.97 total 370.79 last 7.78) - open/close 5884 (time 1786313866.99 total 381.81 last 16.15) - open/close 6002 (time 1786313877.18 total 392.00 last 11.58) - open/close 6135 (time 1786313887.49 total 402.31 last 12.90) - open/close 6260 (time 1786313898.17 total 412.99 last 11.70) - open/close 6422 (time 1786313908.44 total 423.25 last 15.78) - open/close 6587 (time 1786313918.71 total 433.52 last 16.07) - open/close 6816 (time 1786313928.71 total 443.52 last 22.90) - open/close 7030 (time 1786313939.47 total 454.28 last 19.89) - open/close 7267 (time 1786313949.48 total 464.30 last 23.67) - open/close 7380 (time 1786313960.41 total 475.22 last 10.34) - open/close 7459 (time 1786313970.44 total 485.25 last 7.88) - open/close 7693 (time 1786313980.45 total 495.27 last 23.36) - open/close 7765 (time 1786313990.68 total 505.50 last 7.04) - open/close 7933 (time 1786314001.25 total 516.06 last 15.91) - open/close 8179 (time 1786314011.25 total 526.07 last 24.58) - open/close 8354 (time 1786314021.79 total 536.61 last 16.61) - open/close 8575 (time 1786314032.89 total 547.71 last 19.91) - open/close 8720 (time 1786314043.01 total 557.83 last 14.32) - open/close 8858 (time 1786314053.04 total 567.86 last 13.76) - open/close 9005 (time 1786314063.05 total 577.86 last 14.70) - open/close 9142 (time 1786314073.85 total 588.66 last 12.69) - open/close 9346 (time 1786314083.85 total 598.67 last 20.38) - open/close 9531 (time 1786314093.86 total 608.67 last 18.50) - open/close 9771 (time 1786314103.86 total 618.67 last 24.00) - open/close 9923 (time 1786314114.20 total 629.02 last 14.70) - open/close 10000 (time 1786314122.71 total 637.52 last 9.05) - open/close 10090 (time 1786314134.46 total 649.27 last 7.66) - open/close 10295 (time 1786314144.79 total 659.61 last 19.83) - open/close 10428 (time 1786314155.30 total 670.11 last 12.66) - open/close 10581 (time 1786314166.62 total 681.43 last 13.51) - open/close 10684 (time 1786314176.63 total 691.45 last 10.29) - open/close 10802 (time 1786314188.40 total 703.22 last 10.02) - open/close 10997 (time 1786314199.25 total 714.07 last 17.97) - open/close 11123 (time 1786314209.26 total 724.07 last 12.60) - open/close 11225 (time 1786314219.66 total 734.48 last 9.80) - open/close 11352 (time 1786314229.78 total 744.59 last 12.55) - open/close 11423 (time 1786314240.40 total 755.22 last 6.68) - open/close 11599 (time 1786314251.31 total 766.13 last 16.13) - open/close 11706 (time 1786314261.87 total 776.68 last 10.14) - open/close 11805 (time 1786314272.80 total 787.61 last 9.06) - open/close 11899 (time 1786314283.50 total 798.31 last 8.78) - open/close 12041 (time 1786314294.40 total 809.22 last 13.02) - open/close 12191 (time 1786314305.19 total 820.01 last 13.91) - open/close 12296 (time 1786314315.22 total 830.04 last 10.47) - open/close 12417 (time 1786314325.25 total 840.06 last 12.07) - open/close 12599 (time 1786314335.91 total 850.72 last 17.07) - open/close 12763 (time 1786314345.92 total 860.73 last 16.38) - open/close 12984 (time 1786314356.07 total 870.88 last 21.77) - open/close 13159 (time 1786314366.51 total 881.33 last 16.76) - open/close 13360 (time 1786314376.52 total 891.33 last 20.08) - open/close 13534 (time 1786314386.56 total 901.38 last 17.33) - open/close 13673 (time 1786314397.19 total 912.00 last 13.08) - open/close 13838 (time 1786314407.92 total 922.74 last 15.37) - open/close 13971 (time 1786314417.95 total 932.76 last 13.27) - open/close 14121 (time 1786314428.89 total 943.70 last 13.71) - open/close 14245 (time 1786314440.10 total 954.92 last 11.06) - open/close 14338 (time 1786314451.04 total 965.85 last 8.51) - open/close 14385 (time 1786314461.61 total 976.42 last 4.45) - open/close 14483 (time 1786314471.63 total 986.44 last 9.78) - open/close 14621 (time 1786314481.65 total 996.47 last 13.76) - open/close 14745 (time 1786314492.57 total 1007.38 last 11.36) - open/close 14865 (time 1786314504.87 total 1019.69 last 9.75) - open/close 15017 (time 1786314514.89 total 1029.71 last 15.17) - open/close 15198 (time 1786314525.45 total 1040.27 last 17.14) - open/close 15302 (time 1786314535.46 total 1050.27 last 10.39) - open/close 15420 (time 1786314545.47 total 1060.28 last 11.79) - open/close 15614 (time 1786314556.08 total 1070.90 last 18.28) - open/close 15648 (time 1786314566.46 total 1081.28 last 3.27) - open/close 15782 (time 1786314577.25 total 1092.06 last 12.43) - open/close 15910 (time 1786314587.28 total 1102.10 last 12.75) - open/close 16051 (time 1786314598.23 total 1113.05 last 12.88) - open/close 16155 (time 1786314608.56 total 1123.37 last 10.07) - open/close 16243 (time 1786314620.53 total 1135.34 last 7.35) - open/close 16337 (time 1786314631.61 total 1146.43 last 8.48) - open/close 16412 (time 1786314642.02 total 1156.84 last 7.20) - open/close 17026 (time 1786314652.03 total 1166.84 last 61.39) - open/close 17613 (time 1786314662.04 total 1176.85 last 58.62) - open/close 18225 (time 1786314672.05 total 1186.86 last 61.15) - open/close 18903 (time 1786314682.05 total 1196.86 last 67.79) - open/close 19609 (time 1786314692.05 total 1206.87 last 70.56) - open/close 20000 (time 1786314697.89 total 1212.70 last 67.00) - open/close 20685 (time 1786314707.90 total 1222.72 last 68.41) - open/close 21363 (time 1786314717.91 total 1232.73 last 67.73) - open/close 22006 (time 1786314727.92 total 1242.74 last 64.23) - open/close 22737 (time 1786314737.93 total 1252.74 last 73.04) - open/close 23464 (time 1786314747.94 total 1262.75 last 72.64) - open/close 24174 (time 1786314757.95 total 1272.76 last 70.92) - open/close 24875 (time 1786314767.95 total 1282.77 last 70.09) - open/close 25535 (time 1786314777.95 total 1292.77 last 66.00) - open/close 26226 (time 1786314787.96 total 1302.77 last 69.05) - open/close 26930 (time 1786314797.96 total 1312.78 last 70.37) - open/close 27631 (time 1786314807.97 total 1322.79 last 70.05) - open/close 28298 (time 1786314818.00 total 1332.81 last 66.51) - open/close 29008 (time 1786314828.00 total 1342.82 last 70.98) - open/close 29682 (time 1786314838.01 total 1352.82 last 67.37) - open/close 30000 (time 1786314842.70 total 1357.52 last 67.74) - open/close 30651 (time 1786314852.71 total 1367.53 last 65.02) - open/close 31314 (time 1786314862.71 total 1377.53 last 66.28) - open/close 31968 (time 1786314872.72 total 1387.54 last 65.34) - open/close 32619 (time 1786314882.73 total 1397.54 last 65.08) - open/close 33314 (time 1786314892.73 total 1407.55 last 69.44) - open/close 34009 (time 1786314902.74 total 1417.55 last 69.49) - open/close 34718 (time 1786314912.74 total 1427.55 last 70.88) - open/close 35348 (time 1786314922.75 total 1437.57 last 62.93) - open/close 36011 (time 1786314932.75 total 1447.57 last 66.29) - open/close 36649 (time 1786314942.76 total 1457.58 last 63.74) - open/close 37219 (time 1786314952.76 total 1467.58 last 56.98) - open/close 37844 (time 1786314962.77 total 1477.59 last 62.44) - open/close 38437 (time 1786314972.78 total 1487.60 last 59.25) - open/close 39136 (time 1786314982.79 total 1497.60 last 69.86) - open/close 39763 (time 1786314992.80 total 1507.61 last 62.64) - open/close 40000 (time 1786314996.04 total 1510.86 last 73.02) - open/close 40625 (time 1786315006.05 total 1520.86 last 62.48) - open/close 41320 (time 1786315016.05 total 1530.86 last 69.50) - open/close 42039 (time 1786315026.06 total 1540.87 last 71.85) - open/close 42749 (time 1786315036.07 total 1550.88 last 70.90) - open/close 43427 (time 1786315046.08 total 1560.89 last 67.75) - open/close 44156 (time 1786315056.08 total 1570.90 last 72.87) - open/close 44825 (time 1786315066.10 total 1580.91 last 66.80) - open/close 45546 (time 1786315076.11 total 1590.92 last 72.02) - open/close 46184 (time 1786315086.12 total 1600.93 last 63.74) - open/close 46884 (time 1786315096.12 total 1610.93 last 69.99) - open/close 47558 (time 1786315106.12 total 1620.94 last 67.36) - open/close 48114 (time 1786315116.13 total 1630.94 last 55.58) - open/close 48736 (time 1786315126.13 total 1640.95 last 62.16) - open/close 49365 (time 1786315136.14 total 1650.95 last 62.90) total: 50000 open/close in 1660.54 seconds: 30.11 ops/second - unlinked 0 (time 1786315150 ; total 0 ; last 0) - unlinked 10000 (time 1786315317 ; total 167 ; last 167) - unlinked 20000 (time 1786315497 ; total 347 ; last 180) - unlinked 30000 (time 1786315703 ; total 553 ; last 206) - unlinked 40000 (time 1786315908 ; total 758 ; last 205) total: 50000 unlinks in 959 seconds: 52.137642 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2