== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:58 (1709787118) start mds service on oleg349-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg349-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 236 (time 1709787150.88 total 10.00 last 23.59) - open/close 468 (time 1709787161.75 total 20.88 last 21.34) - open/close 642 (time 1709787172.26 total 31.38 last 16.56) - open/close 994 (time 1709787182.26 total 41.39 last 35.19) - open/close 1267 (time 1709787192.39 total 51.51 last 26.95) - open/close 1372 (time 1709787202.99 total 62.12 last 9.90) - open/close 1642 (time 1709787213.02 total 72.14 last 26.93) - open/close 1892 (time 1709787223.35 total 82.48 last 24.19) - open/close 2028 (time 1709787233.98 total 93.10 last 12.80) - open/close 2177 (time 1709787244.79 total 103.91 last 13.78) - open/close 2382 (time 1709787254.80 total 113.92 last 20.48) - open/close 2500 (time 1709787264.99 total 124.11 last 11.58) - open/close 2766 (time 1709787275.28 total 134.41 last 25.83) - open/close 2911 (time 1709787287.32 total 146.45 last 12.04) - open/close 3044 (time 1709787297.34 total 156.47 last 13.28) - open/close 3183 (time 1709787307.48 total 166.61 last 13.71) - open/close 3311 (time 1709787317.78 total 176.90 last 12.43) - open/close 3505 (time 1709787328.76 total 187.88 last 17.67) - open/close 3666 (time 1709787339.75 total 198.88 last 14.65) - open/close 3828 (time 1709787349.76 total 208.89 last 16.19) - open/close 3975 (time 1709787360.65 total 219.77 last 13.50) - open/close 4172 (time 1709787371.50 total 230.62 last 18.16) - open/close 4340 (time 1709787381.50 total 240.63 last 16.80) - open/close 4585 (time 1709787391.81 total 250.93 last 23.78) - open/close 4798 (time 1709787401.82 total 260.94 last 21.27) - open/close 4959 (time 1709787412.38 total 271.51 last 15.24) - open/close 5142 (time 1709787422.72 total 281.85 last 17.69) - open/close 5294 (time 1709787432.81 total 291.93 last 15.08) - open/close 5452 (time 1709787442.99 total 302.12 last 15.51) - open/close 5578 (time 1709787453.74 total 312.86 last 11.73) - open/close 5776 (time 1709787464.39 total 323.51 last 18.59) - open/close 5940 (time 1709787474.57 total 333.69 last 16.11) - open/close 6080 (time 1709787484.69 total 343.81 last 13.84) - open/close 6236 (time 1709787494.69 total 353.82 last 15.59) - open/close 6453 (time 1709787505.64 total 364.77 last 19.82) - open/close 6693 (time 1709787516.22 total 375.34 last 22.69) - open/close 6905 (time 1709787527.25 total 386.38 last 19.21) - open/close 7079 (time 1709787538.14 total 397.26 last 15.99) - open/close 7179 (time 1709787548.39 total 407.51 last 9.76) - open/close 7331 (time 1709787558.70 total 417.83 last 14.73) - open/close 7510 (time 1709787569.63 total 428.75 last 16.38) - open/close 7723 (time 1709787580.63 total 439.75 last 19.36) - open/close 7824 (time 1709787590.95 total 450.08 last 9.78) - open/close 8011 (time 1709787601.43 total 460.55 last 17.85) - open/close 8131 (time 1709787611.43 total 470.56 last 12.00) - open/close 8332 (time 1709787621.45 total 480.57 last 20.07) - open/close 8442 (time 1709787631.45 total 490.58 last 11.00) - open/close 8664 (time 1709787641.48 total 500.61 last 22.14) - open/close 8834 (time 1709787651.48 total 510.61 last 17.00) - open/close 8969 (time 1709787662.42 total 521.54 last 12.34) - open/close 9136 (time 1709787672.44 total 531.57 last 16.66) - open/close 9364 (time 1709787682.47 total 541.60 last 22.74) - open/close 9485 (time 1709787692.48 total 551.61 last 12.08) - open/close 9654 (time 1709787702.51 total 561.64 last 16.85) - open/close 9822 (time 1709787712.53 total 571.65 last 16.77) - open/close 9997 (time 1709787722.54 total 581.66 last 17.48) - open/close 10131 (time 1709787732.55 total 591.68 last 13.38) - open/close 10329 (time 1709787742.68 total 601.81 last 19.55) - open/close 10531 (time 1709787752.94 total 612.07 last 19.68) - open/close 10819 (time 1709787762.97 total 622.09 last 28.73) - open/close 11364 (time 1709787772.98 total 632.11 last 54.41) - open/close 11858 (time 1709787783.00 total 642.12 last 49.33) - open/close 12349 (time 1709787793.00 total 652.13 last 49.07) - open/close 12895 (time 1709787803.02 total 662.14 last 54.54) - open/close 13436 (time 1709787813.03 total 672.16 last 54.01) - open/close 13916 (time 1709787823.04 total 682.16 last 47.97) - open/close 14412 (time 1709787833.05 total 692.18 last 49.53) - open/close 14854 (time 1709787843.06 total 702.18 last 44.17) - open/close 15339 (time 1709787853.07 total 712.19 last 48.47) - open/close 15769 (time 1709787863.08 total 722.21 last 42.94) - open/close 16197 (time 1709787873.15 total 732.27 last 42.51) - open/close 16578 (time 1709787883.16 total 742.28 last 38.06) - open/close 17015 (time 1709787893.17 total 752.29 last 43.67) - open/close 17455 (time 1709787903.19 total 762.31 last 43.91) - open/close 17893 (time 1709787913.19 total 772.31 last 43.79) - open/close 18297 (time 1709787923.19 total 782.31 last 40.40) - open/close 18683 (time 1709787933.20 total 792.32 last 38.57) - open/close 19067 (time 1709787943.21 total 802.34 last 38.34) - open/close 19449 (time 1709787953.23 total 812.36 last 38.13) - open/close 19829 (time 1709787963.25 total 822.37 last 37.94) - open/close 20000 (time 1709787967.72 total 826.84 last 38.28) - open/close 20352 (time 1709787977.75 total 836.88 last 35.08) - open/close 20768 (time 1709787987.77 total 846.89 last 41.52) - open/close 21192 (time 1709787997.80 total 856.92 last 42.28) - open/close 21548 (time 1709788007.81 total 866.93 last 35.58) - open/close 21915 (time 1709788017.82 total 876.94 last 36.65) - open/close 22226 (time 1709788027.83 total 886.95 last 31.07) - open/close 22562 (time 1709788037.85 total 896.97 last 33.53) - open/close 22889 (time 1709788047.87 total 907.00 last 32.63) - open/close 23211 (time 1709788057.89 total 917.02 last 32.13) - open/close 23544 (time 1709788067.93 total 927.05 last 33.19) - open/close 23920 (time 1709788077.94 total 937.06 last 37.56) - open/close 24349 (time 1709788087.96 total 947.09 last 42.81) - open/close 24784 (time 1709788097.96 total 957.09 last 43.50) - open/close 25207 (time 1709788107.97 total 967.10 last 42.25) - open/close 25637 (time 1709788117.99 total 977.11 last 42.94) - open/close 26055 (time 1709788128.01 total 987.13 last 41.73) - open/close 26480 (time 1709788138.02 total 997.14 last 42.45) - open/close 26901 (time 1709788148.03 total 1007.16 last 42.03) - open/close 27302 (time 1709788158.05 total 1017.17 last 40.03) - open/close 27715 (time 1709788168.06 total 1027.18 last 41.27) - open/close 28111 (time 1709788178.06 total 1037.18 last 39.58) - open/close 28515 (time 1709788188.06 total 1047.19 last 40.38) - open/close 28903 (time 1709788198.08 total 1057.20 last 38.75) - open/close 29272 (time 1709788208.10 total 1067.23 last 36.80) - open/close 29657 (time 1709788218.10 total 1077.23 last 38.50) - open/close 30000 (time 1709788226.79 total 1085.92 last 39.48) - open/close 30363 (time 1709788236.82 total 1095.95 last 36.19) - open/close 30719 (time 1709788246.82 total 1105.95 last 35.59) - open/close 31042 (time 1709788256.83 total 1115.95 last 32.28) - open/close 31389 (time 1709788266.84 total 1125.96 last 34.67) - open/close 31729 (time 1709788276.84 total 1135.96 last 34.00) - open/close 32063 (time 1709788286.86 total 1145.98 last 33.34) - open/close 32377 (time 1709788296.88 total 1156.00 last 31.34) - open/close 32715 (time 1709788306.90 total 1166.03 last 33.71) - open/close 33047 (time 1709788316.91 total 1176.03 last 33.18) - open/close 33386 (time 1709788326.92 total 1186.05 last 33.86) - open/close 33732 (time 1709788336.93 total 1196.06 last 34.57) - open/close 34066 (time 1709788346.96 total 1206.08 last 33.32) - open/close 34405 (time 1709788356.96 total 1216.08 last 33.90) - open/close 34709 (time 1709788366.99 total 1226.11 last 30.30) - open/close 35025 (time 1709788377.01 total 1236.13 last 31.55) - open/close 35336 (time 1709788387.03 total 1246.15 last 31.03) - open/close 35663 (time 1709788397.05 total 1256.17 last 32.64) - open/close 35975 (time 1709788407.06 total 1266.18 last 31.17) - open/close 36296 (time 1709788417.08 total 1276.20 last 32.03) - open/close 36616 (time 1709788427.10 total 1286.22 last 31.93) - open/close 36916 (time 1709788437.12 total 1296.25 last 29.93) - open/close 37207 (time 1709788447.15 total 1306.27 last 29.02) - open/close 37512 (time 1709788457.18 total 1316.30 last 30.41) - open/close 37805 (time 1709788467.20 total 1326.32 last 29.24) - open/close 38113 (time 1709788477.23 total 1336.35 last 30.71) - open/close 38401 (time 1709788487.25 total 1346.37 last 28.75) - open/close 38690 (time 1709788497.25 total 1356.38 last 28.88) - open/close 38982 (time 1709788507.26 total 1366.39 last 29.17) - open/close 39278 (time 1709788517.27 total 1376.39 last 29.59) - open/close 39566 (time 1709788527.30 total 1386.42 last 28.71) - open/close 39862 (time 1709788537.30 total 1396.43 last 29.58) - open/close 40000 (time 1709788542.10 total 1401.22 last 28.79) - open/close 40286 (time 1709788552.11 total 1411.23 last 28.57) - open/close 40568 (time 1709788562.14 total 1421.26 last 28.12) - open/close 40864 (time 1709788572.15 total 1431.27 last 29.57) - open/close 41158 (time 1709788582.15 total 1441.28 last 29.38) - open/close 41453 (time 1709788592.18 total 1451.31 last 29.42) - open/close 41747 (time 1709788602.19 total 1461.32 last 29.37) - open/close 42036 (time 1709788612.21 total 1471.33 last 28.85) - open/close 42333 (time 1709788622.23 total 1481.36 last 29.63) - open/close 42626 (time 1709788632.26 total 1491.38 last 29.23) - open/close 42912 (time 1709788642.26 total 1501.39 last 28.58) - open/close 43203 (time 1709788652.26 total 1511.39 last 29.10) - open/close 43487 (time 1709788662.28 total 1521.41 last 28.34) - open/close 43763 (time 1709788672.30 total 1531.42 last 27.57) - open/close 44028 (time 1709788682.32 total 1541.44 last 26.45) - open/close 44291 (time 1709788692.35 total 1551.47 last 26.22) - open/close 44557 (time 1709788702.37 total 1561.49 last 26.55) - open/close 44826 (time 1709788712.38 total 1571.51 last 26.86) - open/close 45091 (time 1709788722.41 total 1581.54 last 26.41) - open/close 45370 (time 1709788732.45 total 1591.57 last 27.80) - open/close 45646 (time 1709788742.48 total 1601.60 last 27.52) - open/close 45918 (time 1709788752.49 total 1611.62 last 27.16) - open/close 46172 (time 1709788762.52 total 1621.64 last 25.34) - open/close 46415 (time 1709788772.52 total 1631.65 last 24.29) - open/close 46663 (time 1709788782.56 total 1641.68 last 24.71) - open/close 46908 (time 1709788792.56 total 1651.69 last 24.48) - open/close 47137 (time 1709788802.60 total 1661.72 last 22.82) - open/close 47386 (time 1709788812.64 total 1671.76 last 24.80) - open/close 47622 (time 1709788822.68 total 1681.80 last 23.51) - open/close 47865 (time 1709788832.68 total 1691.81 last 24.28) - open/close 48101 (time 1709788842.69 total 1701.82 last 23.57) - open/close 48335 (time 1709788852.70 total 1711.82 last 23.40) - open/close 48568 (time 1709788862.70 total 1721.82 last 23.30) - open/close 48803 (time 1709788872.72 total 1731.85 last 23.44) - open/close 49031 (time 1709788882.73 total 1741.85 last 22.78) - open/close 49281 (time 1709788892.74 total 1751.87 last 24.97) - open/close 49525 (time 1709788902.76 total 1761.88 last 24.37) - open/close 49769 (time 1709788912.78 total 1771.90 last 24.35) total: 50000 open/close in 1781.29 seconds: 28.07 ops/second - unlinked 0 (time 1709788923 ; total 0 ; last 0) - unlinked 10000 (time 1709789422 ; total 499 ; last 499) - unlinked 20000 (time 1709789864 ; total 941 ; last 442) - unlinked 30000 (time 1709790181 ; total 1258 ; last 317) - unlinked 40000 (time 1709790334 ; total 1411 ; last 153) total: 50000 unlinks in 1457 seconds: 34.317089 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server unloading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2