== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:52 (1709787172) start mds service on oleg215-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg215-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock oleg215-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 119 (time 1709787203.77 total 10.60 last 11.23) - open/close 373 (time 1709787214.15 total 20.98 last 24.48) - open/close 539 (time 1709787224.17 total 31.00 last 16.57) - open/close 805 (time 1709787234.42 total 41.25 last 25.94) - open/close 1026 (time 1709787244.44 total 51.27 last 22.06) - open/close 1146 (time 1709787254.46 total 61.29 last 11.98) - open/close 1319 (time 1709787264.47 total 71.30 last 17.28) - open/close 1504 (time 1709787275.36 total 82.19 last 16.99) - open/close 1726 (time 1709787285.52 total 92.35 last 21.86) - open/close 1857 (time 1709787297.44 total 104.27 last 10.99) - open/close 2059 (time 1709787308.24 total 115.07 last 18.71) - open/close 2259 (time 1709787318.25 total 125.07 last 19.99) - open/close 2449 (time 1709787328.27 total 135.09 last 18.96) - open/close 2660 (time 1709787338.94 total 145.77 last 19.77) - open/close 2800 (time 1709787349.27 total 156.10 last 13.55) - open/close 2949 (time 1709787359.40 total 166.23 last 14.71) - open/close 3064 (time 1709787369.43 total 176.26 last 11.46) - open/close 3199 (time 1709787379.76 total 186.59 last 13.07) - open/close 3372 (time 1709787390.55 total 197.38 last 16.03) - open/close 3506 (time 1709787400.58 total 207.41 last 13.36) - open/close 3762 (time 1709787411.21 total 218.03 last 24.09) - open/close 3976 (time 1709787421.85 total 228.68 last 20.11) - open/close 4170 (time 1709787431.87 total 238.69 last 19.36) - open/close 4250 (time 1709787441.88 total 248.71 last 7.99) - open/close 4480 (time 1709787452.10 total 258.93 last 22.49) - open/close 4644 (time 1709787462.17 total 269.00 last 16.29) - open/close 4853 (time 1709787473.00 total 279.82 last 19.30) - open/close 5110 (time 1709787483.76 total 290.58 last 23.89) - open/close 5360 (time 1709787494.62 total 301.45 last 23.02) - open/close 5539 (time 1709787504.63 total 311.45 last 17.88) - open/close 5746 (time 1709787514.64 total 321.46 last 20.68) - open/close 5883 (time 1709787524.65 total 331.48 last 13.68) - open/close 6098 (time 1709787535.01 total 341.84 last 20.74) - open/close 6285 (time 1709787545.97 total 352.80 last 17.07) - open/close 6529 (time 1709787556.42 total 363.25 last 23.35) - open/close 6707 (time 1709787566.97 total 373.79 last 16.88) - open/close 6849 (time 1709787576.97 total 383.80 last 14.19) - open/close 6989 (time 1709787586.97 total 393.80 last 14.00) - open/close 7176 (time 1709787597.19 total 404.01 last 18.31) - open/close 7308 (time 1709787607.20 total 414.02 last 13.19) - open/close 7525 (time 1709787617.20 total 424.03 last 21.69) - open/close 7689 (time 1709787627.22 total 434.04 last 16.38) - open/close 7832 (time 1709787637.26 total 444.09 last 14.23) - open/close 8002 (time 1709787648.24 total 455.07 last 15.49) - open/close 8239 (time 1709787658.27 total 465.09 last 23.65) - open/close 8454 (time 1709787668.27 total 475.09 last 21.50) - open/close 8648 (time 1709787678.54 total 485.37 last 18.88) - open/close 8843 (time 1709787688.69 total 495.52 last 19.21) - open/close 9075 (time 1709787698.70 total 505.53 last 23.18) - open/close 9250 (time 1709787709.56 total 516.39 last 16.11) - open/close 9424 (time 1709787719.57 total 526.39 last 17.39) - open/close 9615 (time 1709787729.58 total 536.41 last 19.07) - open/close 9810 (time 1709787740.61 total 547.44 last 17.68) - open/close 9997 (time 1709787751.35 total 558.18 last 17.41) - open/close 10118 (time 1709787762.25 total 569.08 last 11.10) - open/close 10260 (time 1709787772.68 total 579.51 last 13.62) - open/close 10473 (time 1709787785.34 total 592.17 last 16.82) - open/close 10909 (time 1709787795.36 total 602.19 last 43.52) - open/close 11448 (time 1709787805.36 total 612.19 last 53.90) - open/close 11957 (time 1709787815.37 total 622.20 last 50.87) - open/close 12410 (time 1709787825.38 total 632.21 last 45.25) - open/close 12937 (time 1709787835.39 total 642.21 last 52.67) - open/close 13390 (time 1709787845.41 total 652.23 last 45.20) - open/close 13794 (time 1709787855.41 total 662.24 last 40.39) - open/close 14189 (time 1709787865.42 total 672.25 last 39.45) - open/close 14624 (time 1709787875.43 total 682.25 last 43.47) - open/close 15034 (time 1709787885.44 total 692.26 last 40.96) - open/close 15429 (time 1709787895.44 total 702.27 last 39.49) - open/close 15811 (time 1709787905.47 total 712.30 last 38.07) - open/close 16223 (time 1709787915.48 total 722.30 last 41.18) - open/close 16620 (time 1709787925.48 total 732.31 last 39.68) - open/close 17039 (time 1709787935.49 total 742.32 last 41.86) - open/close 17471 (time 1709787945.53 total 752.35 last 43.04) - open/close 17939 (time 1709787955.53 total 762.36 last 46.78) - open/close 18152 (time 1709787971.15 total 777.98 last 13.63) - open/close 18608 (time 1709787981.17 total 787.99 last 45.55) - open/close 19065 (time 1709787991.17 total 798.00 last 45.68) - open/close 19521 (time 1709788001.19 total 808.02 last 45.51) - open/close 20000 (time 1709788011.00 total 817.83 last 48.83) - open/close 20533 (time 1709788021.01 total 827.84 last 53.25) - open/close 21077 (time 1709788031.02 total 837.85 last 54.35) - open/close 21619 (time 1709788041.02 total 847.85 last 54.18) - open/close 22128 (time 1709788051.03 total 857.86 last 50.87) - open/close 22652 (time 1709788061.04 total 867.87 last 52.32) - open/close 23129 (time 1709788071.05 total 877.88 last 47.67) - open/close 23589 (time 1709788081.06 total 887.89 last 45.95) - open/close 24071 (time 1709788091.08 total 897.90 last 48.13) - open/close 24526 (time 1709788101.10 total 907.93 last 45.39) - open/close 24964 (time 1709788111.10 total 917.93 last 43.78) - open/close 25411 (time 1709788121.12 total 927.95 last 44.63) - open/close 25857 (time 1709788131.12 total 937.95 last 44.60) - open/close 26285 (time 1709788141.14 total 947.97 last 42.71) - open/close 26720 (time 1709788151.15 total 957.98 last 43.46) - open/close 27162 (time 1709788161.17 total 968.00 last 44.10) - open/close 27598 (time 1709788171.18 total 978.00 last 43.59) - open/close 28030 (time 1709788181.20 total 988.02 last 43.12) - open/close 28435 (time 1709788191.20 total 998.02 last 40.49) - open/close 28828 (time 1709788201.22 total 1008.05 last 39.22) - open/close 29232 (time 1709788211.22 total 1018.05 last 40.38) - open/close 29648 (time 1709788221.23 total 1028.06 last 41.56) - open/close 30000 (time 1709788230.00 total 1036.82 last 40.17) - open/close 30384 (time 1709788240.01 total 1046.83 last 38.36) - open/close 30767 (time 1709788250.02 total 1056.85 last 38.23) - open/close 31141 (time 1709788260.05 total 1066.87 last 37.32) - open/close 31518 (time 1709788270.07 total 1076.90 last 37.61) - open/close 31895 (time 1709788280.08 total 1086.90 last 37.69) - open/close 32228 (time 1709788290.09 total 1096.92 last 33.25) - open/close 32567 (time 1709788300.10 total 1106.93 last 33.86) - open/close 32906 (time 1709788310.11 total 1116.93 last 33.89) - open/close 33224 (time 1709788320.12 total 1126.95 last 31.76) - open/close 33550 (time 1709788330.14 total 1136.97 last 32.52) - open/close 33860 (time 1709788340.17 total 1146.99 last 30.93) - open/close 34176 (time 1709788350.19 total 1157.02 last 31.53) - open/close 34496 (time 1709788360.21 total 1167.03 last 31.95) - open/close 34814 (time 1709788370.23 total 1177.06 last 31.71) - open/close 35121 (time 1709788380.25 total 1187.08 last 30.65) - open/close 35430 (time 1709788390.27 total 1197.10 last 30.85) - open/close 35730 (time 1709788400.29 total 1207.11 last 29.95) - open/close 36058 (time 1709788410.30 total 1217.12 last 32.76) - open/close 36391 (time 1709788420.31 total 1227.14 last 33.26) - open/close 36728 (time 1709788430.33 total 1237.15 last 33.64) - open/close 37076 (time 1709788440.36 total 1247.19 last 34.69) - open/close 37412 (time 1709788450.36 total 1257.19 last 33.59) - open/close 37730 (time 1709788460.39 total 1267.22 last 31.71) - open/close 38039 (time 1709788470.42 total 1277.25 last 30.80) - open/close 38365 (time 1709788480.45 total 1287.28 last 32.51) - open/close 38706 (time 1709788490.47 total 1297.30 last 34.03) - open/close 39045 (time 1709788500.48 total 1307.30 last 33.88) - open/close 39393 (time 1709788510.50 total 1317.33 last 34.72) - open/close 39731 (time 1709788520.50 total 1327.33 last 33.79) - open/close 40000 (time 1709788528.79 total 1335.61 last 32.47) - open/close 40334 (time 1709788538.81 total 1345.64 last 33.33) - open/close 40673 (time 1709788548.82 total 1355.64 last 33.87) - open/close 40999 (time 1709788558.84 total 1365.67 last 32.52) - open/close 41319 (time 1709788568.86 total 1375.68 last 31.96) - open/close 41637 (time 1709788578.89 total 1385.71 last 31.71) - open/close 41959 (time 1709788588.90 total 1395.73 last 32.14) - open/close 42278 (time 1709788598.93 total 1405.76 last 31.82) - open/close 42598 (time 1709788608.96 total 1415.78 last 31.91) - open/close 42919 (time 1709788618.97 total 1425.79 last 32.07) - open/close 43237 (time 1709788628.98 total 1435.80 last 31.77) - open/close 43539 (time 1709788638.98 total 1445.80 last 30.19) - open/close 43826 (time 1709788649.02 total 1455.84 last 28.59) - open/close 44112 (time 1709788659.03 total 1465.86 last 28.55) - open/close 44418 (time 1709788669.05 total 1475.88 last 30.53) - open/close 44705 (time 1709788679.06 total 1485.89 last 28.67) - open/close 45000 (time 1709788689.07 total 1495.90 last 29.48) - open/close 45293 (time 1709788699.08 total 1505.91 last 29.25) - open/close 45590 (time 1709788709.12 total 1515.94 last 29.61) - open/close 45889 (time 1709788719.13 total 1525.95 last 29.87) - open/close 46186 (time 1709788729.13 total 1535.96 last 29.70) - open/close 46479 (time 1709788739.14 total 1545.96 last 29.28) - open/close 46775 (time 1709788749.14 total 1555.97 last 29.59) - open/close 47067 (time 1709788759.15 total 1565.98 last 29.17) - open/close 47361 (time 1709788769.16 total 1575.99 last 29.37) - open/close 47646 (time 1709788779.19 total 1586.02 last 28.42) - open/close 47927 (time 1709788789.20 total 1596.03 last 28.06) - open/close 48202 (time 1709788799.22 total 1606.05 last 27.45) - open/close 48492 (time 1709788809.24 total 1616.06 last 28.95) - open/close 48774 (time 1709788819.27 total 1626.09 last 28.11) - open/close 49047 (time 1709788829.30 total 1636.12 last 27.22) - open/close 49319 (time 1709788839.32 total 1646.15 last 27.14) - open/close 49596 (time 1709788849.34 total 1656.17 last 27.64) - open/close 49875 (time 1709788859.36 total 1666.18 last 27.85) total: 50000 open/close in 1670.73 seconds: 29.93 ops/second - unlinked 0 (time 1709788864 ; total 0 ; last 0) - unlinked 10000 (time 1709789294 ; total 430 ; last 430) - unlinked 20000 (time 1709789650 ; total 786 ; last 356) - unlinked 30000 (time 1709789898 ; total 1034 ; last 248) - unlinked 40000 (time 1709790030 ; total 1166 ; last 132) total: 50000 unlinks in 1212 seconds: 41.254124 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2