== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:59:06 (1709787546) start mds service on oleg225-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg225-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg225-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock oleg225-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1709787600.19 total 10.13 last 20.43) - open/close 391 (time 1709787610.20 total 20.14 last 18.39) - open/close 632 (time 1709787620.98 total 30.92 last 22.35) - open/close 830 (time 1709787631.30 total 41.24 last 19.19) - open/close 1064 (time 1709787641.88 total 51.82 last 22.13) - open/close 1322 (time 1709787652.43 total 62.37 last 24.45) - open/close 1494 (time 1709787662.44 total 72.38 last 17.19) - open/close 1672 (time 1709787672.44 total 82.38 last 17.80) - open/close 1803 (time 1709787682.84 total 92.78 last 12.60) - open/close 2037 (time 1709787692.85 total 102.79 last 23.38) - open/close 2285 (time 1709787703.11 total 113.05 last 24.17) - open/close 2422 (time 1709787714.46 total 124.40 last 12.07) - open/close 2608 (time 1709787725.04 total 134.98 last 17.58) - open/close 2744 (time 1709787735.75 total 145.69 last 12.70) - open/close 2953 (time 1709787745.77 total 155.71 last 20.87) - open/close 3216 (time 1709787756.76 total 166.70 last 23.93) - open/close 3388 (time 1709787767.09 total 177.03 last 16.65) - open/close 3495 (time 1709787777.10 total 187.04 last 10.69) - open/close 3636 (time 1709787787.56 total 197.50 last 13.48) - open/close 3779 (time 1709787798.41 total 208.35 last 13.17) - open/close 3878 (time 1709787809.15 total 219.09 last 9.22) - open/close 3971 (time 1709787819.15 total 229.09 last 9.29) - open/close 4143 (time 1709787829.23 total 239.17 last 17.07) - open/close 4383 (time 1709787840.01 total 249.95 last 22.27) - open/close 4525 (time 1709787850.02 total 259.96 last 14.19) - open/close 4737 (time 1709787860.03 total 269.97 last 21.16) - open/close 4887 (time 1709787870.40 total 280.34 last 14.47) - open/close 5002 (time 1709787880.47 total 290.41 last 11.42) - open/close 5148 (time 1709787890.48 total 300.42 last 14.59) - open/close 5359 (time 1709787900.48 total 310.42 last 21.10) - open/close 5489 (time 1709787910.50 total 320.44 last 12.97) - open/close 5697 (time 1709787920.57 total 330.51 last 20.66) - open/close 5916 (time 1709787931.29 total 341.23 last 20.42) - open/close 6075 (time 1709787942.20 total 352.14 last 14.58) - open/close 6257 (time 1709787952.20 total 362.14 last 18.20) - open/close 6485 (time 1709787962.59 total 372.53 last 21.95) - open/close 6819 (time 1709787972.59 total 382.53 last 33.39) - open/close 7060 (time 1709787982.61 total 392.55 last 24.06) - open/close 7333 (time 1709787992.90 total 402.84 last 26.53) - open/close 7524 (time 1709788003.57 total 413.51 last 17.89) - open/close 7673 (time 1709788013.57 total 423.51 last 14.90) - open/close 7938 (time 1709788023.71 total 433.65 last 26.14) - open/close 8172 (time 1709788034.30 total 444.24 last 22.09) - open/close 8504 (time 1709788044.90 total 454.84 last 31.33) - open/close 8747 (time 1709788055.69 total 465.63 last 22.52) - open/close 8994 (time 1709788066.64 total 476.58 last 22.57) - open/close 9382 (time 1709788077.65 total 487.59 last 35.22) - open/close 9731 (time 1709788087.66 total 497.60 last 34.86) - open/close 10000 (time 1709788091.26 total 501.20 last 74.87) - open/close 10740 (time 1709788101.27 total 511.21 last 73.91) - open/close 11475 (time 1709788111.27 total 521.21 last 73.49) - open/close 12166 (time 1709788121.28 total 531.22 last 69.05) - open/close 12834 (time 1709788131.28 total 541.22 last 66.77) - open/close 13500 (time 1709788141.29 total 551.23 last 66.54) - open/close 14126 (time 1709788151.29 total 561.23 last 62.58) - open/close 14762 (time 1709788161.31 total 571.25 last 63.51) - open/close 15365 (time 1709788171.32 total 581.26 last 60.22) - open/close 15990 (time 1709788181.33 total 591.27 last 62.43) - open/close 16605 (time 1709788191.34 total 601.28 last 61.43) - open/close 17198 (time 1709788201.35 total 611.29 last 59.28) - open/close 17794 (time 1709788211.36 total 621.30 last 59.54) - open/close 18393 (time 1709788221.36 total 631.30 last 59.90) - open/close 18965 (time 1709788231.37 total 641.31 last 57.13) - open/close 19506 (time 1709788241.38 total 651.32 last 54.04) - open/close 20000 (time 1709788250.35 total 660.29 last 55.07) - open/close 20537 (time 1709788260.37 total 670.31 last 53.61) - open/close 21034 (time 1709788270.38 total 680.32 last 49.67) - open/close 21533 (time 1709788280.39 total 690.33 last 49.83) - open/close 21974 (time 1709788290.40 total 700.34 last 44.06) - open/close 22436 (time 1709788300.42 total 710.36 last 46.12) - open/close 22894 (time 1709788310.42 total 720.36 last 45.80) - open/close 23341 (time 1709788320.42 total 730.36 last 44.67) - open/close 23783 (time 1709788330.43 total 740.37 last 44.18) - open/close 24196 (time 1709788340.43 total 750.37 last 41.29) - open/close 24611 (time 1709788350.43 total 760.37 last 41.48) - open/close 25028 (time 1709788360.44 total 770.38 last 41.69) - open/close 25436 (time 1709788370.44 total 780.38 last 40.80) - open/close 25832 (time 1709788380.46 total 790.40 last 39.51) - open/close 26205 (time 1709788390.46 total 800.41 last 37.28) - open/close 26606 (time 1709788400.48 total 810.42 last 40.04) - open/close 27026 (time 1709788410.50 total 820.44 last 41.93) - open/close 27429 (time 1709788420.51 total 830.45 last 40.26) - open/close 27842 (time 1709788430.51 total 840.45 last 41.29) - open/close 28250 (time 1709788440.52 total 850.46 last 40.74) - open/close 28657 (time 1709788450.54 total 860.48 last 40.66) - open/close 29058 (time 1709788460.54 total 870.48 last 40.07) - open/close 29446 (time 1709788470.56 total 880.50 last 38.72) - open/close 29858 (time 1709788480.58 total 890.52 last 41.13) - open/close 30000 (time 1709788484.11 total 894.05 last 40.20) - open/close 30391 (time 1709788494.13 total 904.07 last 39.02) - open/close 30793 (time 1709788504.14 total 914.08 last 40.18) - open/close 31195 (time 1709788514.16 total 924.10 last 40.13) - open/close 31591 (time 1709788524.17 total 934.11 last 39.53) - open/close 31972 (time 1709788534.19 total 944.13 last 38.04) - open/close 32349 (time 1709788544.21 total 954.15 last 37.61) - open/close 32729 (time 1709788554.22 total 964.16 last 37.95) - open/close 33105 (time 1709788564.24 total 974.18 last 37.54) - open/close 33481 (time 1709788574.25 total 984.19 last 37.55) - open/close 33863 (time 1709788584.28 total 994.22 last 38.11) - open/close 34237 (time 1709788594.29 total 1004.23 last 37.35) - open/close 34614 (time 1709788604.29 total 1014.23 last 37.69) - open/close 34992 (time 1709788614.30 total 1024.24 last 37.75) - open/close 35358 (time 1709788624.31 total 1034.25 last 36.60) - open/close 35706 (time 1709788634.31 total 1044.25 last 34.79) - open/close 36036 (time 1709788644.32 total 1054.26 last 32.98) - open/close 36379 (time 1709788654.33 total 1064.27 last 34.26) - open/close 36728 (time 1709788664.34 total 1074.28 last 34.87) - open/close 37071 (time 1709788674.35 total 1084.29 last 34.23) - open/close 37406 (time 1709788684.37 total 1094.31 last 33.46) - open/close 37740 (time 1709788694.38 total 1104.33 last 33.34) - open/close 38078 (time 1709788704.40 total 1114.34 last 33.75) - open/close 38405 (time 1709788714.42 total 1124.36 last 32.63) - open/close 38746 (time 1709788724.43 total 1134.37 last 34.08) - open/close 39077 (time 1709788734.45 total 1144.39 last 33.03) - open/close 39392 (time 1709788744.46 total 1154.40 last 31.47) - open/close 39708 (time 1709788754.47 total 1164.41 last 31.58) - open/close 40000 (time 1709788763.63 total 1173.57 last 31.87) - open/close 40316 (time 1709788773.65 total 1183.59 last 31.55) - open/close 40617 (time 1709788783.65 total 1193.59 last 30.07) - open/close 40922 (time 1709788793.66 total 1203.60 last 30.48) - open/close 41235 (time 1709788803.69 total 1213.63 last 31.21) - open/close 41541 (time 1709788813.69 total 1223.63 last 30.58) - open/close 41836 (time 1709788823.73 total 1233.67 last 29.40) - open/close 42128 (time 1709788833.74 total 1243.68 last 29.16) - open/close 42429 (time 1709788843.76 total 1253.70 last 30.05) - open/close 42721 (time 1709788853.78 total 1263.72 last 29.15) - open/close 43000 (time 1709788863.78 total 1273.72 last 27.88) - open/close 43267 (time 1709788873.82 total 1283.76 last 26.61) - open/close 43538 (time 1709788883.85 total 1293.79 last 27.01) - open/close 43793 (time 1709788893.86 total 1303.80 last 25.48) - open/close 44050 (time 1709788903.87 total 1313.81 last 25.67) - open/close 44299 (time 1709788913.90 total 1323.84 last 24.84) - open/close 44556 (time 1709788923.91 total 1333.85 last 25.66) - open/close 44822 (time 1709788933.93 total 1343.87 last 26.55) - open/close 45089 (time 1709788943.94 total 1353.88 last 26.68) - open/close 45349 (time 1709788953.95 total 1363.89 last 25.97) - open/close 45615 (time 1709788963.97 total 1373.91 last 26.55) - open/close 45877 (time 1709788974.01 total 1383.95 last 26.11) - open/close 46133 (time 1709788984.02 total 1393.96 last 25.57) - open/close 46388 (time 1709788994.05 total 1403.99 last 25.42) - open/close 46647 (time 1709789004.06 total 1414.00 last 25.86) - open/close 46899 (time 1709789014.07 total 1424.01 last 25.18) - open/close 47155 (time 1709789024.09 total 1434.03 last 25.55) - open/close 47411 (time 1709789034.12 total 1444.06 last 25.52) - open/close 47663 (time 1709789044.14 total 1454.08 last 25.15) - open/close 47912 (time 1709789054.17 total 1464.11 last 24.83) - open/close 48160 (time 1709789064.22 total 1474.16 last 24.69) - open/close 48411 (time 1709789074.25 total 1484.19 last 25.02) - open/close 48658 (time 1709789084.28 total 1494.22 last 24.62) - open/close 48897 (time 1709789094.31 total 1504.25 last 23.84) - open/close 49134 (time 1709789104.33 total 1514.27 last 23.66) - open/close 49374 (time 1709789114.33 total 1524.27 last 23.99) - open/close 49611 (time 1709789124.35 total 1534.29 last 23.65) - open/close 49844 (time 1709789134.38 total 1544.32 last 23.24) total: 50000 open/close in 1550.75 seconds: 32.24 ops/second - unlinked 0 (time 1709789141 ; total 0 ; last 0) - unlinked 10000 (time 1709789633 ; total 492 ; last 492) - unlinked 20000 (time 1709790003 ; total 862 ; last 370) - unlinked 30000 (time 1709790197 ; total 1056 ; last 194) - unlinked 40000 (time 1709790265 ; total 1124 ; last 68) total: 50000 unlinks in 1154 seconds: 43.327557 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server unloading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2