== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:00 (1709787240) start mds service on oleg223-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock oleg223-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 141 (time 1709787276.69 total 10.01 last 14.08) - open/close 354 (time 1709787287.42 total 20.75 last 19.85) - open/close 626 (time 1709787297.64 total 30.96 last 26.62) - open/close 755 (time 1709787307.69 total 41.01 last 12.83) - open/close 883 (time 1709787317.80 total 51.12 last 12.66) - open/close 1033 (time 1709787327.83 total 61.16 last 14.95) - open/close 1197 (time 1709787337.85 total 71.18 last 16.37) - open/close 1400 (time 1709787347.87 total 81.20 last 20.26) - open/close 1650 (time 1709787357.98 total 91.31 last 24.72) - open/close 1772 (time 1709787367.99 total 101.31 last 12.20) - open/close 2004 (time 1709787378.60 total 111.92 last 21.86) - open/close 2296 (time 1709787389.34 total 122.66 last 27.19) - open/close 2525 (time 1709787399.37 total 132.70 last 22.83) - open/close 2768 (time 1709787410.33 total 143.66 last 22.17) - open/close 2960 (time 1709787420.51 total 153.83 last 18.87) - open/close 3105 (time 1709787430.91 total 164.24 last 13.94) - open/close 3258 (time 1709787441.93 total 175.26 last 13.88) - open/close 3427 (time 1709787451.95 total 185.28 last 16.87) - open/close 3702 (time 1709787461.98 total 195.31 last 27.42) - open/close 3930 (time 1709787472.67 total 205.99 last 21.33) - open/close 4035 (time 1709787483.44 total 216.77 last 9.74) - open/close 4223 (time 1709787493.70 total 227.03 last 18.32) - open/close 4432 (time 1709787505.17 total 238.49 last 18.23) - open/close 4671 (time 1709787515.17 total 248.50 last 23.89) - open/close 4841 (time 1709787525.49 total 258.82 last 16.47) - open/close 5073 (time 1709787535.50 total 268.82 last 23.19) - open/close 5299 (time 1709787545.59 total 278.91 last 22.40) - open/close 5439 (time 1709787555.59 total 288.92 last 13.99) - open/close 5572 (time 1709787566.40 total 299.73 last 12.31) - open/close 5733 (time 1709787576.42 total 309.74 last 16.08) - open/close 5975 (time 1709787587.17 total 320.50 last 22.50) - open/close 6179 (time 1709787597.19 total 330.52 last 20.36) - open/close 6339 (time 1709787610.24 total 343.57 last 12.26) - open/close 6589 (time 1709787620.28 total 353.61 last 24.89) - open/close 6775 (time 1709787631.31 total 364.64 last 16.87) - open/close 6908 (time 1709787641.34 total 374.67 last 13.26) - open/close 7053 (time 1709787651.82 total 385.14 last 13.84) - open/close 7158 (time 1709787663.18 total 396.50 last 9.24) - open/close 7287 (time 1709787673.99 total 407.31 last 11.93) - open/close 7473 (time 1709787684.04 total 417.37 last 18.50) - open/close 7669 (time 1709787694.44 total 427.77 last 18.84) - open/close 7899 (time 1709787705.18 total 438.50 last 21.43) - open/close 7992 (time 1709787715.19 total 448.51 last 9.29) - open/close 8197 (time 1709787726.01 total 459.34 last 18.94) - open/close 8376 (time 1709787736.02 total 469.35 last 17.88) - open/close 8505 (time 1709787746.34 total 479.67 last 12.50) - open/close 8729 (time 1709787756.82 total 490.14 last 21.38) - open/close 8886 (time 1709787766.91 total 500.24 last 15.55) - open/close 9028 (time 1709787776.93 total 510.26 last 14.17) - open/close 9176 (time 1709787787.93 total 521.26 last 13.46) - open/close 9398 (time 1709787798.61 total 531.93 last 20.80) - open/close 9567 (time 1709787808.75 total 542.08 last 16.66) - open/close 9698 (time 1709787819.77 total 553.10 last 11.88) - open/close 9826 (time 1709787829.79 total 563.11 last 12.78) - open/close 10000 (time 1709787837.40 total 570.73 last 22.85) - open/close 10226 (time 1709787848.40 total 581.72 last 20.56) - open/close 10428 (time 1709787858.40 total 591.73 last 20.19) - open/close 10896 (time 1709787868.44 total 601.77 last 46.63) - open/close 11404 (time 1709787878.45 total 611.78 last 50.75) - open/close 11829 (time 1709787888.45 total 621.78 last 42.49) - open/close 12326 (time 1709787898.46 total 631.79 last 49.65) - open/close 12721 (time 1709787917.67 total 651.00 last 20.56) - open/close 13099 (time 1709787927.69 total 661.02 last 37.72) - open/close 13590 (time 1709787937.70 total 671.03 last 49.05) - open/close 14107 (time 1709787947.71 total 681.03 last 51.68) - open/close 14647 (time 1709787957.72 total 691.04 last 53.95) - open/close 15200 (time 1709787967.73 total 701.05 last 55.25) - open/close 15718 (time 1709787977.74 total 711.06 last 51.73) - open/close 16230 (time 1709787987.74 total 721.07 last 51.20) - open/close 16733 (time 1709787997.75 total 731.07 last 50.25) - open/close 17272 (time 1709788007.75 total 741.08 last 53.89) - open/close 17848 (time 1709788017.75 total 751.08 last 57.58) - open/close 18459 (time 1709788027.77 total 761.09 last 61.03) - open/close 19059 (time 1709788037.77 total 771.10 last 59.96) - open/close 19641 (time 1709788047.78 total 781.10 last 58.18) - open/close 20000 (time 1709788054.00 total 787.32 last 57.70) - open/close 20548 (time 1709788064.02 total 797.34 last 54.70) - open/close 21063 (time 1709788074.02 total 807.34 last 51.49) - open/close 21569 (time 1709788084.02 total 817.35 last 50.58) - open/close 22084 (time 1709788094.04 total 827.36 last 51.44) - open/close 22572 (time 1709788104.04 total 837.36 last 48.79) - open/close 23042 (time 1709788114.04 total 847.37 last 46.99) - open/close 23529 (time 1709788124.06 total 857.38 last 48.61) - open/close 24000 (time 1709788134.07 total 867.40 last 47.04) - open/close 24465 (time 1709788144.08 total 877.41 last 46.44) - open/close 24924 (time 1709788154.10 total 887.43 last 45.82) - open/close 25385 (time 1709788164.10 total 897.43 last 46.09) - open/close 25861 (time 1709788174.12 total 907.44 last 47.55) - open/close 26315 (time 1709788184.12 total 917.44 last 45.38) - open/close 26749 (time 1709788194.13 total 927.46 last 43.34) - open/close 27157 (time 1709788204.14 total 937.46 last 40.79) - open/close 27592 (time 1709788214.14 total 947.46 last 43.49) - open/close 28027 (time 1709788224.15 total 957.47 last 43.45) - open/close 28444 (time 1709788234.16 total 967.49 last 41.64) - open/close 28852 (time 1709788244.18 total 977.51 last 40.73) - open/close 29256 (time 1709788254.20 total 987.53 last 40.33) - open/close 29645 (time 1709788264.21 total 997.53 last 38.87) - open/close 30000 (time 1709788273.13 total 1006.46 last 39.77) - open/close 30383 (time 1709788283.14 total 1016.47 last 38.28) - open/close 30735 (time 1709788293.15 total 1026.47 last 35.17) - open/close 31089 (time 1709788303.17 total 1036.50 last 35.32) - open/close 31439 (time 1709788313.20 total 1046.52 last 34.90) - open/close 31781 (time 1709788323.22 total 1056.55 last 34.12) - open/close 32114 (time 1709788333.22 total 1066.55 last 33.29) - open/close 32435 (time 1709788343.23 total 1076.55 last 32.10) - open/close 32769 (time 1709788353.25 total 1086.57 last 33.33) - open/close 33099 (time 1709788363.26 total 1096.59 last 32.94) - open/close 33422 (time 1709788373.27 total 1106.59 last 32.29) - open/close 33747 (time 1709788383.28 total 1116.61 last 32.45) - open/close 34056 (time 1709788393.28 total 1126.61 last 30.90) - open/close 34374 (time 1709788403.31 total 1136.64 last 31.71) - open/close 34724 (time 1709788413.32 total 1146.65 last 34.97) - open/close 35063 (time 1709788423.33 total 1156.66 last 33.86) - open/close 35416 (time 1709788433.34 total 1166.66 last 35.28) - open/close 35781 (time 1709788443.34 total 1176.66 last 36.50) - open/close 36131 (time 1709788453.36 total 1186.69 last 34.93) - open/close 36440 (time 1709788463.38 total 1196.70 last 30.84) - open/close 36775 (time 1709788473.40 total 1206.73 last 33.42) - open/close 37117 (time 1709788483.41 total 1216.73 last 34.18) - open/close 37461 (time 1709788493.42 total 1226.74 last 34.36) - open/close 37819 (time 1709788503.44 total 1236.77 last 35.71) - open/close 38175 (time 1709788513.45 total 1246.77 last 35.60) - open/close 38521 (time 1709788523.46 total 1256.79 last 34.54) - open/close 38859 (time 1709788533.48 total 1266.80 last 33.75) - open/close 39205 (time 1709788543.48 total 1276.81 last 34.59) - open/close 39555 (time 1709788553.50 total 1286.83 last 34.93) - open/close 39887 (time 1709788563.52 total 1296.85 last 33.12) - open/close 40000 (time 1709788566.89 total 1300.22 last 33.52) - open/close 40324 (time 1709788576.91 total 1310.23 last 32.36) - open/close 40654 (time 1709788586.91 total 1320.23 last 32.99) - open/close 40990 (time 1709788596.91 total 1330.24 last 33.59) - open/close 41316 (time 1709788606.92 total 1340.24 last 32.59) - open/close 41645 (time 1709788616.93 total 1350.26 last 32.85) - open/close 41971 (time 1709788626.94 total 1360.26 last 32.58) - open/close 42284 (time 1709788636.95 total 1370.27 last 31.27) - open/close 42580 (time 1709788646.95 total 1380.28 last 29.60) - open/close 42868 (time 1709788656.95 total 1390.28 last 28.79) - open/close 43184 (time 1709788666.97 total 1400.29 last 31.56) - open/close 43479 (time 1709788676.99 total 1410.31 last 29.44) - open/close 43781 (time 1709788686.99 total 1420.32 last 30.19) - open/close 44081 (time 1709788697.01 total 1430.34 last 29.94) - open/close 44391 (time 1709788707.04 total 1440.37 last 30.92) - open/close 44695 (time 1709788717.05 total 1450.38 last 30.36) - open/close 45012 (time 1709788727.07 total 1460.40 last 31.64) - open/close 45313 (time 1709788737.09 total 1470.42 last 30.04) - open/close 45627 (time 1709788747.09 total 1480.42 last 31.40) - open/close 45931 (time 1709788757.11 total 1490.44 last 30.35) - open/close 46236 (time 1709788767.12 total 1500.45 last 30.47) - open/close 46533 (time 1709788777.15 total 1510.47 last 29.63) - open/close 46825 (time 1709788787.17 total 1520.50 last 29.13) - open/close 47112 (time 1709788797.18 total 1530.51 last 28.67) - open/close 47408 (time 1709788807.20 total 1540.53 last 29.54) - open/close 47703 (time 1709788817.23 total 1550.56 last 29.42) - open/close 47983 (time 1709788827.24 total 1560.57 last 27.97) - open/close 48262 (time 1709788837.26 total 1570.59 last 27.84) - open/close 48543 (time 1709788847.28 total 1580.61 last 28.04) - open/close 48830 (time 1709788857.29 total 1590.62 last 28.68) - open/close 49107 (time 1709788867.33 total 1600.65 last 27.60) - open/close 49390 (time 1709788877.36 total 1610.69 last 28.21) - open/close 49675 (time 1709788887.39 total 1620.72 last 28.40) - open/close 49949 (time 1709788897.43 total 1630.76 last 27.30) total: 50000 open/close in 1632.61 seconds: 30.63 ops/second - unlinked 0 (time 1709788900 ; total 0 ; last 0) - unlinked 10000 (time 1709789340 ; total 440 ; last 440) - unlinked 20000 (time 1709789705 ; total 805 ; last 365) - unlinked 30000 (time 1709789951 ; total 1051 ; last 246) - unlinked 40000 (time 1709790068 ; total 1168 ; last 117) total: 50000 unlinks in 1206 seconds: 41.459370 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server unloading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2