== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:00 (1709786940) start mds service on oleg252-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg252-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg252-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock oleg252-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 212 (time 1709786988.32 total 10.91 last 19.44) - open/close 355 (time 1709786998.34 total 20.92 last 14.28) - open/close 523 (time 1709787008.76 total 31.34 last 16.12) - open/close 688 (time 1709787019.08 total 41.66 last 15.99) - open/close 970 (time 1709787029.99 total 52.58 last 25.83) - open/close 1128 (time 1709787040.05 total 62.63 last 15.72) - open/close 1259 (time 1709787050.05 total 72.64 last 13.10) - open/close 1385 (time 1709787060.07 total 82.65 last 12.58) - open/close 1568 (time 1709787070.25 total 92.84 last 17.98) - open/close 1715 (time 1709787080.39 total 102.97 last 14.50) - open/close 1905 (time 1709787091.25 total 113.84 last 17.49) - open/close 2070 (time 1709787102.09 total 124.67 last 15.23) - open/close 2344 (time 1709787112.09 total 134.68 last 27.39) - open/close 2363 (time 1709787123.43 total 146.02 last 1.68) - open/close 2602 (time 1709787134.15 total 156.74 last 22.30) - open/close 2759 (time 1709787144.95 total 167.54 last 14.54) - open/close 2954 (time 1709787155.56 total 178.14 last 18.39) - open/close 3231 (time 1709787165.56 total 188.15 last 27.69) - open/close 3359 (time 1709787175.74 total 198.33 last 12.58) - open/close 3567 (time 1709787185.74 total 208.33 last 20.80) - open/close 3717 (time 1709787196.38 total 218.97 last 14.10) - open/close 3790 (time 1709787206.82 total 229.41 last 6.99) - open/close 4138 (time 1709787217.32 total 239.91 last 33.13) - open/close 4375 (time 1709787228.31 total 250.89 last 21.58) - open/close 4442 (time 1709787238.82 total 261.41 last 6.37) - open/close 4651 (time 1709787248.82 total 271.41 last 20.89) - open/close 4869 (time 1709787259.68 total 282.27 last 20.08) - open/close 5033 (time 1709787269.99 total 292.58 last 15.90) - open/close 5139 (time 1709787280.33 total 302.92 last 10.25) - open/close 5269 (time 1709787300.23 total 322.82 last 6.53) - open/close 5355 (time 1709787310.37 total 332.96 last 8.48) - open/close 5457 (time 1709787320.50 total 343.09 last 10.07) - open/close 5604 (time 1709787331.20 total 353.79 last 13.74) - open/close 5803 (time 1709787341.21 total 363.80 last 19.88) - open/close 5931 (time 1709787352.04 total 374.62 last 11.82) - open/close 6198 (time 1709787362.32 total 384.91 last 25.95) - open/close 6367 (time 1709787372.33 total 394.92 last 16.89) - open/close 6503 (time 1709787383.28 total 405.87 last 12.42) - open/close 6655 (time 1709787394.73 total 417.32 last 13.28) - open/close 6917 (time 1709787404.83 total 427.42 last 25.94) - open/close 7132 (time 1709787415.27 total 437.86 last 20.59) - open/close 7424 (time 1709787425.29 total 447.88 last 29.14) - open/close 7615 (time 1709787435.83 total 458.42 last 18.12) - open/close 7820 (time 1709787445.84 total 468.42 last 20.49) - open/close 7999 (time 1709787455.85 total 478.43 last 17.88) - open/close 8164 (time 1709787465.88 total 488.46 last 16.45) - open/close 8344 (time 1709787476.60 total 499.18 last 16.79) - open/close 8443 (time 1709787486.87 total 509.46 last 9.64) - open/close 8610 (time 1709787496.87 total 519.46 last 16.70) - open/close 8818 (time 1709787506.89 total 529.48 last 20.76) - open/close 8957 (time 1709787517.38 total 539.96 last 13.25) - open/close 9099 (time 1709787528.34 total 550.93 last 12.95) - open/close 9276 (time 1709787538.37 total 560.96 last 17.64) - open/close 9449 (time 1709787549.04 total 571.63 last 16.22) - open/close 9547 (time 1709787559.07 total 581.66 last 9.77) - open/close 9776 (time 1709787569.08 total 591.67 last 22.87) - open/close 10000 (time 1709787574.57 total 597.16 last 40.84) - open/close 10460 (time 1709787584.62 total 607.21 last 45.75) - open/close 10876 (time 1709787594.66 total 617.25 last 41.43) - open/close 11251 (time 1709787614.46 total 637.04 last 18.95) - open/close 11663 (time 1709787624.46 total 647.05 last 41.17) - open/close 12059 (time 1709787634.47 total 657.06 last 39.56) - open/close 12515 (time 1709787644.48 total 667.06 last 45.59) - open/close 12912 (time 1709787654.50 total 677.09 last 39.59) - open/close 13287 (time 1709787664.51 total 687.10 last 37.46) - open/close 13642 (time 1709787674.51 total 697.10 last 35.50) - open/close 14043 (time 1709787684.52 total 707.11 last 40.08) - open/close 14399 (time 1709787694.53 total 717.11 last 35.57) - open/close 14737 (time 1709787704.53 total 727.12 last 33.79) - open/close 15031 (time 1709787714.55 total 737.14 last 29.35) - open/close 15328 (time 1709787724.56 total 747.15 last 29.66) - open/close 15643 (time 1709787734.59 total 757.17 last 31.42) - open/close 15959 (time 1709787744.59 total 767.18 last 31.59) - open/close 16279 (time 1709787754.61 total 777.20 last 31.95) - open/close 16596 (time 1709787764.62 total 787.20 last 31.67) - open/close 16941 (time 1709787774.64 total 797.23 last 34.42) - open/close 17260 (time 1709787784.66 total 807.25 last 31.84) - open/close 17570 (time 1709787794.68 total 817.26 last 30.95) - open/close 17899 (time 1709787804.69 total 827.28 last 32.84) - open/close 18202 (time 1709787814.73 total 837.32 last 30.19) - open/close 18506 (time 1709787824.76 total 847.34 last 30.32) - open/close 18820 (time 1709787834.76 total 857.35 last 31.39) - open/close 19105 (time 1709787844.78 total 867.36 last 28.45) - open/close 19356 (time 1709787854.78 total 877.37 last 25.09) - open/close 19579 (time 1709787864.81 total 887.40 last 22.23) - open/close 19874 (time 1709787874.83 total 897.41 last 29.45) - open/close 20000 (time 1709787879.62 total 902.20 last 26.30) - open/close 20256 (time 1709787889.64 total 912.23 last 25.55) - open/close 20498 (time 1709787899.65 total 922.24 last 24.17) - open/close 20753 (time 1709787909.65 total 932.24 last 25.49) - open/close 21006 (time 1709787919.68 total 942.27 last 25.23) - open/close 21278 (time 1709787929.70 total 952.29 last 27.13) - open/close 21558 (time 1709787939.71 total 962.29 last 27.99) - open/close 21869 (time 1709787949.71 total 972.30 last 31.08) - open/close 22217 (time 1709787959.72 total 982.31 last 34.77) - open/close 22550 (time 1709787969.73 total 992.31 last 33.28) - open/close 22566 (time 1709787981.24 total 1003.83 last 1.39) - open/close 22906 (time 1709787991.27 total 1013.85 last 33.93) - open/close 23265 (time 1709788001.28 total 1023.86 last 35.86) - open/close 23641 (time 1709788011.29 total 1033.88 last 37.53) - open/close 24041 (time 1709788021.31 total 1043.90 last 39.93) - open/close 24445 (time 1709788031.33 total 1053.91 last 40.34) - open/close 24870 (time 1709788041.35 total 1063.94 last 42.40) - open/close 25266 (time 1709788051.37 total 1073.96 last 39.52) - open/close 25672 (time 1709788061.38 total 1083.97 last 40.54) - open/close 26047 (time 1709788071.40 total 1093.98 last 37.45) - open/close 26436 (time 1709788081.41 total 1104.00 last 38.85) - open/close 26831 (time 1709788091.42 total 1114.01 last 39.46) - open/close 27202 (time 1709788101.43 total 1124.01 last 37.08) - open/close 27576 (time 1709788111.43 total 1134.02 last 37.38) - open/close 27950 (time 1709788121.45 total 1144.04 last 37.33) - open/close 28313 (time 1709788131.48 total 1154.07 last 36.20) - open/close 28675 (time 1709788141.48 total 1164.07 last 36.20) - open/close 29010 (time 1709788151.50 total 1174.08 last 33.44) - open/close 29352 (time 1709788161.50 total 1184.09 last 34.17) - open/close 29699 (time 1709788171.52 total 1194.11 last 34.65) - open/close 30000 (time 1709788180.08 total 1202.67 last 35.16) - open/close 30325 (time 1709788190.09 total 1212.68 last 32.47) - open/close 30646 (time 1709788200.10 total 1222.68 last 32.08) - open/close 30970 (time 1709788210.11 total 1232.69 last 32.37) - open/close 31300 (time 1709788220.12 total 1242.70 last 32.97) - open/close 31638 (time 1709788230.13 total 1252.71 last 33.77) - open/close 31982 (time 1709788240.14 total 1262.73 last 34.35) - open/close 32326 (time 1709788250.15 total 1272.74 last 34.35) - open/close 32667 (time 1709788260.16 total 1282.75 last 34.08) - open/close 32978 (time 1709788270.19 total 1292.78 last 31.00) - open/close 33306 (time 1709788280.20 total 1302.79 last 32.76) - open/close 33586 (time 1709788290.22 total 1312.81 last 27.94) - open/close 33883 (time 1709788300.24 total 1322.83 last 29.65) - open/close 34168 (time 1709788310.27 total 1332.86 last 28.40) - open/close 34453 (time 1709788320.31 total 1342.90 last 28.40) - open/close 34743 (time 1709788330.33 total 1352.91 last 28.95) - open/close 35024 (time 1709788340.34 total 1362.93 last 28.05) - open/close 35316 (time 1709788350.35 total 1372.94 last 29.18) - open/close 35614 (time 1709788360.36 total 1382.95 last 29.78) - open/close 35911 (time 1709788370.37 total 1392.96 last 29.65) - open/close 36195 (time 1709788380.39 total 1402.97 last 28.37) - open/close 36471 (time 1709788390.41 total 1413.00 last 27.52) - open/close 36760 (time 1709788400.44 total 1423.03 last 28.82) - open/close 37068 (time 1709788410.47 total 1433.06 last 30.72) - open/close 37368 (time 1709788420.48 total 1443.07 last 29.96) - open/close 37677 (time 1709788430.49 total 1453.07 last 30.88) - open/close 37988 (time 1709788440.51 total 1463.10 last 31.03) - open/close 38296 (time 1709788450.52 total 1473.10 last 30.78) - open/close 38588 (time 1709788460.53 total 1483.12 last 29.16) - open/close 38881 (time 1709788470.56 total 1493.15 last 29.22) - open/close 39187 (time 1709788480.58 total 1503.17 last 30.54) - open/close 39496 (time 1709788490.58 total 1513.17 last 30.89) - open/close 39809 (time 1709788500.61 total 1523.19 last 31.22) - open/close 40000 (time 1709788506.73 total 1529.32 last 31.17) - open/close 40307 (time 1709788516.75 total 1539.34 last 30.65) - open/close 40614 (time 1709788526.78 total 1549.37 last 30.61) - open/close 40916 (time 1709788536.79 total 1559.37 last 30.18) - open/close 41226 (time 1709788546.80 total 1569.39 last 30.95) - open/close 41532 (time 1709788556.83 total 1579.42 last 30.51) - open/close 41826 (time 1709788566.86 total 1589.45 last 29.31) - open/close 42119 (time 1709788576.89 total 1599.48 last 29.20) - open/close 42413 (time 1709788586.90 total 1609.49 last 29.38) - open/close 42711 (time 1709788596.91 total 1619.49 last 29.78) - open/close 43012 (time 1709788606.92 total 1629.51 last 30.06) - open/close 43299 (time 1709788616.95 total 1639.54 last 28.60) - open/close 43583 (time 1709788626.96 total 1649.54 last 28.39) - open/close 43857 (time 1709788636.98 total 1659.57 last 27.33) - open/close 44123 (time 1709788647.00 total 1669.58 last 26.57) - open/close 44393 (time 1709788657.02 total 1679.61 last 26.94) - open/close 44678 (time 1709788667.05 total 1689.64 last 28.41) - open/close 44948 (time 1709788677.09 total 1699.68 last 26.89) - open/close 45222 (time 1709788687.10 total 1709.69 last 27.37) - open/close 45494 (time 1709788697.13 total 1719.72 last 27.12) - open/close 45769 (time 1709788707.13 total 1729.72 last 27.48) - open/close 46047 (time 1709788717.15 total 1739.74 last 27.75) - open/close 46330 (time 1709788727.18 total 1749.77 last 28.23) - open/close 46605 (time 1709788737.21 total 1759.79 last 27.42) - open/close 46877 (time 1709788747.22 total 1769.81 last 27.16) - open/close 47141 (time 1709788757.23 total 1779.81 last 26.39) - open/close 47416 (time 1709788767.23 total 1789.81 last 27.49) - open/close 47684 (time 1709788777.23 total 1799.82 last 26.79) - open/close 47948 (time 1709788787.25 total 1809.83 last 26.36) - open/close 48215 (time 1709788797.25 total 1819.84 last 26.68) - open/close 48479 (time 1709788807.27 total 1829.86 last 26.35) - open/close 48744 (time 1709788817.28 total 1839.87 last 26.48) - open/close 49001 (time 1709788827.29 total 1849.88 last 25.67) - open/close 49249 (time 1709788837.32 total 1859.91 last 24.74) - open/close 49503 (time 1709788847.32 total 1869.91 last 25.39) - open/close 49751 (time 1709788857.34 total 1879.93 last 24.76) total: 50000 open/close in 1889.73 seconds: 26.46 ops/second - unlinked 0 (time 1709788868 ; total 0 ; last 0) - unlinked 10000 (time 1709789322 ; total 454 ; last 454) - unlinked 20000 (time 1709789697 ; total 829 ; last 375) - unlinked 30000 (time 1709789957 ; total 1089 ; last 260) - unlinked 40000 (time 1709790079 ; total 1211 ; last 122) total: 50000 unlinks in 1253 seconds: 39.904228 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server unloading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2