== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:57 (1709787057) start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1709787103.88 total 10.35 last 19.81) - open/close 423 (time 1709787114.48 total 20.94 last 20.57) - open/close 691 (time 1709787124.90 total 31.37 last 25.72) - open/close 822 (time 1709787134.92 total 41.38 last 13.08) - open/close 1029 (time 1709787145.72 total 52.19 last 19.16) - open/close 1228 (time 1709787156.74 total 63.20 last 18.06) - open/close 1487 (time 1709787166.74 total 73.21 last 25.89) - open/close 1640 (time 1709787177.19 total 83.65 last 14.65) - open/close 1723 (time 1709787187.42 total 93.89 last 8.11) - open/close 1916 (time 1709787198.40 total 104.87 last 17.58) - open/close 2123 (time 1709787208.42 total 114.88 last 20.67) - open/close 2352 (time 1709787218.42 total 124.89 last 22.88) - open/close 2599 (time 1709787228.54 total 135.00 last 24.42) - open/close 2700 (time 1709787239.20 total 145.67 last 9.47) - open/close 2883 (time 1709787249.41 total 155.87 last 17.93) - open/close 2996 (time 1709787259.70 total 166.16 last 10.98) - open/close 3276 (time 1709787269.71 total 176.18 last 27.96) - open/close 3512 (time 1709787280.48 total 186.95 last 21.92) - open/close 3679 (time 1709787290.56 total 197.03 last 16.57) - open/close 3844 (time 1709787301.29 total 207.76 last 15.37) - open/close 4045 (time 1709787311.46 total 217.93 last 19.77) - open/close 4246 (time 1709787321.48 total 227.95 last 20.07) - open/close 4353 (time 1709787333.25 total 239.72 last 9.09) - open/close 4542 (time 1709787343.26 total 249.72 last 18.89) - open/close 4653 (time 1709787359.56 total 266.03 last 6.81) - open/close 4811 (time 1709787371.18 total 277.64 last 13.61) - open/close 4974 (time 1709787381.92 total 288.38 last 15.18) - open/close 5149 (time 1709787392.60 total 299.06 last 16.38) - open/close 5328 (time 1709787402.61 total 309.08 last 17.87) - open/close 5491 (time 1709787413.09 total 319.56 last 15.55) - open/close 5690 (time 1709787423.10 total 329.57 last 19.88) - open/close 5815 (time 1709787433.64 total 340.10 last 11.87) - open/close 5958 (time 1709787443.64 total 350.11 last 14.30) - open/close 6205 (time 1709787453.64 total 360.11 last 24.69) - open/close 6314 (time 1709787463.93 total 370.40 last 10.59) - open/close 6529 (time 1709787474.12 total 380.59 last 21.11) - open/close 6708 (time 1709787484.14 total 390.61 last 17.86) - open/close 6899 (time 1709787494.17 total 400.63 last 19.05) - open/close 7112 (time 1709787504.39 total 410.86 last 20.83) - open/close 7232 (time 1709787514.80 total 421.26 last 11.54) - open/close 7397 (time 1709787524.82 total 431.28 last 16.47) - open/close 7530 (time 1709787534.86 total 441.32 last 13.24) - open/close 7707 (time 1709787545.61 total 452.08 last 16.46) - open/close 7878 (time 1709787555.62 total 462.09 last 17.09) - open/close 8017 (time 1709787565.63 total 472.09 last 13.89) - open/close 8186 (time 1709787576.14 total 482.61 last 16.07) - open/close 8307 (time 1709787586.15 total 492.62 last 12.09) - open/close 8450 (time 1709787596.17 total 502.63 last 14.27) - open/close 8489 (time 1709787608.28 total 514.74 last 3.22) - open/close 8680 (time 1709787618.32 total 524.78 last 19.02) - open/close 8898 (time 1709787628.41 total 534.88 last 21.60) - open/close 9136 (time 1709787639.42 total 545.89 last 21.61) - open/close 9260 (time 1709787649.43 total 555.90 last 12.38) - open/close 9458 (time 1709787659.46 total 565.92 last 19.75) - open/close 9671 (time 1709787669.54 total 576.00 last 21.13) - open/close 9827 (time 1709787679.55 total 586.01 last 15.59) - open/close 10000 (time 1709787683.71 total 590.17 last 41.61) - open/close 10535 (time 1709787693.72 total 600.18 last 53.45) - open/close 11048 (time 1709787703.73 total 610.20 last 51.23) - open/close 11539 (time 1709787713.74 total 620.21 last 49.04) - open/close 11730 (time 1709787729.54 total 636.01 last 12.09) - open/close 12181 (time 1709787739.54 total 646.01 last 45.09) - open/close 12628 (time 1709787749.58 total 656.05 last 44.52) - open/close 13077 (time 1709787759.60 total 666.07 last 44.81) - open/close 13556 (time 1709787769.63 total 676.09 last 47.79) - open/close 13956 (time 1709787779.64 total 686.11 last 39.94) - open/close 14408 (time 1709787789.66 total 696.12 last 45.13) - open/close 14846 (time 1709787799.67 total 706.13 last 43.77) - open/close 15273 (time 1709787809.67 total 716.14 last 42.67) - open/close 15677 (time 1709787819.68 total 726.15 last 40.37) - open/close 16087 (time 1709787829.69 total 736.16 last 40.96) - open/close 16493 (time 1709787839.70 total 746.17 last 40.56) - open/close 16848 (time 1709787849.77 total 756.24 last 35.25) - open/close 17179 (time 1709787859.79 total 766.25 last 33.05) - open/close 17541 (time 1709787869.80 total 776.26 last 36.15) - open/close 17890 (time 1709787879.81 total 786.28 last 34.84) - open/close 18240 (time 1709787889.83 total 796.29 last 34.95) - open/close 18575 (time 1709787899.84 total 806.31 last 33.45) - open/close 18922 (time 1709787909.85 total 816.32 last 34.66) - open/close 19299 (time 1709787919.89 total 826.35 last 37.59) - open/close 19617 (time 1709787929.91 total 836.37 last 31.73) - open/close 20000 (time 1709787939.65 total 846.12 last 39.30) - open/close 20393 (time 1709787949.67 total 856.13 last 39.26) - open/close 20819 (time 1709787959.68 total 866.15 last 42.52) - open/close 21231 (time 1709787969.69 total 876.16 last 41.16) - open/close 21613 (time 1709787979.70 total 886.17 last 38.17) - open/close 22012 (time 1709787989.71 total 896.17 last 39.88) - open/close 22410 (time 1709787999.72 total 906.19 last 39.74) - open/close 22857 (time 1709788009.73 total 916.20 last 44.65) - open/close 23321 (time 1709788019.73 total 926.20 last 46.40) - open/close 23818 (time 1709788029.75 total 936.22 last 49.61) - open/close 24304 (time 1709788039.75 total 946.22 last 48.59) - open/close 24769 (time 1709788049.76 total 956.22 last 46.48) - open/close 25244 (time 1709788059.77 total 966.23 last 47.46) - open/close 25691 (time 1709788069.78 total 976.25 last 44.62) - open/close 26114 (time 1709788079.79 total 986.25 last 42.29) - open/close 26548 (time 1709788089.80 total 996.27 last 43.34) - open/close 26970 (time 1709788099.81 total 1006.28 last 42.14) - open/close 27381 (time 1709788109.82 total 1016.29 last 41.06) - open/close 27792 (time 1709788119.83 total 1026.29 last 41.08) - open/close 28204 (time 1709788129.84 total 1036.30 last 41.17) - open/close 28612 (time 1709788139.85 total 1046.31 last 40.74) - open/close 29009 (time 1709788149.87 total 1056.33 last 39.62) - open/close 29420 (time 1709788159.87 total 1066.34 last 41.09) - open/close 29825 (time 1709788169.89 total 1076.36 last 40.41) - open/close 30000 (time 1709788174.25 total 1080.71 last 40.18) - open/close 30394 (time 1709788184.26 total 1090.73 last 39.35) - open/close 30772 (time 1709788194.27 total 1100.73 last 37.77) - open/close 31135 (time 1709788204.28 total 1110.75 last 36.24) - open/close 31520 (time 1709788214.30 total 1120.76 last 38.44) - open/close 31905 (time 1709788224.30 total 1130.77 last 38.48) - open/close 32274 (time 1709788234.32 total 1140.79 last 36.83) - open/close 32635 (time 1709788244.35 total 1150.81 last 36.02) - open/close 32992 (time 1709788254.37 total 1160.83 last 35.63) - open/close 33339 (time 1709788264.39 total 1170.86 last 34.61) - open/close 33699 (time 1709788274.40 total 1180.86 last 35.98) - open/close 34037 (time 1709788284.42 total 1190.89 last 33.72) - open/close 34351 (time 1709788294.45 total 1200.91 last 31.33) - open/close 34678 (time 1709788304.45 total 1210.91 last 32.70) - open/close 34994 (time 1709788314.46 total 1220.93 last 31.56) - open/close 35301 (time 1709788324.47 total 1230.94 last 30.66) - open/close 35599 (time 1709788334.49 total 1240.95 last 29.76) - open/close 35893 (time 1709788344.53 total 1250.99 last 29.29) - open/close 36190 (time 1709788354.55 total 1261.01 last 29.64) - open/close 36495 (time 1709788364.56 total 1271.03 last 30.45) - open/close 36786 (time 1709788374.57 total 1281.04 last 29.06) - open/close 37078 (time 1709788384.59 total 1291.06 last 29.15) - open/close 37364 (time 1709788394.62 total 1301.09 last 28.51) - open/close 37665 (time 1709788404.63 total 1311.10 last 30.07) - open/close 37978 (time 1709788414.66 total 1321.12 last 31.22) - open/close 38288 (time 1709788424.67 total 1331.13 last 30.98) - open/close 38613 (time 1709788434.68 total 1341.14 last 32.47) - open/close 38940 (time 1709788444.69 total 1351.16 last 32.65) - open/close 39263 (time 1709788454.70 total 1361.17 last 32.25) - open/close 39546 (time 1709788464.71 total 1371.17 last 28.29) - open/close 39850 (time 1709788474.72 total 1381.19 last 30.36) - open/close 40000 (time 1709788479.55 total 1386.01 last 31.07) - open/close 40320 (time 1709788489.55 total 1396.02 last 32.00) - open/close 40649 (time 1709788499.56 total 1406.03 last 32.87) - open/close 40978 (time 1709788509.57 total 1416.04 last 32.85) - open/close 41301 (time 1709788519.60 total 1426.06 last 32.22) - open/close 41617 (time 1709788529.61 total 1436.08 last 31.56) - open/close 41940 (time 1709788539.62 total 1446.09 last 32.27) - open/close 42261 (time 1709788549.63 total 1456.10 last 32.06) - open/close 42575 (time 1709788559.64 total 1466.11 last 31.37) - open/close 42886 (time 1709788569.66 total 1476.12 last 31.06) - open/close 43189 (time 1709788579.68 total 1486.14 last 30.24) - open/close 43492 (time 1709788589.68 total 1496.15 last 30.28) - open/close 43802 (time 1709788599.68 total 1506.15 last 30.99) - open/close 44112 (time 1709788609.72 total 1516.18 last 30.90) - open/close 44420 (time 1709788619.73 total 1526.19 last 30.76) - open/close 44726 (time 1709788629.74 total 1536.21 last 30.55) - open/close 45015 (time 1709788639.76 total 1546.23 last 28.85) - open/close 45288 (time 1709788649.78 total 1556.24 last 27.26) - open/close 45565 (time 1709788659.80 total 1566.27 last 27.64) - open/close 45862 (time 1709788669.81 total 1576.27 last 29.67) - open/close 46133 (time 1709788679.82 total 1586.28 last 27.07) - open/close 46419 (time 1709788689.83 total 1596.30 last 28.56) - open/close 46705 (time 1709788699.83 total 1606.30 last 28.60) - open/close 46990 (time 1709788709.87 total 1616.33 last 28.40) - open/close 47282 (time 1709788719.90 total 1626.36 last 29.12) - open/close 47574 (time 1709788729.92 total 1636.39 last 29.12) - open/close 47862 (time 1709788739.94 total 1646.41 last 28.75) - open/close 48152 (time 1709788749.98 total 1656.44 last 28.89) - open/close 48436 (time 1709788759.99 total 1666.46 last 28.35) - open/close 48726 (time 1709788770.02 total 1676.48 last 28.93) - open/close 49003 (time 1709788780.02 total 1686.48 last 27.69) - open/close 49278 (time 1709788790.02 total 1696.49 last 27.50) - open/close 49557 (time 1709788800.04 total 1706.51 last 27.84) - open/close 49835 (time 1709788810.06 total 1716.53 last 27.74) total: 50000 open/close in 1722.69 seconds: 29.02 ops/second - unlinked 0 (time 1709788817 ; total 0 ; last 0) - unlinked 10000 (time 1709789242 ; total 425 ; last 425) - unlinked 20000 (time 1709789597 ; total 780 ; last 355) - unlinked 30000 (time 1709789849 ; total 1032 ; last 252) - unlinked 40000 (time 1709789981 ; total 1164 ; last 132) total: 50000 unlinks in 1211 seconds: 41.288193 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2