== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:59:27 (1789675167) start mds service on oleg432-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 17 12:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: kmemleak disabled oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg432-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg432-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 197 (time 1789675256.94 total 10.00 last 19.70) - open/close 357 (time 1789675267.49 total 20.55 last 15.16) - open/close 560 (time 1789675277.50 total 30.56 last 20.28) - open/close 705 (time 1789675287.92 total 40.98 last 13.92) - open/close 836 (time 1789675297.92 total 50.98 last 13.10) - open/close 998 (time 1789675307.93 total 60.98 last 16.19) - open/close 1147 (time 1789675317.93 total 70.98 last 14.90) - open/close 1352 (time 1789675328.89 total 81.95 last 18.70) - open/close 1468 (time 1789675338.91 total 91.97 last 11.58) - open/close 1541 (time 1789675349.37 total 102.43 last 6.98) - open/close 1743 (time 1789675359.98 total 113.04 last 19.04) - open/close 1995 (time 1789675370.99 total 124.05 last 22.89) - open/close 2156 (time 1789675381.18 total 134.24 last 15.80) - open/close 2341 (time 1789675391.66 total 144.72 last 17.65) - open/close 2592 (time 1789675401.71 total 154.77 last 24.99) - open/close 2688 (time 1789675412.27 total 165.33 last 9.09) - open/close 2791 (time 1789675423.53 total 176.59 last 9.15) - open/close 2897 (time 1789675433.53 total 186.59 last 10.60) - open/close 3058 (time 1789675443.64 total 196.70 last 15.93) - open/close 3179 (time 1789675453.94 total 206.99 last 11.75) - open/close 3310 (time 1789675463.96 total 217.02 last 13.07) - open/close 3441 (time 1789675474.17 total 227.23 last 12.83) - open/close 3558 (time 1789675484.92 total 237.98 last 10.89) - open/close 3729 (time 1789675496.00 total 249.05 last 15.44) - open/close 3933 (time 1789675506.68 total 259.74 last 19.09) - open/close 4063 (time 1789675516.69 total 269.75 last 12.99) - open/close 4269 (time 1789675527.10 total 280.15 last 19.79) - open/close 4302 (time 1789675538.24 total 291.30 last 2.96) - open/close 4489 (time 1789675549.24 total 302.30 last 17.01) - open/close 4730 (time 1789675561.13 total 314.19 last 20.27) - open/close 4972 (time 1789675571.15 total 324.21 last 24.16) - open/close 5194 (time 1789675582.05 total 335.11 last 20.36) - open/close 5372 (time 1789675592.95 total 346.01 last 16.33) - open/close 5585 (time 1789675603.57 total 356.63 last 20.05) - open/close 5836 (time 1789675613.59 total 366.65 last 25.06) - open/close 6051 (time 1789675623.72 total 376.77 last 21.23) - open/close 6244 (time 1789675633.78 total 386.84 last 19.17) - open/close 6348 (time 1789675644.34 total 397.40 last 9.85) - open/close 6472 (time 1789675655.02 total 408.08 last 11.61) - open/close 6561 (time 1789675665.52 total 418.58 last 8.48) - open/close 6786 (time 1789675675.52 total 428.58 last 22.49) - open/close 6889 (time 1789675686.12 total 439.18 last 9.72) - open/close 6970 (time 1789675696.98 total 450.04 last 7.46) - open/close 7064 (time 1789675707.01 total 460.07 last 9.37) - open/close 7135 (time 1789675717.19 total 470.25 last 6.97) - open/close 7280 (time 1789675728.18 total 481.24 last 13.20) - open/close 7444 (time 1789675738.20 total 491.26 last 16.36) - open/close 7525 (time 1789675749.01 total 502.07 last 7.50) - open/close 7705 (time 1789675759.17 total 512.23 last 17.71) - open/close 7893 (time 1789675771.33 total 524.39 last 15.47) - open/close 8060 (time 1789675781.93 total 534.99 last 15.75) - open/close 8208 (time 1789675794.55 total 547.61 last 11.73) - open/close 8365 (time 1789675804.57 total 557.63 last 15.66) - open/close 8466 (time 1789675814.60 total 567.66 last 10.07) - open/close 8512 (time 1789675825.39 total 578.45 last 4.26) - open/close 8610 (time 1789675836.21 total 589.27 last 9.06) - open/close 8667 (time 1789675847.16 total 600.22 last 5.21) - open/close 8829 (time 1789675857.30 total 610.35 last 15.98) - open/close 8976 (time 1789675867.33 total 620.38 last 14.65) - open/close 9027 (time 1789675878.27 total 631.33 last 4.66) - open/close 9071 (time 1789675888.81 total 641.87 last 4.18) - open/close 9215 (time 1789675899.24 total 652.30 last 13.80) - open/close 9343 (time 1789675909.25 total 662.31 last 12.79) - open/close 9545 (time 1789675919.92 total 672.98 last 18.93) - open/close 9824 (time 1789675929.93 total 682.99 last 27.88) - open/close 9912 (time 1789675942.58 total 695.64 last 6.96) - open/close 10000 (time 1789675947.50 total 700.56 last 17.88) - open/close 10256 (time 1789675957.51 total 710.57 last 25.58) - open/close 10388 (time 1789675967.52 total 720.58 last 13.19) - open/close 10602 (time 1789675977.59 total 730.64 last 21.26) - open/close 10700 (time 1789675988.28 total 741.34 last 9.16) - open/close 10981 (time 1789676000.39 total 753.45 last 23.21) - open/close 11173 (time 1789676010.62 total 763.68 last 18.77) - open/close 11302 (time 1789676021.36 total 774.42 last 12.00) - open/close 11456 (time 1789676031.67 total 784.73 last 14.94) - open/close 11704 (time 1789676042.67 total 795.73 last 22.54) - open/close 11897 (time 1789676055.04 total 808.10 last 15.60) - open/close 12114 (time 1789676066.23 total 819.28 last 19.41) - open/close 12345 (time 1789676076.72 total 829.78 last 22.01) - open/close 12508 (time 1789676089.13 total 842.19 last 13.13) - open/close 12695 (time 1789676099.14 total 852.20 last 18.69) - open/close 12914 (time 1789676109.14 total 862.20 last 21.90) - open/close 13085 (time 1789676119.27 total 872.33 last 16.88) - open/close 13318 (time 1789676129.57 total 882.63 last 22.61) - open/close 13564 (time 1789676140.02 total 893.08 last 23.54) - open/close 13728 (time 1789676150.83 total 903.89 last 15.18) - open/close 13903 (time 1789676160.95 total 914.00 last 17.29) - open/close 13988 (time 1789676170.95 total 924.01 last 8.50) - open/close 14126 (time 1789676180.96 total 934.02 last 13.79) - open/close 14350 (time 1789676191.61 total 944.67 last 21.03) - open/close 14540 (time 1789676201.71 total 954.77 last 18.81) - open/close 14787 (time 1789676214.53 total 967.58 last 19.28) - open/close 14961 (time 1789676225.39 total 978.45 last 16.01) - open/close 15188 (time 1789676236.03 total 989.09 last 21.34) - open/close 15381 (time 1789676246.33 total 999.39 last 18.74) - open/close 15512 (time 1789676256.35 total 1009.41 last 13.07) - open/close 15694 (time 1789676266.87 total 1019.92 last 17.30) - open/close 15883 (time 1789676277.87 total 1030.93 last 17.18) - open/close 16051 (time 1789676288.90 total 1041.96 last 15.23) - open/close 16203 (time 1789676299.24 total 1052.30 last 14.70) - open/close 16406 (time 1789676310.19 total 1063.25 last 18.54) - open/close 16822 (time 1789676320.20 total 1073.25 last 41.57) - open/close 17368 (time 1789676330.22 total 1083.28 last 54.45) - open/close 17863 (time 1789676340.25 total 1093.30 last 49.39) - open/close 18246 (time 1789676350.25 total 1103.31 last 38.27) - open/close 18821 (time 1789676360.25 total 1113.31 last 57.50) - open/close 19346 (time 1789676370.26 total 1123.32 last 52.45) - open/close 19947 (time 1789676380.27 total 1133.33 last 60.06) - open/close 20607 (time 1789676390.27 total 1143.33 last 65.99) - open/close 21231 (time 1789676400.27 total 1153.33 last 62.39) - open/close 21843 (time 1789676410.27 total 1163.33 last 61.19) - open/close 22585 (time 1789676420.28 total 1173.33 last 74.20) - open/close 23334 (time 1789676430.28 total 1183.34 last 74.85) - open/close 24013 (time 1789676440.29 total 1193.35 last 67.86) - open/close 24686 (time 1789676450.29 total 1203.35 last 67.26) - open/close 25324 (time 1789676460.30 total 1213.35 last 63.79) - open/close 26083 (time 1789676470.30 total 1223.36 last 75.87) - open/close 26731 (time 1789676480.31 total 1233.36 last 64.76) - open/close 27310 (time 1789676490.31 total 1243.37 last 57.89) - open/close 27988 (time 1789676500.35 total 1253.41 last 67.49) - open/close 28666 (time 1789676510.36 total 1263.42 last 67.78) - open/close 29289 (time 1789676520.37 total 1273.43 last 62.24) - open/close 29928 (time 1789676530.38 total 1283.44 last 63.81) - open/close 30694 (time 1789676540.39 total 1293.45 last 76.52) - open/close 31391 (time 1789676550.43 total 1303.49 last 69.42) - open/close 31911 (time 1789676560.44 total 1313.50 last 51.95) - open/close 32622 (time 1789676570.44 total 1323.50 last 71.10) - open/close 33248 (time 1789676580.45 total 1333.51 last 62.56) - open/close 33869 (time 1789676590.46 total 1343.52 last 62.04) - open/close 34519 (time 1789676600.46 total 1353.52 last 64.98) - open/close 35102 (time 1789676610.48 total 1363.54 last 58.20) - open/close 35697 (time 1789676620.48 total 1373.54 last 59.48) - open/close 36291 (time 1789676630.49 total 1383.55 last 59.35) - open/close 36945 (time 1789676640.49 total 1393.55 last 65.40) - open/close 37608 (time 1789676650.49 total 1403.55 last 66.30) - open/close 38298 (time 1789676660.50 total 1413.56 last 68.95) - open/close 38930 (time 1789676670.51 total 1423.56 last 63.15) - open/close 39536 (time 1789676680.54 total 1433.59 last 60.42) - open/close 40000 (time 1789676689.54 total 1442.60 last 51.53) - open/close 40620 (time 1789676699.56 total 1452.62 last 61.84) - open/close 41285 (time 1789676709.57 total 1462.63 last 66.48) - open/close 41915 (time 1789676719.58 total 1472.63 last 62.95) - open/close 42586 (time 1789676729.58 total 1482.64 last 67.07) - open/close 43247 (time 1789676739.59 total 1492.64 last 66.06) - open/close 43915 (time 1789676749.59 total 1502.65 last 66.79) - open/close 44600 (time 1789676759.60 total 1512.65 last 68.45) - open/close 45251 (time 1789676769.60 total 1522.65 last 65.10) - open/close 45885 (time 1789676779.60 total 1532.65 last 63.40) - open/close 46604 (time 1789676789.61 total 1542.67 last 71.80) - open/close 47226 (time 1789676799.64 total 1552.70 last 62.01) - open/close 47864 (time 1789676809.65 total 1562.70 last 63.77) - open/close 48564 (time 1789676819.65 total 1572.70 last 69.99) - open/close 49273 (time 1789676829.65 total 1582.71 last 70.88) - open/close 49986 (time 1789676839.66 total 1592.72 last 71.22) total: 50000 open/close in 1592.87 seconds: 31.39 ops/second - unlinked 0 (time 1789676844 ; total 0 ; last 0) - unlinked 10000 (time 1789676995 ; total 151 ; last 151) - unlinked 20000 (time 1789677148 ; total 304 ; last 153) - unlinked 30000 (time 1789677312 ; total 468 ; last 164) - unlinked 40000 (time 1789677468 ; total 624 ; last 156) total: 50000 unlinks in 771 seconds: 64.850845 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: kmemleak disabled modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2