== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:22:03 (1788844923) start mds service on oleg118-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: kmemleak disabled oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg118-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg118-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1788845016.55 total 10.72 last 19.04) - open/close 393 (time 1788845026.57 total 20.73 last 18.87) - open/close 586 (time 1788845036.59 total 30.76 last 19.25) - open/close 781 (time 1788845047.14 total 41.31 last 18.48) - open/close 871 (time 1788845057.30 total 51.47 last 8.86) - open/close 1086 (time 1788845067.31 total 61.48 last 21.47) - open/close 1324 (time 1788845077.32 total 71.49 last 23.78) - open/close 1500 (time 1788845087.32 total 81.49 last 17.60) - open/close 1645 (time 1788845097.33 total 91.49 last 14.50) - open/close 1736 (time 1788845109.59 total 103.76 last 7.42) - open/close 1879 (time 1788845120.31 total 114.47 last 13.35) - open/close 1988 (time 1788845130.33 total 124.50 last 10.87) - open/close 2081 (time 1788845140.35 total 134.52 last 9.28) - open/close 2210 (time 1788845150.35 total 144.52 last 12.90) - open/close 2370 (time 1788845160.67 total 154.84 last 15.51) - open/close 2596 (time 1788845171.17 total 165.33 last 21.54) - open/close 2817 (time 1788845181.18 total 175.34 last 22.08) - open/close 3035 (time 1788845191.21 total 185.38 last 21.72) - open/close 3186 (time 1788845201.49 total 195.65 last 14.70) - open/close 3426 (time 1788845211.60 total 205.77 last 23.72) - open/close 3696 (time 1788845222.24 total 216.40 last 25.40) - open/close 3876 (time 1788845232.87 total 227.04 last 16.92) - open/close 4110 (time 1788845242.90 total 237.06 last 23.35) - open/close 4227 (time 1788845252.99 total 247.16 last 11.59) - open/close 4342 (time 1788845263.20 total 257.36 last 11.27) - open/close 4533 (time 1788845273.97 total 268.14 last 17.73) - open/close 4650 (time 1788845285.01 total 279.18 last 10.60) - open/close 4898 (time 1788845295.01 total 289.18 last 24.79) - open/close 5112 (time 1788845305.80 total 299.97 last 19.84) - open/close 5269 (time 1788845315.81 total 309.98 last 15.69) - open/close 5402 (time 1788845325.82 total 319.99 last 13.28) - open/close 5560 (time 1788845336.52 total 330.69 last 14.76) - open/close 5798 (time 1788845346.53 total 340.69 last 23.79) - open/close 6014 (time 1788845356.53 total 350.70 last 21.59) - open/close 6133 (time 1788845366.54 total 360.70 last 11.89) - open/close 6386 (time 1788845377.34 total 371.50 last 23.43) - open/close 6475 (time 1788845388.02 total 382.19 last 8.33) - open/close 6709 (time 1788845398.67 total 392.84 last 21.97) - open/close 6901 (time 1788845408.69 total 402.86 last 19.16) - open/close 7073 (time 1788845418.90 total 413.07 last 16.85) - open/close 7150 (time 1788845429.47 total 423.64 last 7.28) - open/close 7230 (time 1788845439.91 total 434.08 last 7.66) - open/close 7417 (time 1788845449.94 total 444.11 last 18.64) - open/close 7578 (time 1788845459.95 total 454.12 last 16.09) - open/close 7824 (time 1788845469.97 total 464.14 last 24.55) - open/close 8030 (time 1788845480.57 total 474.74 last 19.44) - open/close 8213 (time 1788845491.86 total 486.02 last 16.21) - open/close 8416 (time 1788845502.43 total 496.60 last 19.20) - open/close 8594 (time 1788845512.45 total 506.61 last 17.77) - open/close 8745 (time 1788845524.33 total 518.50 last 12.70) - open/close 8874 (time 1788845534.34 total 528.50 last 12.90) - open/close 9093 (time 1788845544.99 total 539.15 last 20.56) - open/close 9282 (time 1788845556.06 total 550.22 last 17.07) - open/close 9365 (time 1788845566.06 total 560.23 last 8.30) - open/close 9488 (time 1788845576.28 total 570.45 last 12.03) - open/close 9610 (time 1788845586.91 total 581.07 last 11.49) - open/close 9790 (time 1788845597.16 total 591.33 last 17.55) - open/close 9988 (time 1788845607.70 total 601.87 last 18.78) - open/close 10148 (time 1788845617.72 total 611.89 last 15.97) - open/close 10291 (time 1788845628.69 total 622.86 last 13.04) - open/close 10515 (time 1788845640.04 total 634.21 last 19.74) - open/close 10720 (time 1788845650.66 total 644.83 last 19.31) - open/close 10842 (time 1788845660.83 total 655.00 last 11.99) - open/close 10984 (time 1788845671.23 total 665.40 last 13.66) - open/close 11139 (time 1788845683.87 total 678.03 last 12.27) - open/close 11274 (time 1788845694.04 total 688.21 last 13.27) - open/close 11537 (time 1788845704.85 total 699.02 last 24.32) - open/close 11799 (time 1788845714.87 total 709.04 last 26.15) - open/close 11996 (time 1788845726.20 total 720.36 last 17.40) - open/close 12173 (time 1788845736.75 total 730.92 last 16.77) - open/close 12317 (time 1788845747.55 total 741.72 last 13.34) - open/close 12472 (time 1788845758.05 total 752.22 last 14.75) - open/close 12562 (time 1788845768.48 total 762.65 last 8.63) - open/close 12799 (time 1788845778.51 total 772.67 last 23.64) - open/close 12856 (time 1788845788.88 total 783.05 last 5.50) - open/close 12926 (time 1788845799.15 total 793.32 last 6.82) - open/close 13093 (time 1788845809.16 total 803.33 last 16.69) - open/close 13248 (time 1788845820.12 total 814.29 last 14.13) - open/close 13518 (time 1788845831.07 total 825.24 last 24.67) - open/close 13723 (time 1788845841.08 total 835.25 last 20.47) - open/close 13937 (time 1788845851.97 total 846.13 last 19.66) - open/close 13973 (time 1788845862.27 total 856.43 last 3.50) - open/close 14094 (time 1788845872.35 total 866.51 last 12.00) - open/close 14202 (time 1788845882.37 total 876.53 last 10.78) - open/close 14387 (time 1788845892.44 total 886.61 last 18.37) - open/close 14505 (time 1788845903.15 total 897.32 last 11.02) - open/close 14696 (time 1788845913.16 total 907.33 last 19.08) - open/close 14873 (time 1788845923.19 total 917.36 last 17.65) - open/close 15033 (time 1788845933.91 total 928.08 last 14.93) - open/close 15206 (time 1788845944.93 total 939.09 last 15.71) - open/close 15385 (time 1788845955.73 total 949.90 last 16.57) - open/close 15600 (time 1788845966.07 total 960.24 last 20.79) - open/close 15867 (time 1788845976.74 total 970.90 last 25.04) - open/close 16103 (time 1788845986.75 total 980.91 last 23.57) - open/close 16238 (time 1788845996.75 total 990.92 last 13.50) - open/close 16462 (time 1788846009.50 total 1003.67 last 17.57) - open/close 16652 (time 1788846019.81 total 1013.98 last 18.42) - open/close 16750 (time 1788846030.31 total 1024.48 last 9.33) - open/close 16921 (time 1788846041.04 total 1035.21 last 15.94) - open/close 17172 (time 1788846051.06 total 1045.22 last 25.07) - open/close 17845 (time 1788846061.06 total 1055.23 last 67.29) - open/close 18545 (time 1788846071.07 total 1065.24 last 69.89) - open/close 19178 (time 1788846081.11 total 1075.28 last 63.07) - open/close 19817 (time 1788846091.14 total 1085.30 last 63.74) - open/close 20000 (time 1788846095.29 total 1089.46 last 44.08) - open/close 20602 (time 1788846105.30 total 1099.47 last 60.15) - open/close 21241 (time 1788846115.30 total 1109.47 last 63.88) - open/close 21883 (time 1788846125.31 total 1119.47 last 64.17) - open/close 22501 (time 1788846135.32 total 1129.49 last 61.70) - open/close 23147 (time 1788846145.33 total 1139.50 last 64.53) - open/close 23835 (time 1788846155.34 total 1149.51 last 68.75) - open/close 24535 (time 1788846165.34 total 1159.51 last 70.00) - open/close 25149 (time 1788846175.35 total 1169.52 last 61.34) - open/close 25866 (time 1788846185.35 total 1179.52 last 71.67) - open/close 26549 (time 1788846195.36 total 1189.53 last 68.23) - open/close 27258 (time 1788846205.37 total 1199.54 last 70.83) - open/close 27941 (time 1788846215.38 total 1209.55 last 68.25) - open/close 28597 (time 1788846225.38 total 1219.55 last 65.59) - open/close 29292 (time 1788846235.38 total 1229.55 last 69.50) - open/close 29982 (time 1788846245.40 total 1239.56 last 68.91) - open/close 30743 (time 1788846255.40 total 1249.57 last 76.06) - open/close 31262 (time 1788846265.41 total 1259.58 last 51.85) - open/close 31844 (time 1788846275.42 total 1269.58 last 58.16) - open/close 32412 (time 1788846285.42 total 1279.59 last 56.77) - open/close 32994 (time 1788846295.43 total 1289.59 last 58.18) - open/close 33559 (time 1788846305.44 total 1299.60 last 56.45) - open/close 34104 (time 1788846315.45 total 1309.61 last 54.43) - open/close 34712 (time 1788846325.47 total 1319.63 last 60.69) - open/close 35296 (time 1788846335.47 total 1329.64 last 58.37) - open/close 35876 (time 1788846345.48 total 1339.65 last 57.92) - open/close 36522 (time 1788846355.49 total 1349.66 last 64.55) - open/close 37201 (time 1788846365.50 total 1359.67 last 67.84) - open/close 37878 (time 1788846375.50 total 1369.67 last 67.69) - open/close 38512 (time 1788846385.51 total 1379.68 last 63.35) - open/close 39137 (time 1788846395.52 total 1389.69 last 62.41) - open/close 39756 (time 1788846405.53 total 1399.70 last 61.87) - open/close 40000 (time 1788846408.29 total 1402.45 last 88.44) - open/close 40698 (time 1788846418.30 total 1412.47 last 69.69) - open/close 41331 (time 1788846428.30 total 1422.47 last 63.29) - open/close 42006 (time 1788846438.31 total 1432.48 last 67.47) - open/close 42699 (time 1788846448.31 total 1442.48 last 69.27) - open/close 43408 (time 1788846458.33 total 1452.49 last 70.82) - open/close 44110 (time 1788846468.33 total 1462.50 last 70.17) - open/close 44658 (time 1788846478.33 total 1472.50 last 54.80) - open/close 45273 (time 1788846488.35 total 1482.51 last 61.40) - open/close 45891 (time 1788846498.37 total 1492.54 last 61.67) - open/close 46474 (time 1788846508.37 total 1502.54 last 58.30) - open/close 47118 (time 1788846518.38 total 1512.55 last 64.32) - open/close 47714 (time 1788846528.39 total 1522.56 last 59.53) - open/close 48367 (time 1788846538.40 total 1532.57 last 65.22) - open/close 49043 (time 1788846548.41 total 1542.58 last 67.56) - open/close 49721 (time 1788846558.42 total 1552.59 last 67.71) total: 50000 open/close in 1555.87 seconds: 32.14 ops/second - unlinked 0 (time 1788846567 ; total 0 ; last 0) - unlinked 10000 (time 1788846724 ; total 157 ; last 157) - unlinked 20000 (time 1788846885 ; total 318 ; last 161) - unlinked 30000 (time 1788847038 ; total 471 ; last 153) - unlinked 40000 (time 1788847191 ; total 624 ; last 153) total: 50000 unlinks in 777 seconds: 64.350067 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: kmemleak disabled modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2