== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:12:05 (1788847925) start mds service on oleg323-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:48 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: kmemleak disabled oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg323-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg323-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1788848026.17 total 10.18 last 13.26) - open/close 194 (time 1788848036.34 total 20.35 last 5.80) - open/close 267 (time 1788848046.63 total 30.64 last 7.09) - open/close 411 (time 1788848057.14 total 41.15 last 13.71) - open/close 592 (time 1788848067.70 total 51.71 last 17.13) - open/close 845 (time 1788848077.84 total 61.85 last 24.96) - open/close 1099 (time 1788848088.24 total 72.24 last 24.43) - open/close 1263 (time 1788848098.25 total 82.26 last 16.38) - open/close 1322 (time 1788848109.74 total 93.75 last 5.14) - open/close 1543 (time 1788848119.99 total 103.99 last 21.57) - open/close 1685 (time 1788848130.70 total 114.71 last 13.25) - open/close 1815 (time 1788848141.24 total 125.25 last 12.34) - open/close 2020 (time 1788848151.26 total 135.26 last 20.46) - open/close 2156 (time 1788848161.27 total 145.28 last 13.58) - open/close 2275 (time 1788848172.13 total 156.14 last 10.96) - open/close 2392 (time 1788848183.82 total 167.83 last 10.01) - open/close 2533 (time 1788848196.73 total 180.74 last 10.92) - open/close 2703 (time 1788848206.75 total 190.75 last 16.98) - open/close 2840 (time 1788848217.15 total 201.16 last 13.17) - open/close 2965 (time 1788848228.15 total 212.16 last 11.37) - open/close 3197 (time 1788848239.07 total 223.07 last 21.25) - open/close 3334 (time 1788848249.34 total 233.34 last 13.34) - open/close 3515 (time 1788848259.36 total 243.36 last 18.07) - open/close 3742 (time 1788848271.35 total 255.35 last 18.93) - open/close 3844 (time 1788848283.14 total 267.14 last 8.65) - open/close 3965 (time 1788848293.15 total 277.16 last 12.08) - open/close 4020 (time 1788848304.62 total 288.63 last 4.79) - open/close 4139 (time 1788848315.31 total 299.32 last 11.13) - open/close 4233 (time 1788848325.94 total 309.94 last 8.84) - open/close 4316 (time 1788848336.23 total 320.24 last 8.06) - open/close 4403 (time 1788848346.85 total 330.86 last 8.19) - open/close 4603 (time 1788848357.56 total 341.56 last 18.68) - open/close 4797 (time 1788848367.67 total 351.68 last 19.19) - open/close 4999 (time 1788848378.15 total 362.16 last 19.27) - open/close 5203 (time 1788848388.17 total 372.18 last 20.36) - open/close 5439 (time 1788848398.99 total 382.99 last 21.82) - open/close 5668 (time 1788848409.91 total 393.92 last 20.96) - open/close 5849 (time 1788848419.92 total 403.93 last 18.09) - open/close 6066 (time 1788848430.78 total 414.79 last 19.98) - open/close 6296 (time 1788848441.64 total 425.64 last 21.19) - open/close 6440 (time 1788848452.86 total 436.87 last 12.83) - open/close 6545 (time 1788848463.27 total 447.28 last 10.09) - open/close 6699 (time 1788848474.29 total 458.30 last 13.97) - open/close 6858 (time 1788848484.30 total 468.30 last 15.90) - open/close 6979 (time 1788848494.33 total 478.33 last 12.06) - open/close 7085 (time 1788848504.35 total 488.36 last 10.57) - open/close 7151 (time 1788848514.36 total 498.37 last 6.59) - open/close 7286 (time 1788848524.85 total 508.86 last 12.87) - open/close 7365 (time 1788848535.82 total 519.83 last 7.20) - open/close 7473 (time 1788848547.50 total 531.51 last 9.24) - open/close 7605 (time 1788848558.04 total 542.05 last 12.53) - open/close 7746 (time 1788848568.05 total 552.06 last 14.08) - open/close 7874 (time 1788848578.07 total 562.08 last 12.78) - open/close 7995 (time 1788848589.50 total 573.51 last 10.59) - open/close 8154 (time 1788848600.33 total 584.33 last 14.69) - open/close 8269 (time 1788848610.33 total 594.34 last 11.50) - open/close 8397 (time 1788848621.24 total 605.25 last 11.73) - open/close 8482 (time 1788848632.02 total 616.03 last 7.89) - open/close 8622 (time 1788848643.05 total 627.06 last 12.69) - open/close 8816 (time 1788848653.50 total 637.51 last 18.57) - open/close 9056 (time 1788848663.54 total 647.55 last 23.90) - open/close 9279 (time 1788848673.55 total 657.55 last 22.29) - open/close 9469 (time 1788848683.55 total 667.56 last 18.99) - open/close 9705 (time 1788848693.56 total 677.56 last 23.59) - open/close 9946 (time 1788848703.56 total 687.56 last 24.10) - open/close 10000 (time 1788848708.79 total 692.79 last 10.33) - open/close 10160 (time 1788848719.51 total 703.51 last 14.92) - open/close 10314 (time 1788848729.56 total 713.57 last 15.32) - open/close 10339 (time 1788848740.61 total 724.62 last 2.26) - open/close 10441 (time 1788848751.12 total 735.13 last 9.71) - open/close 10634 (time 1788848761.15 total 745.16 last 19.24) - open/close 10807 (time 1788848772.05 total 756.05 last 15.88) - open/close 11034 (time 1788848782.06 total 766.07 last 22.68) - open/close 11203 (time 1788848792.09 total 776.10 last 16.85) - open/close 11434 (time 1788848802.30 total 786.31 last 22.62) - open/close 11571 (time 1788848812.85 total 796.86 last 12.98) - open/close 11776 (time 1788848822.87 total 806.87 last 20.47) - open/close 11863 (time 1788848833.26 total 817.26 last 8.37) - open/close 12053 (time 1788848843.38 total 827.38 last 18.77) - open/close 12293 (time 1788848854.18 total 838.19 last 22.21) - open/close 12467 (time 1788848864.19 total 848.20 last 17.39) - open/close 12610 (time 1788848874.21 total 858.22 last 14.27) - open/close 12743 (time 1788848884.21 total 868.22 last 13.30) - open/close 12890 (time 1788848895.29 total 879.29 last 13.27) - open/close 12987 (time 1788848905.99 total 889.99 last 9.06) - open/close 13166 (time 1788848916.17 total 900.18 last 17.58) - open/close 13401 (time 1788848926.18 total 910.19 last 23.47) - open/close 13613 (time 1788848936.90 total 920.91 last 19.78) - open/close 13851 (time 1788848946.92 total 930.92 last 23.76) - open/close 13982 (time 1788848956.93 total 940.94 last 13.08) - open/close 14212 (time 1788848966.93 total 950.94 last 23.00) - open/close 14366 (time 1788848976.96 total 960.97 last 15.36) - open/close 14486 (time 1788848986.97 total 970.97 last 11.99) - open/close 14741 (time 1788848999.52 total 983.53 last 20.31) - open/close 14900 (time 1788849010.16 total 994.16 last 14.95) - open/close 15082 (time 1788849020.94 total 1004.95 last 16.88) - open/close 15324 (time 1788849030.95 total 1014.96 last 24.17) - open/close 15522 (time 1788849041.01 total 1025.02 last 19.68) - open/close 15711 (time 1788849051.02 total 1035.03 last 18.88) - open/close 15897 (time 1788849061.04 total 1045.04 last 18.57) - open/close 16045 (time 1788849072.56 total 1056.57 last 12.84) - open/close 16226 (time 1788849082.63 total 1066.64 last 17.98) - open/close 16354 (time 1788849092.63 total 1076.64 last 12.80) - open/close 17017 (time 1788849102.64 total 1086.65 last 66.23) - open/close 17713 (time 1788849112.65 total 1096.66 last 69.56) - open/close 18334 (time 1788849122.65 total 1106.66 last 62.09) - open/close 18994 (time 1788849132.65 total 1116.66 last 66.00) - open/close 19643 (time 1788849142.66 total 1126.67 last 64.84) - open/close 20000 (time 1788849148.99 total 1132.99 last 56.47) - open/close 20647 (time 1788849158.99 total 1143.00 last 64.67) - open/close 21300 (time 1788849169.00 total 1153.01 last 65.22) - open/close 21882 (time 1788849179.02 total 1163.03 last 58.10) - open/close 22463 (time 1788849189.02 total 1173.03 last 58.09) - open/close 23071 (time 1788849199.03 total 1183.03 last 60.77) - open/close 23746 (time 1788849209.03 total 1193.04 last 67.45) - open/close 24400 (time 1788849219.03 total 1203.04 last 65.40) - open/close 25018 (time 1788849229.03 total 1213.04 last 61.80) - open/close 25649 (time 1788849239.05 total 1223.06 last 63.01) - open/close 26256 (time 1788849249.06 total 1233.07 last 60.63) - open/close 26888 (time 1788849259.07 total 1243.08 last 63.13) - open/close 27517 (time 1788849269.11 total 1253.11 last 62.69) - open/close 28189 (time 1788849279.13 total 1263.14 last 67.01) - open/close 28838 (time 1788849289.16 total 1273.17 last 64.73) - open/close 29518 (time 1788849299.16 total 1283.17 last 67.99) - open/close 30000 (time 1788849306.41 total 1290.42 last 66.46) - open/close 30670 (time 1788849316.41 total 1300.42 last 67.00) - open/close 31330 (time 1788849326.42 total 1310.43 last 65.95) - open/close 31981 (time 1788849336.43 total 1320.43 last 65.07) - open/close 32684 (time 1788849346.43 total 1330.44 last 70.30) - open/close 33343 (time 1788849356.43 total 1340.44 last 65.88) - open/close 33999 (time 1788849366.43 total 1350.44 last 65.60) - open/close 34677 (time 1788849376.45 total 1360.45 last 67.71) - open/close 35348 (time 1788849386.45 total 1370.46 last 67.08) - open/close 36019 (time 1788849396.47 total 1380.48 last 66.96) - open/close 36712 (time 1788849406.48 total 1390.49 last 69.24) - open/close 37395 (time 1788849416.49 total 1400.49 last 68.25) - open/close 38038 (time 1788849426.51 total 1410.51 last 64.16) - open/close 38637 (time 1788849436.52 total 1420.53 last 59.80) - open/close 39260 (time 1788849446.55 total 1430.55 last 62.17) - open/close 39938 (time 1788849456.55 total 1440.56 last 67.76) - open/close 40586 (time 1788849466.56 total 1450.56 last 64.76) - open/close 41227 (time 1788849476.56 total 1460.57 last 64.08) - open/close 41868 (time 1788849486.56 total 1470.57 last 64.08) - open/close 42524 (time 1788849496.57 total 1480.58 last 65.54) - open/close 43238 (time 1788849506.58 total 1490.59 last 71.35) - open/close 43926 (time 1788849516.59 total 1500.59 last 68.76) - open/close 44545 (time 1788849526.59 total 1510.59 last 61.90) - open/close 45247 (time 1788849536.59 total 1520.60 last 70.17) - open/close 45944 (time 1788849546.60 total 1530.61 last 69.64) - open/close 46659 (time 1788849556.61 total 1540.62 last 71.38) - open/close 47339 (time 1788849566.62 total 1550.62 last 67.99) - open/close 48049 (time 1788849576.62 total 1560.62 last 70.99) - open/close 48773 (time 1788849586.62 total 1570.63 last 72.35) - open/close 49467 (time 1788849596.63 total 1580.64 last 69.36) total: 50000 open/close in 1588.41 seconds: 31.48 ops/second - unlinked 0 (time 1788849608 ; total 0 ; last 0) - unlinked 10000 (time 1788849752 ; total 144 ; last 144) - unlinked 20000 (time 1788849881 ; total 273 ; last 129) - unlinked 30000 (time 1788850010 ; total 402 ; last 129) - unlinked 40000 (time 1788850119 ; total 511 ; last 109) total: 50000 unlinks in 607 seconds: 82.372322 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: kmemleak disabled modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2