== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:00:33 (1788642033) start mds service on oleg128-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:19 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: kmemleak disabled oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg128-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg128-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg128-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 114 (time 1788642124.66 total 13.15 last 8.67) - open/close 356 (time 1788642135.22 total 23.70 last 22.93) - open/close 477 (time 1788642145.38 total 33.86 last 11.91) - open/close 628 (time 1788642155.38 total 43.87 last 15.09) - open/close 810 (time 1788642165.55 total 54.03 last 17.91) - open/close 883 (time 1788642175.56 total 64.04 last 7.29) - open/close 1086 (time 1788642186.42 total 74.90 last 18.70) - open/close 1253 (time 1788642199.32 total 87.81 last 12.94) - open/close 1446 (time 1788642209.37 total 97.85 last 19.21) - open/close 1627 (time 1788642219.37 total 107.86 last 18.09) - open/close 1846 (time 1788642229.37 total 117.86 last 21.90) - open/close 2052 (time 1788642239.40 total 127.89 last 20.54) - open/close 2229 (time 1788642249.98 total 138.47 last 16.73) - open/close 2370 (time 1788642260.51 total 148.99 last 13.39) - open/close 2523 (time 1788642271.46 total 159.95 last 13.97) - open/close 2656 (time 1788642281.47 total 169.96 last 13.28) - open/close 2767 (time 1788642291.50 total 179.98 last 11.07) - open/close 2840 (time 1788642304.35 total 192.84 last 5.68) - open/close 3048 (time 1788642314.37 total 202.86 last 20.77) - open/close 3211 (time 1788642325.22 total 213.71 last 15.02) - open/close 3377 (time 1788642335.23 total 223.71 last 16.60) - open/close 3479 (time 1788642345.23 total 233.71 last 10.20) - open/close 3597 (time 1788642355.86 total 244.34 last 11.10) - open/close 3706 (time 1788642365.88 total 254.36 last 10.88) - open/close 3823 (time 1788642375.89 total 264.37 last 11.69) - open/close 4012 (time 1788642385.91 total 274.40 last 18.85) - open/close 4202 (time 1788642395.92 total 284.40 last 18.99) - open/close 4358 (time 1788642406.04 total 294.52 last 15.42) - open/close 4463 (time 1788642416.06 total 304.54 last 10.48) - open/close 4651 (time 1788642426.20 total 314.69 last 18.54) - open/close 4781 (time 1788642437.03 total 325.52 last 12.00) - open/close 4940 (time 1788642447.89 total 336.38 last 14.64) - open/close 5164 (time 1788642458.66 total 347.14 last 20.81) - open/close 5367 (time 1788642469.27 total 357.76 last 19.12) - open/close 5533 (time 1788642479.85 total 368.34 last 15.69) - open/close 5788 (time 1788642489.86 total 378.34 last 25.48) - open/close 5910 (time 1788642500.64 total 389.12 last 11.32) - open/close 6078 (time 1788642510.99 total 399.47 last 16.23) - open/close 6340 (time 1788642521.04 total 409.52 last 26.07) - open/close 6463 (time 1788642531.98 total 420.46 last 11.24) - open/close 6545 (time 1788642542.56 total 431.04 last 7.75) - open/close 6641 (time 1788642552.57 total 441.05 last 9.59) - open/close 6840 (time 1788642562.57 total 451.05 last 19.90) - open/close 7048 (time 1788642572.89 total 461.37 last 20.16) - open/close 7200 (time 1788642583.12 total 471.60 last 14.85) - open/close 7387 (time 1788642593.13 total 481.62 last 18.68) - open/close 7428 (time 1788642603.60 total 492.09 last 3.92) - open/close 7555 (time 1788642613.61 total 502.09 last 12.70) - open/close 7687 (time 1788642624.06 total 512.55 last 12.63) - open/close 7851 (time 1788642634.08 total 522.56 last 16.38) - open/close 7938 (time 1788642645.01 total 533.49 last 7.96) - open/close 8152 (time 1788642657.89 total 546.37 last 16.62) - open/close 8371 (time 1788642668.27 total 556.75 last 21.11) - open/close 8543 (time 1788642678.27 total 566.75 last 17.20) - open/close 8796 (time 1788642688.27 total 576.76 last 25.29) - open/close 8953 (time 1788642704.04 total 592.53 last 9.96) - open/close 9067 (time 1788642714.06 total 602.54 last 11.38) - open/close 9205 (time 1788642724.40 total 612.89 last 13.34) - open/close 9430 (time 1788642734.41 total 622.90 last 22.47) - open/close 9471 (time 1788642744.73 total 633.22 last 3.97) - open/close 9543 (time 1788642754.73 total 643.22 last 7.20) - open/close 9685 (time 1788642764.95 total 653.43 last 13.90) - open/close 9842 (time 1788642774.96 total 663.44 last 15.69) - open/close 10000 (time 1788642781.26 total 669.74 last 25.08) - open/close 10136 (time 1788642791.63 total 680.12 last 13.11) - open/close 10382 (time 1788642801.66 total 690.14 last 24.54) - open/close 10600 (time 1788642812.00 total 700.49 last 21.07) - open/close 10746 (time 1788642824.61 total 713.09 last 11.58) - open/close 10784 (time 1788642835.29 total 723.77 last 3.56) - open/close 11006 (time 1788642845.29 total 733.77 last 22.20) - open/close 11160 (time 1788642856.28 total 744.76 last 14.02) - open/close 11278 (time 1788642866.32 total 754.81 last 11.74) - open/close 11513 (time 1788642876.35 total 764.84 last 23.43) - open/close 11621 (time 1788642886.67 total 775.16 last 10.46) - open/close 11740 (time 1788642897.30 total 785.78 last 11.20) - open/close 11870 (time 1788642908.14 total 796.62 last 11.99) - open/close 12015 (time 1788642919.01 total 807.50 last 13.33) - open/close 12133 (time 1788642929.02 total 817.50 last 11.80) - open/close 12284 (time 1788642939.49 total 827.97 last 14.42) - open/close 12543 (time 1788642950.18 total 838.66 last 24.23) - open/close 12762 (time 1788642960.19 total 848.67 last 21.87) - open/close 13002 (time 1788642971.15 total 859.64 last 21.89) - open/close 13252 (time 1788642981.65 total 870.13 last 23.82) - open/close 13424 (time 1788642994.48 total 882.96 last 13.41) - open/close 13639 (time 1788643004.48 total 892.96 last 21.49) - open/close 13876 (time 1788643014.50 total 902.99 last 23.64) - open/close 14105 (time 1788643024.51 total 912.99 last 22.90) - open/close 14305 (time 1788643035.42 total 923.91 last 18.32) - open/close 14552 (time 1788643045.43 total 933.91 last 24.69) - open/close 14760 (time 1788643055.46 total 943.95 last 20.73) - open/close 14904 (time 1788643066.06 total 954.55 last 13.58) - open/close 15058 (time 1788643076.62 total 965.10 last 14.59) - open/close 15214 (time 1788643086.63 total 975.11 last 15.59) - open/close 15477 (time 1788643096.64 total 985.12 last 26.27) - open/close 15762 (time 1788643106.64 total 995.12 last 28.50) - open/close 15992 (time 1788643116.77 total 1005.25 last 22.71) - open/close 16234 (time 1788643127.53 total 1016.02 last 22.47) - open/close 16485 (time 1788643138.19 total 1026.67 last 23.56) - open/close 16694 (time 1788643148.96 total 1037.44 last 19.40) - open/close 17179 (time 1788643158.96 total 1047.45 last 48.48) - open/close 17983 (time 1788643168.97 total 1057.45 last 80.38) - open/close 18803 (time 1788643179.04 total 1067.53 last 81.36) - open/close 19698 (time 1788643189.05 total 1077.53 last 89.49) - open/close 20000 (time 1788643191.75 total 1080.24 last 111.57) - open/close 20865 (time 1788643201.75 total 1090.24 last 86.49) - open/close 21569 (time 1788643211.76 total 1100.24 last 70.35) - open/close 22566 (time 1788643221.76 total 1110.25 last 99.68) - open/close 23513 (time 1788643231.76 total 1120.25 last 94.68) - open/close 24413 (time 1788643241.77 total 1130.26 last 89.91) - open/close 25256 (time 1788643251.79 total 1140.28 last 84.15) - open/close 26121 (time 1788643261.79 total 1150.28 last 86.48) - open/close 27084 (time 1788643271.80 total 1160.28 last 96.27) - open/close 28018 (time 1788643281.80 total 1170.29 last 93.36) - open/close 28962 (time 1788643291.81 total 1180.29 last 94.32) - open/close 29978 (time 1788643301.82 total 1190.30 last 101.51) - open/close 30922 (time 1788643311.82 total 1200.31 last 94.36) - open/close 31852 (time 1788643321.83 total 1210.31 last 92.96) - open/close 32845 (time 1788643331.83 total 1220.31 last 99.30) - open/close 33860 (time 1788643341.84 total 1230.32 last 101.41) - open/close 34743 (time 1788643351.84 total 1240.33 last 88.22) - open/close 35611 (time 1788643361.84 total 1250.33 last 86.80) - open/close 36543 (time 1788643371.85 total 1260.34 last 93.14) - open/close 37411 (time 1788643381.86 total 1270.35 last 86.70) - open/close 38259 (time 1788643391.87 total 1280.36 last 84.71) - open/close 39113 (time 1788643401.88 total 1290.36 last 85.35) - open/close 40000 (time 1788643411.16 total 1299.65 last 95.56) - open/close 40997 (time 1788643421.16 total 1309.65 last 99.68) - open/close 41983 (time 1788643431.17 total 1319.65 last 98.57) - open/close 42903 (time 1788643441.17 total 1329.65 last 91.97) - open/close 43903 (time 1788643451.17 total 1339.66 last 99.96) - open/close 44777 (time 1788643461.18 total 1349.66 last 87.34) - open/close 45745 (time 1788643471.18 total 1359.67 last 96.75) - open/close 46632 (time 1788643481.19 total 1369.67 last 88.65) - open/close 47383 (time 1788643491.26 total 1379.74 last 74.59) - open/close 48222 (time 1788643501.27 total 1389.76 last 83.78) - open/close 49168 (time 1788643511.28 total 1399.77 last 94.51) total: 50000 open/close in 1409.46 seconds: 35.47 ops/second - unlinked 0 (time 1788643525 ; total 0 ; last 0) - unlinked 10000 (time 1788643642 ; total 117 ; last 117) - unlinked 20000 (time 1788643743 ; total 218 ; last 101) - unlinked 30000 (time 1788643819 ; total 294 ; last 76) - unlinked 40000 (time 1788643888 ; total 363 ; last 69) total: 50000 unlinks in 433 seconds: 115.473442 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server unloading modules via unload_modules_local on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing unload_modules_local oleg128-server: kmemleak disabled modules unloaded. pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2