== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:37:36 (1788838656) start mds service on oleg606-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing load_modules_local oleg606-server: kmemleak disabled oleg606-server: Loading modules from /home/green/git/lustre-release/lustre oleg606-server: detected 4 online CPUs by sysfs oleg606-server: MODOPTS_LIBCFS= oleg606-server: Force libcfs to create 2 CPU partitions oleg606-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg606-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg606-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg606-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg606-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1788838735.63 total 10.01 last 24.78) - open/close 422 (time 1788838747.00 total 21.38 last 15.30) - open/close 561 (time 1788838757.01 total 31.38 last 13.90) - open/close 777 (time 1788838767.01 total 41.39 last 21.58) - open/close 985 (time 1788838777.02 total 51.39 last 20.79) - open/close 1228 (time 1788838787.76 total 62.14 last 22.62) - open/close 1497 (time 1788838797.76 total 72.14 last 26.89) - open/close 1680 (time 1788838810.41 total 84.79 last 14.46) - open/close 1930 (time 1788838820.42 total 94.79 last 24.99) - open/close 2055 (time 1788838831.25 total 105.62 last 11.54) - open/close 2252 (time 1788838841.25 total 115.63 last 19.69) - open/close 2391 (time 1788838851.26 total 125.63 last 13.89) - open/close 2644 (time 1788838861.27 total 135.65 last 25.27) - open/close 2882 (time 1788838871.28 total 145.66 last 23.77) - open/close 3117 (time 1788838881.80 total 156.18 last 22.34) - open/close 3356 (time 1788838892.27 total 166.65 last 22.82) - open/close 3589 (time 1788838902.41 total 176.79 last 22.98) - open/close 3765 (time 1788838915.16 total 189.54 last 13.81) - open/close 4008 (time 1788838925.16 total 199.54 last 24.29) - open/close 4105 (time 1788838935.29 total 209.67 last 9.58) - open/close 4274 (time 1788838945.66 total 220.03 last 16.30) - open/close 4559 (time 1788838955.66 total 230.04 last 28.48) - open/close 4774 (time 1788838965.67 total 240.05 last 21.47) - open/close 5063 (time 1788838976.38 total 250.75 last 27.01) - open/close 5289 (time 1788838986.42 total 260.79 last 22.51) - open/close 5541 (time 1788838996.42 total 270.80 last 25.20) - open/close 5745 (time 1788839007.09 total 281.46 last 19.12) - open/close 5937 (time 1788839017.10 total 291.47 last 19.18) - open/close 6160 (time 1788839028.16 total 302.54 last 20.16) - open/close 6406 (time 1788839038.58 total 312.95 last 23.62) - open/close 6651 (time 1788839049.09 total 323.47 last 23.30) - open/close 6930 (time 1788839059.91 total 334.29 last 25.79) - open/close 7177 (time 1788839069.93 total 344.30 last 24.66) - open/close 7381 (time 1788839080.63 total 355.01 last 19.06) - open/close 7634 (time 1788839090.94 total 365.32 last 24.53) - open/close 7886 (time 1788839101.37 total 375.75 last 24.17) - open/close 8176 (time 1788839111.92 total 386.30 last 27.49) - open/close 8438 (time 1788839122.60 total 396.98 last 24.53) - open/close 8681 (time 1788839132.85 total 407.22 last 23.72) - open/close 8962 (time 1788839142.89 total 417.26 last 27.99) - open/close 9204 (time 1788839153.01 total 427.39 last 23.90) - open/close 9455 (time 1788839163.60 total 437.97 last 23.71) - open/close 9692 (time 1788839173.92 total 448.29 last 22.96) - open/close 9935 (time 1788839183.93 total 458.30 last 24.28) - open/close 10000 (time 1788839187.21 total 461.59 last 19.80) - open/close 10265 (time 1788839197.21 total 471.59 last 26.50) - open/close 10422 (time 1788839207.24 total 481.61 last 15.66) - open/close 10650 (time 1788839217.72 total 492.10 last 21.75) - open/close 10901 (time 1788839228.46 total 502.84 last 23.36) - open/close 11153 (time 1788839239.29 total 513.66 last 23.29) - open/close 11319 (time 1788839249.82 total 524.20 last 15.75) - open/close 11467 (time 1788839259.82 total 534.20 last 14.80) - open/close 11698 (time 1788839270.34 total 544.71 last 21.98) - open/close 11877 (time 1788839280.34 total 554.72 last 17.89) - open/close 12117 (time 1788839292.76 total 567.13 last 19.33) - open/close 12360 (time 1788839303.06 total 577.43 last 23.59) - open/close 12606 (time 1788839314.04 total 588.42 last 22.39) - open/close 12865 (time 1788839325.04 total 599.42 last 23.55) - open/close 13043 (time 1788839335.28 total 609.66 last 17.38) - open/close 13296 (time 1788839345.30 total 619.67 last 25.27) - open/close 13556 (time 1788839356.09 total 630.47 last 24.09) - open/close 13838 (time 1788839367.16 total 641.54 last 25.47) - open/close 14067 (time 1788839377.47 total 651.84 last 22.22) - open/close 14305 (time 1788839387.48 total 661.85 last 23.78) - open/close 14551 (time 1788839397.48 total 671.86 last 24.58) - open/close 14771 (time 1788839407.50 total 681.87 last 21.97) - open/close 15050 (time 1788839417.50 total 691.88 last 27.89) - open/close 15265 (time 1788839427.83 total 702.21 last 20.81) - open/close 15516 (time 1788839437.85 total 712.22 last 25.07) - open/close 15732 (time 1788839447.85 total 722.23 last 21.59) - open/close 15890 (time 1788839459.38 total 733.76 last 13.70) - open/close 16048 (time 1788839469.63 total 744.01 last 15.42) - open/close 16328 (time 1788839479.64 total 754.02 last 27.97) - open/close 16578 (time 1788839489.78 total 764.15 last 24.66) - open/close 16823 (time 1788839500.47 total 774.84 last 22.92) - open/close 17057 (time 1788839511.10 total 785.48 last 22.00) - open/close 17295 (time 1788839521.93 total 796.30 last 21.99) - open/close 17497 (time 1788839532.34 total 806.72 last 19.40) - open/close 17745 (time 1788839542.36 total 816.73 last 24.76) - open/close 17968 (time 1788839552.65 total 827.03 last 21.66) - open/close 18102 (time 1788839562.70 total 837.08 last 13.33) - open/close 18326 (time 1788839572.74 total 847.11 last 22.33) - open/close 18529 (time 1788839582.74 total 857.12 last 20.28) - open/close 18705 (time 1788839592.75 total 867.13 last 17.59) - open/close 18921 (time 1788839602.95 total 877.32 last 21.18) - open/close 19159 (time 1788839613.51 total 887.89 last 22.52) - open/close 19340 (time 1788839623.61 total 897.99 last 17.92) - open/close 20000 (time 1788839633.35 total 907.73 last 67.79) - open/close 20742 (time 1788839643.35 total 917.73 last 74.19) - open/close 21471 (time 1788839653.36 total 927.73 last 72.86) - open/close 22186 (time 1788839663.36 total 937.73 last 71.50) - open/close 22935 (time 1788839673.36 total 947.74 last 74.90) - open/close 23655 (time 1788839683.36 total 957.74 last 71.98) - open/close 24368 (time 1788839693.37 total 967.74 last 71.25) - open/close 25076 (time 1788839703.38 total 977.75 last 70.74) - open/close 25749 (time 1788839713.38 total 987.76 last 67.27) - open/close 26384 (time 1788839723.39 total 997.76 last 63.47) - open/close 27099 (time 1788839733.42 total 1007.79 last 71.27) - open/close 27866 (time 1788839743.42 total 1017.80 last 76.67) - open/close 28594 (time 1788839753.42 total 1027.80 last 72.77) - open/close 29337 (time 1788839763.44 total 1037.81 last 74.20) - open/close 30000 (time 1788839772.59 total 1046.96 last 72.48) - open/close 30777 (time 1788839782.60 total 1056.97 last 77.62) - open/close 31483 (time 1788839792.60 total 1066.98 last 70.58) - open/close 32284 (time 1788839802.61 total 1076.99 last 80.02) - open/close 33061 (time 1788839812.61 total 1086.99 last 77.68) - open/close 33838 (time 1788839822.62 total 1097.00 last 77.60) - open/close 34701 (time 1788839832.64 total 1107.02 last 86.16) - open/close 35434 (time 1788839842.65 total 1117.02 last 73.24) - open/close 36169 (time 1788839852.65 total 1127.03 last 73.47) - open/close 36922 (time 1788839862.65 total 1137.03 last 75.30) - open/close 37692 (time 1788839872.66 total 1147.03 last 76.97) - open/close 38408 (time 1788839882.66 total 1157.04 last 71.58) - open/close 39094 (time 1788839892.66 total 1167.04 last 68.58) - open/close 39846 (time 1788839902.67 total 1177.05 last 75.12) - open/close 40615 (time 1788839912.67 total 1187.05 last 76.89) - open/close 41401 (time 1788839922.68 total 1197.05 last 78.57) - open/close 42160 (time 1788839932.69 total 1207.07 last 75.78) - open/close 42904 (time 1788839942.70 total 1217.07 last 74.39) - open/close 43689 (time 1788839952.70 total 1227.08 last 78.48) - open/close 44504 (time 1788839962.71 total 1237.08 last 81.44) - open/close 45232 (time 1788839972.71 total 1247.08 last 72.80) - open/close 46030 (time 1788839982.72 total 1257.09 last 79.71) - open/close 46746 (time 1788839992.72 total 1267.10 last 71.55) - open/close 47447 (time 1788840002.74 total 1277.12 last 69.98) - open/close 48143 (time 1788840012.75 total 1287.13 last 69.55) - open/close 48809 (time 1788840022.76 total 1297.14 last 66.52) - open/close 49549 (time 1788840032.79 total 1307.16 last 73.82) total: 50000 open/close in 1313.64 seconds: 38.06 ops/second - unlinked 0 (time 1788840043 ; total 0 ; last 0) - unlinked 10000 (time 1788840190 ; total 147 ; last 147) - unlinked 20000 (time 1788840337 ; total 294 ; last 147) - unlinked 30000 (time 1788840485 ; total 442 ; last 148) - unlinked 40000 (time 1788840638 ; total 595 ; last 153) total: 50000 unlinks in 738 seconds: 67.750679 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg606-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg606-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg606-server unloading modules via unload_modules_local on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing unload_modules_local oleg606-server: kmemleak disabled modules unloaded. pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2