== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:46:50 (1788526010) start mds service on oleg222-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 3 15:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: kmemleak disabled oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg222-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 209 (time 1788526120.37 total 10.04 last 20.83) - open/close 395 (time 1788526131.26 total 20.92 last 17.08) - open/close 644 (time 1788526141.27 total 30.93 last 24.87) - open/close 742 (time 1788526151.64 total 41.31 last 9.44) - open/close 886 (time 1788526161.71 total 51.37 last 14.31) - open/close 959 (time 1788526172.20 total 61.87 last 6.96) - open/close 1159 (time 1788526182.21 total 71.88 last 19.98) - open/close 1291 (time 1788526192.94 total 82.60 last 12.31) - open/close 1525 (time 1788526203.81 total 93.48 last 21.52) - open/close 1702 (time 1788526213.85 total 103.52 last 17.63) - open/close 1932 (time 1788526223.86 total 113.53 last 22.97) - open/close 2107 (time 1788526234.66 total 124.32 last 16.21) - open/close 2269 (time 1788526244.73 total 134.39 last 16.09) - open/close 2464 (time 1788526254.97 total 144.64 last 19.03) - open/close 2654 (time 1788526264.98 total 154.64 last 18.99) - open/close 2732 (time 1788526275.00 total 164.66 last 7.78) - open/close 2786 (time 1788526286.90 total 176.57 last 4.54) - open/close 2933 (time 1788526296.92 total 186.59 last 14.67) - open/close 3101 (time 1788526306.93 total 196.60 last 16.78) - open/close 3322 (time 1788526316.95 total 206.61 last 22.07) - open/close 3525 (time 1788526327.16 total 216.82 last 19.88) - open/close 3701 (time 1788526337.19 total 226.86 last 17.54) - open/close 3866 (time 1788526347.48 total 237.15 last 16.03) - open/close 4075 (time 1788526358.27 total 247.93 last 19.38) - open/close 4228 (time 1788526369.06 total 258.72 last 14.18) - open/close 4470 (time 1788526379.07 total 268.73 last 24.18) - open/close 4599 (time 1788526389.07 total 278.74 last 12.89) - open/close 4731 (time 1788526399.45 total 289.11 last 12.72) - open/close 4907 (time 1788526409.45 total 299.12 last 17.59) - open/close 5135 (time 1788526419.47 total 309.13 last 22.77) - open/close 5243 (time 1788526429.60 total 319.27 last 10.66) - open/close 5392 (time 1788526440.60 total 330.26 last 13.55) - open/close 5566 (time 1788526451.24 total 340.90 last 16.36) - open/close 5719 (time 1788526461.33 total 351.00 last 15.15) - open/close 5835 (time 1788526472.02 total 361.68 last 10.86) - open/close 6063 (time 1788526483.95 total 373.61 last 19.11) - open/close 6308 (time 1788526493.95 total 383.62 last 24.49) - open/close 6486 (time 1788526504.22 total 393.88 last 17.34) - open/close 6718 (time 1788526515.81 total 405.48 last 20.01) - open/close 6937 (time 1788526526.75 total 416.41 last 20.03) - open/close 7025 (time 1788526537.91 total 427.58 last 7.88) - open/close 7094 (time 1788526548.18 total 437.85 last 6.72) - open/close 7336 (time 1788526558.25 total 447.92 last 24.03) - open/close 7485 (time 1788526569.32 total 458.98 last 13.47) - open/close 7733 (time 1788526579.32 total 468.98 last 24.79) - open/close 7960 (time 1788526589.98 total 479.65 last 21.29) - open/close 8146 (time 1788526599.98 total 489.65 last 18.60) - open/close 8400 (time 1788526610.00 total 499.66 last 25.36) - open/close 8676 (time 1788526620.70 total 510.36 last 25.79) - open/close 8904 (time 1788526630.88 total 520.54 last 22.40) - open/close 9110 (time 1788526641.67 total 531.33 last 19.09) - open/close 9330 (time 1788526652.39 total 542.06 last 20.51) - open/close 9599 (time 1788526662.41 total 552.07 last 26.86) - open/close 9803 (time 1788526673.44 total 563.11 last 18.48) - open/close 9965 (time 1788526683.74 total 573.40 last 15.73) - open/close 10000 (time 1788526690.12 total 579.78 last 5.49) - open/close 10087 (time 1788526700.26 total 589.93 last 8.58) - open/close 10302 (time 1788526710.27 total 599.94 last 21.47) - open/close 10558 (time 1788526721.19 total 610.85 last 23.46) - open/close 10699 (time 1788526731.44 total 621.10 last 13.76) - open/close 10874 (time 1788526742.38 total 632.05 last 15.98) - open/close 10990 (time 1788526752.47 total 642.13 last 11.50) - open/close 11164 (time 1788526762.47 total 652.13 last 17.40) - open/close 11340 (time 1788526772.47 total 662.13 last 17.60) - open/close 11578 (time 1788526783.26 total 672.92 last 22.06) - open/close 11855 (time 1788526793.27 total 682.94 last 27.66) - open/close 12080 (time 1788526803.80 total 693.47 last 21.36) - open/close 12334 (time 1788526813.82 total 703.48 last 25.36) - open/close 12617 (time 1788526823.84 total 713.50 last 28.25) - open/close 12841 (time 1788526834.04 total 723.70 last 21.96) - open/close 13038 (time 1788526844.04 total 733.70 last 19.70) - open/close 13267 (time 1788526854.30 total 743.97 last 22.32) - open/close 13503 (time 1788526864.31 total 753.97 last 23.58) - open/close 13719 (time 1788526874.59 total 764.26 last 21.00) - open/close 13900 (time 1788526884.96 total 774.63 last 17.45) - open/close 14165 (time 1788526895.27 total 784.94 last 25.70) - open/close 14423 (time 1788526905.83 total 795.49 last 24.45) - open/close 14704 (time 1788526915.83 total 805.49 last 28.10) - open/close 14933 (time 1788526926.20 total 815.87 last 22.07) - open/close 15141 (time 1788526936.21 total 825.88 last 20.78) - open/close 15427 (time 1788526946.22 total 835.89 last 28.57) - open/close 15666 (time 1788526956.95 total 846.62 last 22.28) - open/close 15752 (time 1788526967.30 total 856.96 last 8.31) - open/close 15898 (time 1788526977.46 total 867.12 last 14.37) - open/close 16106 (time 1788526989.93 total 879.59 last 16.68) - open/close 16357 (time 1788526999.94 total 889.61 last 25.07) - open/close 16595 (time 1788527009.94 total 899.61 last 23.80) - open/close 16835 (time 1788527020.25 total 909.91 last 23.29) - open/close 17061 (time 1788527030.26 total 919.92 last 22.58) - open/close 17231 (time 1788527040.48 total 930.15 last 16.63) - open/close 17519 (time 1788527051.18 total 940.84 last 26.93) - open/close 17741 (time 1788527061.19 total 950.86 last 22.17) - open/close 17965 (time 1788527071.96 total 961.63 last 20.79) - open/close 18206 (time 1788527082.71 total 972.38 last 22.42) - open/close 18479 (time 1788527093.04 total 982.71 last 26.43) - open/close 18637 (time 1788527103.05 total 992.71 last 15.79) - open/close 18843 (time 1788527113.14 total 1002.80 last 20.41) - open/close 19238 (time 1788527123.15 total 1012.81 last 39.47) - open/close 20000 (time 1788527132.53 total 1022.20 last 81.20) - open/close 20906 (time 1788527142.54 total 1032.20 last 90.53) - open/close 21855 (time 1788527152.55 total 1042.22 last 94.78) - open/close 22666 (time 1788527162.57 total 1052.23 last 80.97) - open/close 23473 (time 1788527172.57 total 1062.23 last 80.69) - open/close 24279 (time 1788527182.58 total 1072.24 last 80.52) - open/close 25276 (time 1788527192.58 total 1082.24 last 99.70) - open/close 26358 (time 1788527202.58 total 1092.25 last 108.19) - open/close 27308 (time 1788527212.58 total 1102.25 last 95.00) - open/close 28302 (time 1788527222.58 total 1112.25 last 99.36) - open/close 29242 (time 1788527232.59 total 1122.25 last 93.96) - open/close 30000 (time 1788527241.66 total 1131.33 last 83.55) - open/close 30794 (time 1788527251.67 total 1141.33 last 79.37) - open/close 31586 (time 1788527261.67 total 1151.34 last 79.15) - open/close 32406 (time 1788527271.68 total 1161.35 last 81.92) - open/close 33290 (time 1788527281.68 total 1171.35 last 88.37) - open/close 34143 (time 1788527291.70 total 1181.36 last 85.19) - open/close 34979 (time 1788527301.71 total 1191.37 last 83.54) - open/close 35812 (time 1788527311.71 total 1201.37 last 83.28) - open/close 36683 (time 1788527321.71 total 1211.38 last 87.05) - open/close 37614 (time 1788527331.72 total 1221.38 last 93.07) - open/close 38528 (time 1788527341.72 total 1231.38 last 91.38) - open/close 39390 (time 1788527351.73 total 1241.39 last 86.12) - open/close 40000 (time 1788527359.83 total 1249.49 last 75.32) - open/close 40718 (time 1788527369.84 total 1259.51 last 71.70) - open/close 41620 (time 1788527379.85 total 1269.52 last 90.11) - open/close 42519 (time 1788527389.86 total 1279.53 last 89.83) - open/close 43424 (time 1788527399.86 total 1289.53 last 90.47) - open/close 44285 (time 1788527409.87 total 1299.53 last 86.06) - open/close 45137 (time 1788527419.87 total 1309.54 last 85.15) - open/close 45927 (time 1788527429.88 total 1319.54 last 78.96) - open/close 46708 (time 1788527439.89 total 1329.55 last 78.04) - open/close 47457 (time 1788527449.90 total 1339.57 last 74.79) - open/close 48250 (time 1788527459.91 total 1349.57 last 79.23) - open/close 49136 (time 1788527469.92 total 1359.59 last 88.49) - open/close 49914 (time 1788527479.92 total 1369.59 last 77.77) total: 50000 open/close in 1370.62 seconds: 36.48 ops/second - unlinked 0 (time 1788527485 ; total 0 ; last 0) - unlinked 10000 (time 1788527613 ; total 128 ; last 128) - unlinked 20000 (time 1788527738 ; total 253 ; last 125) - unlinked 30000 (time 1788527865 ; total 380 ; last 127) - unlinked 40000 (time 1788528003 ; total 518 ; last 138) total: 50000 unlinks in 647 seconds: 77.279755 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: kmemleak disabled modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2