== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:22:12 (1788441732) start mds service on oleg204-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: kmemleak disabled oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg204-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 73 (time 1788441829.43 total 10.23 last 7.14) - open/close 257 (time 1788441839.66 total 20.47 last 17.97) - open/close 354 (time 1788441849.96 total 30.76 last 9.42) - open/close 481 (time 1788441859.97 total 40.77 last 12.69) - open/close 607 (time 1788441869.98 total 50.78 last 12.59) - open/close 744 (time 1788441880.81 total 61.61 last 12.65) - open/close 873 (time 1788441891.36 total 72.17 last 12.23) - open/close 1026 (time 1788441901.37 total 82.17 last 15.29) - open/close 1178 (time 1788441912.28 total 93.09 last 13.93) - open/close 1314 (time 1788441923.19 total 103.99 last 12.47) - open/close 1511 (time 1788441933.68 total 114.48 last 18.78) - open/close 1745 (time 1788441944.64 total 125.44 last 21.35) - open/close 1937 (time 1788441955.18 total 135.98 last 18.22) - open/close 2135 (time 1788441965.18 total 145.99 last 19.78) - open/close 2345 (time 1788441975.83 total 156.64 last 19.72) - open/close 2448 (time 1788441985.84 total 166.64 last 10.29) - open/close 2567 (time 1788441995.99 total 176.80 last 11.72) - open/close 2789 (time 1788442006.63 total 187.43 last 20.88) - open/close 3068 (time 1788442016.64 total 197.45 last 27.86) - open/close 3260 (time 1788442026.89 total 207.69 last 18.74) - open/close 3483 (time 1788442037.84 total 218.65 last 20.36) - open/close 3685 (time 1788442048.12 total 228.92 last 19.67) - open/close 3891 (time 1788442059.06 total 239.86 last 18.82) - open/close 4049 (time 1788442069.29 total 250.10 last 15.44) - open/close 4211 (time 1788442084.19 total 265.00 last 10.87) - open/close 4398 (time 1788442094.55 total 275.35 last 18.06) - open/close 4598 (time 1788442105.12 total 285.92 last 18.92) - open/close 4812 (time 1788442115.12 total 295.93 last 21.39) - open/close 4980 (time 1788442125.21 total 306.01 last 16.65) - open/close 5173 (time 1788442136.03 total 316.84 last 17.83) - open/close 5354 (time 1788442146.05 total 326.86 last 18.06) - open/close 5453 (time 1788442156.75 total 337.55 last 9.26) - open/close 5669 (time 1788442167.30 total 348.11 last 20.46) - open/close 5892 (time 1788442178.33 total 359.13 last 20.23) - open/close 6099 (time 1788442188.33 total 369.14 last 20.69) - open/close 6233 (time 1788442198.58 total 379.38 last 13.08) - open/close 6437 (time 1788442209.44 total 390.24 last 18.79) - open/close 6582 (time 1788442219.76 total 400.56 last 14.04) - open/close 6775 (time 1788442229.93 total 410.73 last 18.98) - open/close 6979 (time 1788442240.29 total 421.10 last 19.68) - open/close 7062 (time 1788442250.66 total 431.46 last 8.01) - open/close 7234 (time 1788442260.67 total 441.47 last 17.19) - open/close 7507 (time 1788442271.45 total 452.25 last 25.31) - open/close 7642 (time 1788442281.69 total 462.49 last 13.19) - open/close 7724 (time 1788442291.88 total 472.69 last 8.04) - open/close 7873 (time 1788442304.11 total 484.91 last 12.19) - open/close 8035 (time 1788442314.62 total 495.42 last 15.41) - open/close 8218 (time 1788442324.71 total 505.51 last 18.14) - open/close 8412 (time 1788442336.43 total 517.23 last 16.56) - open/close 8542 (time 1788442346.85 total 527.66 last 12.47) - open/close 8647 (time 1788442357.67 total 538.47 last 9.71) - open/close 8760 (time 1788442367.67 total 548.48 last 11.29) - open/close 8919 (time 1788442377.68 total 558.48 last 15.89) - open/close 9110 (time 1788442388.44 total 569.25 last 17.75) - open/close 9278 (time 1788442399.53 total 580.33 last 15.15) - open/close 9396 (time 1788442410.53 total 591.33 last 10.73) - open/close 9628 (time 1788442420.85 total 601.65 last 22.49) - open/close 9755 (time 1788442430.85 total 611.65 last 12.69) - open/close 9832 (time 1788442442.14 total 622.95 last 6.82) - open/close 9909 (time 1788442452.15 total 632.96 last 7.69) - open/close 10000 (time 1788442457.07 total 637.87 last 18.51) - open/close 10105 (time 1788442467.18 total 647.99 last 10.38) - open/close 10257 (time 1788442479.34 total 660.14 last 12.50) - open/close 10449 (time 1788442491.54 total 672.35 last 15.73) - open/close 10629 (time 1788442501.56 total 682.36 last 17.98) - open/close 10823 (time 1788442511.58 total 692.38 last 19.36) - open/close 11042 (time 1788442521.90 total 702.70 last 21.23) - open/close 11299 (time 1788442531.92 total 712.72 last 25.64) - open/close 11494 (time 1788442542.81 total 723.61 last 17.91) - open/close 11756 (time 1788442553.79 total 734.59 last 23.86) - open/close 11959 (time 1788442563.87 total 744.67 last 20.13) - open/close 12216 (time 1788442574.82 total 755.62 last 23.47) - open/close 12415 (time 1788442585.00 total 765.81 last 19.54) - open/close 12541 (time 1788442595.01 total 775.81 last 12.60) - open/close 12737 (time 1788442605.01 total 785.82 last 19.58) - open/close 12879 (time 1788442615.13 total 795.93 last 14.04) - open/close 13109 (time 1788442626.25 total 807.05 last 20.69) - open/close 13368 (time 1788442637.29 total 818.09 last 23.45) - open/close 13572 (time 1788442647.66 total 828.46 last 19.68) - open/close 13733 (time 1788442657.71 total 838.51 last 16.02) - open/close 13850 (time 1788442667.76 total 848.57 last 11.64) - open/close 14035 (time 1788442680.15 total 860.96 last 14.93) - open/close 14139 (time 1788442691.84 total 872.64 last 8.90) - open/close 14233 (time 1788442701.84 total 882.65 last 9.39) - open/close 14322 (time 1788442712.48 total 893.28 last 8.37) - open/close 14568 (time 1788442722.66 total 903.46 last 24.17) - open/close 14759 (time 1788442733.62 total 914.42 last 17.43) - open/close 14976 (time 1788442743.63 total 924.43 last 21.68) - open/close 15177 (time 1788442756.06 total 936.87 last 16.16) - open/close 15422 (time 1788442766.07 total 946.87 last 24.49) - open/close 15685 (time 1788442776.08 total 956.89 last 26.26) - open/close 15894 (time 1788442786.97 total 967.78 last 19.19) - open/close 16124 (time 1788442796.99 total 977.79 last 22.96) - open/close 16262 (time 1788442807.02 total 987.82 last 13.76) - open/close 16497 (time 1788442817.25 total 998.06 last 22.96) - open/close 16735 (time 1788442827.26 total 1008.06 last 23.79) - open/close 16971 (time 1788442838.26 total 1019.06 last 21.45) - open/close 17298 (time 1788442848.26 total 1029.07 last 32.69) - open/close 18098 (time 1788442858.27 total 1039.07 last 79.97) - open/close 18850 (time 1788442868.28 total 1049.08 last 75.15) - open/close 19525 (time 1788442878.28 total 1059.08 last 67.48) - open/close 20000 (time 1788442885.80 total 1066.60 last 63.14) - open/close 20684 (time 1788442895.81 total 1076.62 last 68.31) - open/close 21380 (time 1788442905.83 total 1086.63 last 69.49) - open/close 22146 (time 1788442915.84 total 1096.64 last 76.51) - open/close 22820 (time 1788442925.84 total 1106.65 last 67.38) - open/close 23523 (time 1788442935.84 total 1116.65 last 70.30) - open/close 24177 (time 1788442945.85 total 1126.65 last 65.38) - open/close 24883 (time 1788442955.85 total 1136.65 last 70.59) - open/close 25587 (time 1788442965.86 total 1146.66 last 70.31) - open/close 26293 (time 1788442975.87 total 1156.68 last 70.52) - open/close 27008 (time 1788442985.88 total 1166.69 last 71.43) - open/close 27695 (time 1788442995.89 total 1176.69 last 68.66) - open/close 28435 (time 1788443005.89 total 1186.70 last 73.97) - open/close 29125 (time 1788443015.90 total 1196.70 last 68.98) - open/close 29892 (time 1788443025.90 total 1206.70 last 76.67) - open/close 30521 (time 1788443035.92 total 1216.73 last 62.75) - open/close 31132 (time 1788443045.95 total 1226.75 last 60.96) - open/close 31689 (time 1788443055.96 total 1236.76 last 55.65) - open/close 32293 (time 1788443065.96 total 1246.76 last 60.37) - open/close 33010 (time 1788443075.97 total 1256.77 last 71.65) - open/close 33577 (time 1788443085.98 total 1266.79 last 56.62) - open/close 34227 (time 1788443096.00 total 1276.80 last 64.88) - open/close 34898 (time 1788443106.01 total 1286.81 last 67.03) - open/close 35655 (time 1788443116.03 total 1296.83 last 75.58) - open/close 36383 (time 1788443126.03 total 1306.83 last 72.80) - open/close 37039 (time 1788443136.04 total 1316.84 last 65.55) - open/close 37655 (time 1788443146.05 total 1326.85 last 61.52) - open/close 38253 (time 1788443156.05 total 1336.86 last 59.77) - open/close 38928 (time 1788443166.06 total 1346.87 last 67.43) - open/close 39620 (time 1788443176.06 total 1356.87 last 69.19) - open/close 40000 (time 1788443181.38 total 1362.18 last 71.51) - open/close 40569 (time 1788443191.40 total 1372.20 last 56.80) - open/close 41044 (time 1788443201.40 total 1382.21 last 47.46) - open/close 41646 (time 1788443211.42 total 1392.22 last 60.12) - open/close 42320 (time 1788443221.43 total 1402.23 last 67.33) - open/close 42828 (time 1788443231.43 total 1412.23 last 50.79) - open/close 43487 (time 1788443241.45 total 1422.26 last 65.74) - open/close 44128 (time 1788443251.47 total 1432.27 last 64.01) - open/close 44821 (time 1788443261.48 total 1442.29 last 69.20) - open/close 45434 (time 1788443271.49 total 1452.29 last 61.25) - open/close 46064 (time 1788443281.50 total 1462.30 last 62.96) - open/close 46737 (time 1788443291.50 total 1472.31 last 67.25) - open/close 47334 (time 1788443301.51 total 1482.32 last 59.64) - open/close 47830 (time 1788443311.54 total 1492.35 last 49.46) - open/close 48350 (time 1788443321.54 total 1502.35 last 52.00) - open/close 49001 (time 1788443331.59 total 1512.40 last 64.79) - open/close 49537 (time 1788443341.60 total 1522.40 last 53.57) total: 50000 open/close in 1530.40 seconds: 32.67 ops/second - unlinked 0 (time 1788443354 ; total 0 ; last 0) - unlinked 10000 (time 1788443527 ; total 173 ; last 173) - unlinked 20000 (time 1788443696 ; total 342 ; last 169) - unlinked 30000 (time 1788443874 ; total 520 ; last 178) - unlinked 40000 (time 1788444043 ; total 689 ; last 169) total: 50000 unlinks in 844 seconds: 59.241707 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: kmemleak disabled modules unloaded. pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2