== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:43:18 (1788201798) start mds service on oleg221-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Aug 31 03:29 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: kmemleak disabled oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg221-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg221-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 212 (time 1788201885.24 total 10.01 last 21.18) - open/close 475 (time 1788201895.89 total 20.66 last 24.69) - open/close 679 (time 1788201905.90 total 30.67 last 20.39) - open/close 880 (time 1788201916.11 total 40.88 last 19.68) - open/close 1091 (time 1788201926.78 total 51.55 last 19.77) - open/close 1226 (time 1788201936.81 total 61.58 last 13.46) - open/close 1450 (time 1788201946.82 total 71.59 last 22.39) - open/close 1627 (time 1788201956.83 total 81.61 last 17.67) - open/close 1759 (time 1788201966.98 total 91.75 last 13.01) - open/close 1976 (time 1788201977.93 total 102.70 last 19.83) - open/close 2221 (time 1788201989.00 total 113.77 last 22.13) - open/close 2332 (time 1788201999.35 total 124.12 last 10.72) - open/close 2587 (time 1788202009.38 total 134.15 last 25.43) - open/close 2845 (time 1788202019.38 total 144.15 last 25.79) - open/close 2965 (time 1788202029.39 total 154.16 last 12.00) - open/close 3071 (time 1788202040.90 total 165.67 last 9.21) - open/close 3332 (time 1788202050.90 total 175.67 last 26.10) - open/close 3496 (time 1788202060.92 total 185.69 last 16.37) - open/close 3603 (time 1788202070.96 total 195.73 last 10.66) - open/close 3730 (time 1788202081.59 total 206.36 last 11.94) - open/close 3923 (time 1788202091.59 total 216.37 last 19.29) - open/close 4152 (time 1788202104.57 total 229.34 last 17.65) - open/close 4402 (time 1788202117.61 total 242.38 last 19.17) - open/close 4601 (time 1788202127.80 total 252.57 last 19.53) - open/close 4815 (time 1788202137.81 total 262.58 last 21.38) - open/close 4985 (time 1788202148.87 total 273.64 last 15.37) - open/close 5118 (time 1788202159.35 total 284.12 last 12.69) - open/close 5281 (time 1788202169.36 total 294.13 last 16.28) - open/close 5517 (time 1788202180.43 total 305.20 last 21.32) - open/close 5760 (time 1788202190.98 total 315.75 last 23.04) - open/close 5935 (time 1788202201.92 total 326.69 last 16.00) - open/close 6179 (time 1788202211.93 total 336.70 last 24.38) - open/close 6387 (time 1788202221.93 total 346.70 last 20.80) - open/close 6603 (time 1788202231.94 total 356.71 last 21.57) - open/close 6781 (time 1788202242.38 total 367.15 last 17.05) - open/close 7030 (time 1788202252.53 total 377.30 last 24.54) - open/close 7294 (time 1788202263.56 total 388.33 last 23.94) - open/close 7487 (time 1788202275.52 total 400.29 last 16.13) - open/close 7797 (time 1788202287.11 total 411.88 last 26.76) - open/close 8051 (time 1788202297.13 total 421.90 last 25.36) - open/close 8254 (time 1788202307.54 total 432.31 last 19.49) - open/close 8297 (time 1788202318.02 total 442.80 last 4.10) - open/close 8491 (time 1788202328.26 total 453.03 last 18.96) - open/close 8767 (time 1788202338.46 total 463.23 last 27.06) - open/close 9043 (time 1788202348.46 total 473.23 last 27.59) - open/close 9311 (time 1788202358.47 total 483.25 last 26.77) - open/close 9513 (time 1788202368.48 total 493.25 last 20.19) - open/close 9732 (time 1788202378.49 total 503.26 last 21.89) - open/close 9943 (time 1788202389.24 total 514.01 last 19.62) - open/close 10239 (time 1788202400.19 total 524.96 last 27.04) - open/close 10481 (time 1788202410.20 total 534.97 last 24.16) - open/close 10705 (time 1788202421.17 total 545.94 last 20.42) - open/close 10808 (time 1788202431.87 total 556.64 last 9.63) - open/close 10922 (time 1788202442.90 total 567.67 last 10.33) - open/close 11155 (time 1788202453.45 total 578.22 last 22.09) - open/close 11254 (time 1788202464.12 total 588.89 last 9.28) - open/close 11429 (time 1788202474.13 total 598.90 last 17.48) - open/close 11639 (time 1788202484.80 total 609.57 last 19.68) - open/close 11818 (time 1788202494.80 total 619.57 last 17.90) - open/close 12078 (time 1788202505.35 total 630.12 last 24.65) - open/close 12287 (time 1788202515.70 total 640.47 last 20.19) - open/close 12402 (time 1788202526.29 total 651.06 last 10.86) - open/close 12565 (time 1788202536.83 total 661.60 last 15.46) - open/close 12695 (time 1788202547.21 total 671.98 last 12.52) - open/close 12863 (time 1788202557.22 total 681.99 last 16.79) - open/close 13019 (time 1788202567.22 total 691.99 last 15.59) - open/close 13258 (time 1788202577.49 total 702.26 last 23.28) - open/close 13521 (time 1788202587.51 total 712.28 last 26.24) - open/close 13632 (time 1788202597.51 total 722.28 last 11.10) - open/close 13793 (time 1788202608.26 total 733.03 last 14.97) - open/close 14002 (time 1788202619.26 total 744.03 last 19.01) - open/close 14257 (time 1788202629.27 total 754.04 last 25.48) - open/close 14434 (time 1788202639.99 total 764.76 last 16.50) - open/close 14675 (time 1788202650.00 total 774.77 last 24.08) - open/close 14829 (time 1788202660.02 total 784.80 last 15.36) - open/close 14916 (time 1788202670.14 total 794.91 last 8.60) - open/close 15059 (time 1788202680.14 total 804.91 last 14.30) - open/close 15270 (time 1788202690.76 total 815.53 last 19.87) - open/close 15531 (time 1788202700.76 total 825.53 last 26.10) - open/close 15728 (time 1788202710.78 total 835.55 last 19.67) - open/close 15846 (time 1788202720.79 total 845.57 last 11.78) - open/close 16067 (time 1788202730.80 total 855.57 last 22.09) - open/close 16198 (time 1788202741.31 total 866.08 last 12.47) - open/close 16333 (time 1788202751.32 total 876.09 last 13.49) - open/close 16442 (time 1788202762.24 total 887.01 last 9.98) - open/close 16637 (time 1788202773.00 total 897.77 last 18.13) - open/close 16847 (time 1788202783.01 total 907.78 last 20.99) - open/close 17036 (time 1788202793.02 total 917.79 last 18.87) - open/close 17210 (time 1788202804.86 total 929.63 last 14.70) - open/close 17405 (time 1788202814.88 total 939.65 last 19.45) - open/close 17571 (time 1788202825.39 total 950.17 last 15.79) - open/close 17637 (time 1788202837.26 total 962.03 last 5.56) - open/close 17750 (time 1788202849.22 total 973.99 last 9.45) - open/close 17899 (time 1788202859.33 total 984.10 last 14.74) - open/close 17992 (time 1788202871.25 total 996.02 last 7.80) - open/close 18216 (time 1788202881.26 total 1006.03 last 22.38) - open/close 18460 (time 1788202891.27 total 1016.04 last 24.36) - open/close 19170 (time 1788202901.28 total 1026.05 last 70.94) - open/close 19908 (time 1788202911.29 total 1036.06 last 73.75) - open/close 20649 (time 1788202921.30 total 1046.07 last 74.02) - open/close 21369 (time 1788202931.31 total 1056.08 last 71.95) - open/close 22157 (time 1788202941.31 total 1066.08 last 78.79) - open/close 22821 (time 1788202951.32 total 1076.09 last 66.34) - open/close 23553 (time 1788202961.32 total 1086.10 last 73.14) - open/close 24248 (time 1788202971.34 total 1096.11 last 69.42) - open/close 24874 (time 1788202981.35 total 1106.12 last 62.53) - open/close 25585 (time 1788202991.36 total 1116.13 last 71.03) - open/close 26315 (time 1788203001.38 total 1126.15 last 72.82) - open/close 27056 (time 1788203011.38 total 1136.15 last 74.09) - open/close 27799 (time 1788203021.39 total 1146.16 last 74.26) - open/close 28541 (time 1788203031.41 total 1156.18 last 74.05) - open/close 29244 (time 1788203041.41 total 1166.18 last 70.30) - open/close 30000 (time 1788203050.73 total 1175.50 last 81.13) - open/close 30622 (time 1788203060.73 total 1185.50 last 62.18) - open/close 31273 (time 1788203070.74 total 1195.51 last 65.04) - open/close 31878 (time 1788203080.75 total 1205.52 last 60.44) - open/close 32597 (time 1788203090.75 total 1215.52 last 71.89) - open/close 33231 (time 1788203100.78 total 1225.56 last 63.20) - open/close 33784 (time 1788203110.79 total 1235.56 last 55.27) - open/close 34532 (time 1788203120.79 total 1245.56 last 74.79) - open/close 35175 (time 1788203130.79 total 1255.57 last 64.28) - open/close 35945 (time 1788203140.82 total 1265.59 last 76.80) - open/close 36532 (time 1788203150.83 total 1275.60 last 58.64) - open/close 37117 (time 1788203160.83 total 1285.61 last 58.48) - open/close 37857 (time 1788203170.84 total 1295.61 last 73.96) - open/close 38565 (time 1788203180.84 total 1305.61 last 70.78) - open/close 39368 (time 1788203190.87 total 1315.64 last 80.07) - open/close 40000 (time 1788203198.08 total 1322.85 last 87.67) - open/close 40678 (time 1788203208.09 total 1332.86 last 67.74) - open/close 41431 (time 1788203218.09 total 1342.87 last 75.27) - open/close 42224 (time 1788203228.11 total 1352.88 last 79.20) - open/close 42988 (time 1788203238.12 total 1362.90 last 76.27) - open/close 43604 (time 1788203248.13 total 1372.90 last 61.58) - open/close 44310 (time 1788203258.14 total 1382.91 last 70.53) - open/close 45006 (time 1788203268.14 total 1392.91 last 69.58) - open/close 45691 (time 1788203278.15 total 1402.92 last 68.45) - open/close 46423 (time 1788203288.16 total 1412.93 last 73.14) - open/close 47127 (time 1788203298.17 total 1422.95 last 70.28) - open/close 47957 (time 1788203308.21 total 1432.98 last 82.72) - open/close 48942 (time 1788203318.21 total 1442.99 last 98.44) - open/close 49777 (time 1788203328.22 total 1452.99 last 83.47) total: 50000 open/close in 1455.86 seconds: 34.34 ops/second - unlinked 0 (time 1788203335 ; total 0 ; last 0) - unlinked 10000 (time 1788203473 ; total 138 ; last 138) - unlinked 20000 (time 1788203608 ; total 273 ; last 135) - unlinked 30000 (time 1788203758 ; total 423 ; last 150) - unlinked 40000 (time 1788203909 ; total 574 ; last 151) total: 50000 unlinks in 699 seconds: 71.530762 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: kmemleak disabled modules unloaded. pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2