== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:48:09 (1788943689) start mds service on oleg260-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 8 20:11 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: kmemleak disabled oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg260-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 108 (time 1788943783.88 total 10.03 last 10.76) - open/close 288 (time 1788943793.88 total 20.04 last 17.99) - open/close 472 (time 1788943803.89 total 30.05 last 18.39) - open/close 759 (time 1788943814.87 total 41.03 last 26.13) - open/close 901 (time 1788943824.87 total 51.03 last 14.20) - open/close 1116 (time 1788943834.89 total 61.05 last 21.47) - open/close 1208 (time 1788943845.64 total 71.80 last 8.56) - open/close 1321 (time 1788943855.95 total 82.11 last 10.96) - open/close 1569 (time 1788943865.98 total 92.13 last 24.74) - open/close 1768 (time 1788943876.04 total 102.20 last 19.77) - open/close 2036 (time 1788943886.49 total 112.65 last 25.64) - open/close 2218 (time 1788943897.39 total 123.54 last 16.71) - open/close 2490 (time 1788943907.40 total 133.56 last 27.16) - open/close 2701 (time 1788943917.41 total 143.57 last 21.08) - open/close 2951 (time 1788943928.15 total 154.31 last 23.28) - open/close 3177 (time 1788943938.83 total 164.98 last 21.17) - open/close 3358 (time 1788943948.84 total 175.00 last 18.08) - open/close 3508 (time 1788943959.50 total 185.66 last 14.06) - open/close 3678 (time 1788943970.07 total 196.23 last 16.09) - open/close 3847 (time 1788943980.50 total 206.66 last 16.20) - open/close 4057 (time 1788943990.50 total 216.66 last 20.99) - open/close 4259 (time 1788944000.51 total 226.67 last 20.19) - open/close 4475 (time 1788944010.62 total 236.77 last 21.38) - open/close 4687 (time 1788944020.65 total 246.81 last 21.13) - open/close 4930 (time 1788944030.74 total 256.90 last 24.08) - open/close 5134 (time 1788944041.32 total 267.48 last 19.28) - open/close 5418 (time 1788944052.24 total 278.40 last 26.02) - open/close 5664 (time 1788944063.07 total 289.23 last 22.70) - open/close 5738 (time 1788944075.43 total 301.59 last 5.99) - open/close 5986 (time 1788944085.46 total 311.62 last 24.73) - open/close 6187 (time 1788944096.55 total 322.70 last 18.13) - open/close 6255 (time 1788944106.74 total 332.89 last 6.67) - open/close 6418 (time 1788944116.74 total 342.90 last 16.29) - open/close 6589 (time 1788944127.52 total 353.68 last 15.87) - open/close 6865 (time 1788944137.53 total 363.69 last 27.57) - open/close 7130 (time 1788944148.31 total 374.47 last 24.59) - open/close 7375 (time 1788944158.60 total 384.76 last 23.80) - open/close 7586 (time 1788944168.64 total 394.80 last 21.03) - open/close 7788 (time 1788944179.68 total 405.84 last 18.29) - open/close 7933 (time 1788944191.18 total 417.33 last 12.61) - open/close 8062 (time 1788944202.07 total 428.23 last 11.84) - open/close 8302 (time 1788944212.94 total 439.10 last 22.08) - open/close 8497 (time 1788944224.02 total 450.17 last 17.61) - open/close 8786 (time 1788944234.76 total 460.92 last 26.89) - open/close 9031 (time 1788944244.77 total 470.92 last 24.49) - open/close 9267 (time 1788944254.80 total 480.96 last 23.52) - open/close 9543 (time 1788944264.81 total 490.97 last 27.58) - open/close 9752 (time 1788944274.83 total 500.98 last 20.87) - open/close 10000 (time 1788944283.79 total 509.95 last 27.66) - open/close 10172 (time 1788944294.74 total 520.90 last 15.70) - open/close 10436 (time 1788944304.84 total 531.00 last 26.15) - open/close 10657 (time 1788944314.95 total 541.11 last 21.86) - open/close 10899 (time 1788944324.96 total 551.12 last 24.18) - open/close 11084 (time 1788944335.13 total 561.29 last 18.19) - open/close 11299 (time 1788944345.13 total 571.29 last 21.50) - open/close 11587 (time 1788944355.14 total 581.30 last 28.77) - open/close 11853 (time 1788944365.16 total 591.32 last 26.54) - open/close 12069 (time 1788944376.26 total 602.42 last 19.45) - open/close 12334 (time 1788944386.27 total 612.43 last 26.49) - open/close 12498 (time 1788944396.63 total 622.79 last 15.83) - open/close 12739 (time 1788944406.99 total 633.15 last 23.27) - open/close 12959 (time 1788944417.16 total 643.32 last 21.63) - open/close 13216 (time 1788944427.61 total 653.76 last 24.60) - open/close 13470 (time 1788944438.67 total 664.82 last 22.97) - open/close 13742 (time 1788944449.49 total 675.65 last 25.13) - open/close 13905 (time 1788944460.53 total 686.69 last 14.76) - open/close 14162 (time 1788944470.54 total 696.70 last 25.68) - open/close 14373 (time 1788944481.12 total 707.27 last 19.96) - open/close 14627 (time 1788944492.11 total 718.27 last 23.10) - open/close 14858 (time 1788944502.61 total 728.77 last 22.00) - open/close 15080 (time 1788944512.62 total 738.78 last 22.18) - open/close 15286 (time 1788944523.54 total 749.70 last 18.87) - open/close 15434 (time 1788944534.62 total 760.78 last 13.36) - open/close 15674 (time 1788944545.25 total 771.41 last 22.58) - open/close 15960 (time 1788944556.04 total 782.20 last 26.49) - open/close 16193 (time 1788944566.36 total 792.52 last 22.59) - open/close 16419 (time 1788944576.83 total 802.99 last 21.58) - open/close 16644 (time 1788944586.83 total 812.99 last 22.50) - open/close 16832 (time 1788944596.85 total 823.01 last 18.76) - open/close 17061 (time 1788944607.44 total 833.60 last 21.62) - open/close 17313 (time 1788944620.32 total 846.48 last 19.57) - open/close 17571 (time 1788944630.33 total 856.49 last 25.77) - open/close 17827 (time 1788944640.34 total 866.50 last 25.59) - open/close 18041 (time 1788944651.54 total 877.70 last 19.10) - open/close 18312 (time 1788944661.55 total 887.71 last 27.07) - open/close 18501 (time 1788944673.61 total 899.77 last 15.67) - open/close 18766 (time 1788944683.61 total 909.77 last 26.49) - open/close 18964 (time 1788944693.63 total 919.78 last 19.78) - open/close 19247 (time 1788944703.63 total 929.79 last 28.28) - open/close 20000 (time 1788944711.89 total 938.05 last 91.17) - open/close 20851 (time 1788944721.90 total 948.06 last 85.05) - open/close 21923 (time 1788944731.90 total 958.06 last 107.18) - open/close 22782 (time 1788944741.90 total 968.06 last 85.89) - open/close 23571 (time 1788944751.91 total 978.06 last 78.87) - open/close 24402 (time 1788944761.91 total 988.07 last 83.05) - open/close 25260 (time 1788944771.92 total 998.08 last 85.75) - open/close 26068 (time 1788944781.92 total 1008.08 last 80.76) - open/close 26920 (time 1788944791.92 total 1018.08 last 85.20) - open/close 27907 (time 1788944801.93 total 1028.09 last 98.66) - open/close 28735 (time 1788944811.93 total 1038.09 last 82.77) - open/close 29527 (time 1788944821.94 total 1048.09 last 79.17) - open/close 30000 (time 1788944828.46 total 1054.62 last 72.49) - open/close 30653 (time 1788944838.46 total 1064.62 last 65.28) - open/close 31409 (time 1788944848.47 total 1074.63 last 75.56) - open/close 32207 (time 1788944858.48 total 1084.64 last 79.73) - open/close 33042 (time 1788944868.49 total 1094.65 last 83.38) - open/close 33924 (time 1788944878.50 total 1104.66 last 88.09) - open/close 34779 (time 1788944888.51 total 1114.67 last 85.43) - open/close 35665 (time 1788944898.51 total 1124.67 last 88.58) - open/close 36582 (time 1788944908.52 total 1134.67 last 91.67) - open/close 37488 (time 1788944918.52 total 1144.68 last 90.55) - open/close 38375 (time 1788944928.53 total 1154.68 last 88.66) - open/close 39215 (time 1788944938.54 total 1164.70 last 83.88) - open/close 40000 (time 1788944947.67 total 1173.83 last 85.96) - open/close 40722 (time 1788944957.68 total 1183.84 last 72.16) - open/close 41470 (time 1788944967.68 total 1193.84 last 74.77) - open/close 42278 (time 1788944977.69 total 1203.84 last 80.77) - open/close 43034 (time 1788944987.69 total 1213.85 last 75.56) - open/close 43851 (time 1788944997.70 total 1223.86 last 81.61) - open/close 44728 (time 1788945007.71 total 1233.87 last 87.64) - open/close 45573 (time 1788945017.71 total 1243.87 last 84.44) - open/close 46458 (time 1788945027.73 total 1253.89 last 88.35) - open/close 47186 (time 1788945037.73 total 1263.89 last 72.78) - open/close 48033 (time 1788945047.74 total 1273.89 last 84.67) - open/close 48713 (time 1788945057.74 total 1283.90 last 67.95) - open/close 49532 (time 1788945067.75 total 1293.90 last 81.88) total: 50000 open/close in 1300.01 seconds: 38.46 ops/second - unlinked 0 (time 1788945078 ; total 0 ; last 0) - unlinked 10000 (time 1788945203 ; total 125 ; last 125) - unlinked 20000 (time 1788945329 ; total 251 ; last 126) - unlinked 30000 (time 1788945449 ; total 371 ; last 120) - unlinked 40000 (time 1788945568 ; total 490 ; last 119) total: 50000 unlinks in 613 seconds: 81.566071 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: kmemleak disabled modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2