== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:14:10 (1788588850) start mds service on oleg645-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 09:57 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: kmemleak disabled oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg645-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg645-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg645-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 273 (time 1788588927.61 total 10.88 last 25.08) - open/close 550 (time 1788588937.62 total 20.90 last 27.67) - open/close 797 (time 1788588947.63 total 30.90 last 24.69) - open/close 1056 (time 1788588958.58 total 41.85 last 23.66) - open/close 1344 (time 1788588968.88 total 52.15 last 27.96) - open/close 1565 (time 1788588979.89 total 63.17 last 20.06) - open/close 1876 (time 1788588990.85 total 74.12 last 28.38) - open/close 2133 (time 1788589000.86 total 84.13 last 25.69) - open/close 2388 (time 1788589010.86 total 94.13 last 25.50) - open/close 2617 (time 1788589020.96 total 104.23 last 22.66) - open/close 2880 (time 1788589031.22 total 114.49 last 25.64) - open/close 3211 (time 1788589041.23 total 124.50 last 33.07) - open/close 3457 (time 1788589051.23 total 134.50 last 24.60) - open/close 3726 (time 1788589061.35 total 144.63 last 26.58) - open/close 3977 (time 1788589071.41 total 154.68 last 24.96) - open/close 4228 (time 1788589081.42 total 164.69 last 25.08) - open/close 4537 (time 1788589091.42 total 174.69 last 30.89) - open/close 4816 (time 1788589101.47 total 184.74 last 27.78) - open/close 5010 (time 1788589111.85 total 195.13 last 18.68) - open/close 5100 (time 1788589122.87 total 206.14 last 8.17) - open/close 5360 (time 1788589138.19 total 221.46 last 16.97) - open/close 5525 (time 1788589148.42 total 231.69 last 16.13) - open/close 5677 (time 1788589159.54 total 242.81 last 13.67) - open/close 5958 (time 1788589169.61 total 252.88 last 27.91) - open/close 6205 (time 1788589180.54 total 263.81 last 22.60) - open/close 6381 (time 1788589190.55 total 273.82 last 17.58) - open/close 6540 (time 1788589200.71 total 283.98 last 15.66) - open/close 6795 (time 1788589210.81 total 294.08 last 25.25) - open/close 7043 (time 1788589221.31 total 304.58 last 23.61) - open/close 7293 (time 1788589231.47 total 314.75 last 24.60) - open/close 7495 (time 1788589241.74 total 325.01 last 19.68) - open/close 7723 (time 1788589252.55 total 335.82 last 21.10) - open/close 7877 (time 1788589263.17 total 346.45 last 14.49) - open/close 8087 (time 1788589273.46 total 356.73 last 20.41) - open/close 8365 (time 1788589284.11 total 367.38 last 26.10) - open/close 8582 (time 1788589294.12 total 377.39 last 21.69) - open/close 8873 (time 1788589304.82 total 388.10 last 27.18) - open/close 9171 (time 1788589315.20 total 398.48 last 28.71) - open/close 9440 (time 1788589325.23 total 408.51 last 26.82) - open/close 9717 (time 1788589335.25 total 418.52 last 27.66) - open/close 9999 (time 1788589346.18 total 429.45 last 25.80) - open/close 10233 (time 1788589356.93 total 440.20 last 21.77) - open/close 10506 (time 1788589367.09 total 450.36 last 26.85) - open/close 10740 (time 1788589377.85 total 461.12 last 21.76) - open/close 11021 (time 1788589387.87 total 471.14 last 28.05) - open/close 11366 (time 1788589398.58 total 481.85 last 32.21) - open/close 11617 (time 1788589409.23 total 492.50 last 23.57) - open/close 11910 (time 1788589419.24 total 502.51 last 29.27) - open/close 12161 (time 1788589429.76 total 513.03 last 23.86) - open/close 12461 (time 1788589440.32 total 523.59 last 28.41) - open/close 12758 (time 1788589450.33 total 533.60 last 29.67) - open/close 12896 (time 1788589460.99 total 544.26 last 12.95) - open/close 13021 (time 1788589470.99 total 554.27 last 12.49) - open/close 13211 (time 1788589481.72 total 564.99 last 17.71) - open/close 13424 (time 1788589492.03 total 575.30 last 20.65) - open/close 13627 (time 1788589502.04 total 585.31 last 20.29) - open/close 13820 (time 1788589512.84 total 596.12 last 17.86) - open/close 14080 (time 1788589522.85 total 606.12 last 25.98) - open/close 14299 (time 1788589533.04 total 616.31 last 21.50) - open/close 14546 (time 1788589543.85 total 627.12 last 22.84) - open/close 14765 (time 1788589554.67 total 637.94 last 20.25) - open/close 15028 (time 1788589565.25 total 648.52 last 24.87) - open/close 15276 (time 1788589575.54 total 658.81 last 24.09) - open/close 15499 (time 1788589586.23 total 669.50 last 20.86) - open/close 15769 (time 1788589596.41 total 679.68 last 26.54) - open/close 16066 (time 1788589606.98 total 690.25 last 28.08) - open/close 16359 (time 1788589617.72 total 700.99 last 27.30) - open/close 16607 (time 1788589628.40 total 711.68 last 23.20) - open/close 16856 (time 1788589638.42 total 721.69 last 24.86) - open/close 17085 (time 1788589648.42 total 731.70 last 22.89) - open/close 17338 (time 1788589659.33 total 742.60 last 23.20) - open/close 17596 (time 1788589669.33 total 752.60 last 25.80) - open/close 17861 (time 1788589679.33 total 762.60 last 26.50) - open/close 18118 (time 1788589690.35 total 773.62 last 23.32) - open/close 18390 (time 1788589701.06 total 784.33 last 25.40) - open/close 18613 (time 1788589711.16 total 794.43 last 22.08) - open/close 18874 (time 1788589721.17 total 804.44 last 26.08) - open/close 19084 (time 1788589731.18 total 814.45 last 20.96) - open/close 19322 (time 1788589741.75 total 825.02 last 22.53) - open/close 19563 (time 1788589751.77 total 835.04 last 24.05) - open/close 19743 (time 1788589761.78 total 845.05 last 17.99) - open/close 19950 (time 1788589772.23 total 855.50 last 19.80) - open/close 20497 (time 1788589782.23 total 865.50 last 54.69) - open/close 21373 (time 1788589792.24 total 875.51 last 87.56) - open/close 22259 (time 1788589802.24 total 885.51 last 88.54) - open/close 23080 (time 1788589812.25 total 895.52 last 82.07) - open/close 23892 (time 1788589822.25 total 905.53 last 81.15) - open/close 24750 (time 1788589832.26 total 915.53 last 85.76) - open/close 25524 (time 1788589842.26 total 925.53 last 77.37) - open/close 26325 (time 1788589852.28 total 935.55 last 80.00) - open/close 27203 (time 1788589862.29 total 945.56 last 87.70) - open/close 28028 (time 1788589872.29 total 955.56 last 82.50) - open/close 28787 (time 1788589882.29 total 965.56 last 75.89) - open/close 29483 (time 1788589892.29 total 975.57 last 69.57) - open/close 30000 (time 1788589898.84 total 982.11 last 78.97) - open/close 30839 (time 1788589908.85 total 992.12 last 83.85) - open/close 31661 (time 1788589918.86 total 1002.13 last 82.11) - open/close 32489 (time 1788589928.86 total 1012.14 last 82.74) - open/close 33266 (time 1788589938.87 total 1022.14 last 77.63) - open/close 34155 (time 1788589948.88 total 1032.15 last 88.83) - open/close 35041 (time 1788589958.89 total 1042.16 last 88.56) - open/close 35855 (time 1788589968.89 total 1052.16 last 81.39) - open/close 36599 (time 1788589978.89 total 1062.16 last 74.36) - open/close 37453 (time 1788589988.90 total 1072.17 last 85.35) - open/close 38257 (time 1788589998.92 total 1082.19 last 80.26) - open/close 39029 (time 1788590008.92 total 1092.19 last 77.19) - open/close 39758 (time 1788590018.93 total 1102.21 last 72.77) - open/close 40000 (time 1788590022.00 total 1105.27 last 78.91) - open/close 40832 (time 1788590032.01 total 1115.28 last 83.13) - open/close 41584 (time 1788590042.02 total 1125.29 last 75.13) - open/close 42356 (time 1788590052.03 total 1135.31 last 77.09) - open/close 42986 (time 1788590062.04 total 1145.31 last 62.97) - open/close 43685 (time 1788590072.06 total 1155.33 last 69.79) - open/close 44368 (time 1788590082.08 total 1165.35 last 68.16) - open/close 45188 (time 1788590092.08 total 1175.35 last 81.95) - open/close 45940 (time 1788590102.09 total 1185.36 last 75.16) - open/close 46580 (time 1788590112.09 total 1195.36 last 63.98) - open/close 47280 (time 1788590122.10 total 1205.38 last 69.90) - open/close 47937 (time 1788590132.12 total 1215.39 last 65.61) - open/close 48621 (time 1788590142.14 total 1225.41 last 68.27) - open/close 49398 (time 1788590152.14 total 1235.41 last 77.66) total: 50000 open/close in 1242.85 seconds: 40.23 ops/second - unlinked 0 (time 1788590164 ; total 0 ; last 0) - unlinked 10000 (time 1788590298 ; total 134 ; last 134) - unlinked 20000 (time 1788590423 ; total 259 ; last 125) - unlinked 30000 (time 1788590555 ; total 391 ; last 132) - unlinked 40000 (time 1788590688 ; total 524 ; last 133) total: 50000 unlinks in 651 seconds: 76.804916 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: kmemleak disabled modules unloaded. pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2