== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:18:35 (1788441515) start mds service on oleg207-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: kmemleak disabled oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg207-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg207-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1788441618.17 total 10.87 last 18.31) - open/close 321 (time 1788441628.30 total 21.00 last 12.05) - open/close 491 (time 1788441639.25 total 31.95 last 15.51) - open/close 705 (time 1788441652.19 total 44.89 last 16.55) - open/close 880 (time 1788441663.16 total 55.86 last 15.94) - open/close 1083 (time 1788441673.55 total 66.25 last 19.55) - open/close 1166 (time 1788441683.89 total 76.59 last 8.02) - open/close 1341 (time 1788441694.39 total 87.09 last 16.67) - open/close 1389 (time 1788441705.02 total 97.72 last 4.52) - open/close 1488 (time 1788441715.77 total 108.47 last 9.21) - open/close 1658 (time 1788441725.78 total 118.48 last 16.98) - open/close 1791 (time 1788441735.87 total 128.57 last 13.18) - open/close 1907 (time 1788441745.89 total 138.59 last 11.58) - open/close 2061 (time 1788441756.53 total 149.23 last 14.47) - open/close 2276 (time 1788441766.65 total 159.35 last 21.25) - open/close 2463 (time 1788441777.28 total 169.98 last 17.59) - open/close 2615 (time 1788441787.29 total 179.99 last 15.19) - open/close 2692 (time 1788441797.64 total 190.34 last 7.44) - open/close 2879 (time 1788441808.32 total 201.02 last 17.52) - open/close 2956 (time 1788441820.01 total 212.71 last 6.58) - open/close 3134 (time 1788441830.02 total 222.72 last 17.79) - open/close 3334 (time 1788441840.13 total 232.83 last 19.80) - open/close 3516 (time 1788441851.65 total 244.35 last 15.79) - open/close 3685 (time 1788441862.15 total 254.85 last 16.10) - open/close 3832 (time 1788441872.20 total 264.90 last 14.63) - open/close 4024 (time 1788441882.78 total 275.48 last 18.15) - open/close 4170 (time 1788441893.69 total 286.39 last 13.37) - open/close 4259 (time 1788441904.50 total 297.20 last 8.23) - open/close 4475 (time 1788441914.52 total 307.22 last 21.56) - open/close 4580 (time 1788441924.53 total 317.23 last 10.49) - open/close 4744 (time 1788441935.17 total 327.87 last 15.41) - open/close 4863 (time 1788441945.18 total 337.88 last 11.89) - open/close 5077 (time 1788441957.32 total 350.02 last 17.63) - open/close 5256 (time 1788441967.53 total 360.23 last 17.53) - open/close 5371 (time 1788441978.23 total 370.93 last 10.75) - open/close 5521 (time 1788441988.75 total 381.45 last 14.26) - open/close 5631 (time 1788441998.97 total 391.67 last 10.76) - open/close 5890 (time 1788442008.97 total 401.67 last 25.89) - open/close 6015 (time 1788442018.98 total 411.68 last 12.49) - open/close 6153 (time 1788442029.95 total 422.65 last 12.58) - open/close 6315 (time 1788442040.73 total 433.43 last 15.03) - open/close 6565 (time 1788442050.74 total 443.44 last 24.98) - open/close 6674 (time 1788442061.62 total 454.32 last 10.02) - open/close 6881 (time 1788442071.62 total 464.32 last 20.69) - open/close 6997 (time 1788442081.68 total 474.38 last 11.53) - open/close 7068 (time 1788442091.71 total 484.41 last 7.08) - open/close 7172 (time 1788442102.34 total 495.04 last 9.78) - open/close 7405 (time 1788442112.77 total 505.47 last 22.33) - open/close 7627 (time 1788442123.08 total 515.78 last 21.54) - open/close 7887 (time 1788442134.03 total 526.73 last 23.74) - open/close 8086 (time 1788442144.64 total 537.34 last 18.75) - open/close 8160 (time 1788442154.88 total 547.58 last 7.23) - open/close 8318 (time 1788442165.45 total 558.15 last 14.95) - open/close 8495 (time 1788442176.20 total 568.90 last 16.46) - open/close 8719 (time 1788442186.20 total 578.90 last 22.39) - open/close 8919 (time 1788442196.54 total 589.24 last 19.35) - open/close 8985 (time 1788442206.74 total 599.44 last 6.47) - open/close 9202 (time 1788442217.74 total 610.44 last 19.74) - open/close 9270 (time 1788442227.75 total 620.45 last 6.79) - open/close 9397 (time 1788442238.04 total 630.74 last 12.34) - open/close 9515 (time 1788442248.04 total 640.74 last 11.79) - open/close 9681 (time 1788442258.05 total 650.75 last 16.59) - open/close 9919 (time 1788442268.05 total 660.75 last 23.79) - open/close 10000 (time 1788442271.65 total 664.35 last 22.49) - open/close 10205 (time 1788442281.97 total 674.67 last 19.86) - open/close 10378 (time 1788442292.67 total 685.37 last 16.18) - open/close 10523 (time 1788442302.67 total 695.37 last 14.50) - open/close 10622 (time 1788442312.75 total 705.45 last 9.82) - open/close 10787 (time 1788442322.76 total 715.46 last 16.50) - open/close 10953 (time 1788442334.28 total 726.98 last 14.41) - open/close 11217 (time 1788442344.28 total 736.98 last 26.39) - open/close 11365 (time 1788442354.83 total 747.53 last 14.03) - open/close 11477 (time 1788442365.65 total 758.35 last 10.35) - open/close 11659 (time 1788442375.66 total 768.36 last 18.17) - open/close 11777 (time 1788442385.67 total 778.37 last 11.79) - open/close 11932 (time 1788442396.61 total 789.31 last 14.18) - open/close 12167 (time 1788442407.06 total 799.76 last 22.49) - open/close 12280 (time 1788442417.10 total 809.80 last 11.26) - open/close 12379 (time 1788442427.42 total 820.12 last 9.59) - open/close 12527 (time 1788442437.43 total 830.13 last 14.78) - open/close 12652 (time 1788442450.18 total 842.88 last 9.81) - open/close 12767 (time 1788442460.18 total 852.88 last 11.50) - open/close 12960 (time 1788442470.19 total 862.89 last 19.28) - open/close 13167 (time 1788442480.39 total 873.09 last 20.30) - open/close 13331 (time 1788442491.16 total 883.86 last 15.22) - open/close 13572 (time 1788442501.56 total 894.26 last 23.18) - open/close 13740 (time 1788442512.27 total 904.97 last 15.69) - open/close 13934 (time 1788442522.62 total 915.32 last 18.73) - open/close 14176 (time 1788442532.64 total 925.34 last 24.16) - open/close 14423 (time 1788442542.64 total 935.34 last 24.70) - open/close 14621 (time 1788442553.54 total 946.24 last 18.17) - open/close 14879 (time 1788442563.57 total 956.27 last 25.73) - open/close 15145 (time 1788442573.82 total 966.52 last 25.96) - open/close 15385 (time 1788442583.82 total 976.52 last 23.99) - open/close 15666 (time 1788442593.82 total 986.52 last 28.09) - open/close 15852 (time 1788442604.98 total 997.68 last 16.67) - open/close 16117 (time 1788442614.99 total 1007.69 last 26.47) - open/close 16328 (time 1788442625.41 total 1018.11 last 20.25) - open/close 16627 (time 1788442636.37 total 1029.07 last 27.28) - open/close 16841 (time 1788442648.15 total 1040.85 last 18.17) - open/close 17074 (time 1788442658.15 total 1050.85 last 23.30) - open/close 17693 (time 1788442668.15 total 1060.85 last 61.87) - open/close 18356 (time 1788442678.16 total 1070.86 last 66.27) - open/close 18926 (time 1788442688.18 total 1080.88 last 56.87) - open/close 19618 (time 1788442698.21 total 1090.91 last 69.04) - open/close 20000 (time 1788442704.71 total 1097.41 last 58.76) - open/close 20598 (time 1788442714.71 total 1107.41 last 59.80) - open/close 21445 (time 1788442724.72 total 1117.42 last 84.62) - open/close 22071 (time 1788442734.73 total 1127.43 last 62.54) - open/close 22772 (time 1788442744.73 total 1137.43 last 70.08) - open/close 23561 (time 1788442754.73 total 1147.43 last 78.89) - open/close 24246 (time 1788442764.74 total 1157.44 last 68.45) - open/close 24933 (time 1788442774.74 total 1167.44 last 68.70) - open/close 25578 (time 1788442784.74 total 1177.44 last 64.50) - open/close 26188 (time 1788442794.75 total 1187.45 last 60.94) - open/close 26795 (time 1788442804.77 total 1197.47 last 60.57) - open/close 27430 (time 1788442814.79 total 1207.49 last 63.40) - open/close 28214 (time 1788442824.80 total 1217.49 last 78.33) - open/close 28836 (time 1788442834.80 total 1227.50 last 62.18) - open/close 29601 (time 1788442844.80 total 1237.50 last 76.49) - open/close 30000 (time 1788442850.34 total 1243.04 last 72.06) - open/close 30751 (time 1788442860.36 total 1253.06 last 74.96) - open/close 31476 (time 1788442870.38 total 1263.08 last 72.33) - open/close 32056 (time 1788442880.39 total 1273.09 last 57.94) - open/close 32813 (time 1788442890.39 total 1283.09 last 75.66) - open/close 33498 (time 1788442900.40 total 1293.10 last 68.48) - open/close 34163 (time 1788442910.40 total 1303.10 last 66.46) - open/close 34786 (time 1788442920.41 total 1313.11 last 62.24) - open/close 35375 (time 1788442930.45 total 1323.15 last 58.69) - open/close 35985 (time 1788442940.45 total 1333.15 last 60.97) - open/close 36665 (time 1788442950.46 total 1343.16 last 67.94) - open/close 37384 (time 1788442960.47 total 1353.17 last 71.83) - open/close 38135 (time 1788442970.48 total 1363.18 last 75.04) - open/close 38781 (time 1788442980.49 total 1373.19 last 64.54) - open/close 39474 (time 1788442990.49 total 1383.19 last 69.25) - open/close 40000 (time 1788442997.94 total 1390.64 last 70.65) - open/close 40594 (time 1788443007.94 total 1400.64 last 59.38) - open/close 41206 (time 1788443017.95 total 1410.65 last 61.18) - open/close 41696 (time 1788443027.95 total 1420.65 last 48.96) - open/close 42263 (time 1788443037.97 total 1430.67 last 56.58) - open/close 42977 (time 1788443047.98 total 1440.68 last 71.34) - open/close 43645 (time 1788443057.99 total 1450.69 last 66.73) - open/close 44214 (time 1788443068.00 total 1460.70 last 56.88) - open/close 44803 (time 1788443078.02 total 1470.72 last 58.74) - open/close 45360 (time 1788443088.04 total 1480.74 last 55.62) - open/close 45964 (time 1788443098.06 total 1490.76 last 60.29) - open/close 46554 (time 1788443108.06 total 1500.76 last 58.95) - open/close 47198 (time 1788443118.07 total 1510.77 last 64.37) - open/close 47853 (time 1788443128.09 total 1520.79 last 65.36) - open/close 48435 (time 1788443138.09 total 1530.79 last 58.19) - open/close 49039 (time 1788443148.11 total 1540.81 last 60.32) - open/close 49616 (time 1788443158.11 total 1550.81 last 57.70) total: 50000 open/close in 1556.97 seconds: 32.11 ops/second - unlinked 0 (time 1788443168 ; total 0 ; last 0) - unlinked 10000 (time 1788443336 ; total 168 ; last 168) - unlinked 20000 (time 1788443505 ; total 337 ; last 169) - unlinked 30000 (time 1788443664 ; total 496 ; last 159) - unlinked 40000 (time 1788443830 ; total 662 ; last 166) total: 50000 unlinks in 832 seconds: 60.096153 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: kmemleak disabled modules unloaded. pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2