== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:36:20 (1788842180) start mds service on oleg119-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: kmemleak disabled oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg119-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg119-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg119-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg119-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 150 (time 1788842294.58 total 10.21 last 14.69) - open/close 367 (time 1788842305.47 total 21.10 last 19.93) - open/close 576 (time 1788842316.12 total 31.76 last 19.62) - open/close 711 (time 1788842326.67 total 42.30 last 12.80) - open/close 839 (time 1788842337.44 total 53.07 last 11.89) - open/close 891 (time 1788842348.07 total 63.71 last 4.89) - open/close 1045 (time 1788842359.06 total 74.70 last 14.01) - open/close 1145 (time 1788842369.76 total 85.39 last 9.35) - open/close 1219 (time 1788842379.83 total 95.46 last 7.35) - open/close 1371 (time 1788842390.75 total 106.38 last 13.92) - open/close 1472 (time 1788842401.37 total 117.00 last 9.51) - open/close 1648 (time 1788842412.44 total 128.07 last 15.90) - open/close 1775 (time 1788842422.55 total 138.19 last 12.56) - open/close 1905 (time 1788842433.36 total 148.99 last 12.03) - open/close 2081 (time 1788842444.25 total 159.89 last 16.15) - open/close 2328 (time 1788842454.26 total 169.89 last 24.69) - open/close 2569 (time 1788842465.05 total 180.68 last 22.34) - open/close 2741 (time 1788842475.48 total 191.11 last 16.49) - open/close 2911 (time 1788842485.81 total 201.44 last 16.46) - open/close 3108 (time 1788842496.36 total 212.00 last 18.66) - open/close 3223 (time 1788842507.26 total 222.89 last 10.56) - open/close 3441 (time 1788842518.32 total 233.96 last 19.70) - open/close 3594 (time 1788842528.33 total 243.96 last 15.30) - open/close 3785 (time 1788842538.37 total 254.01 last 19.01) - open/close 3903 (time 1788842548.39 total 264.02 last 11.78) - open/close 4054 (time 1788842558.43 total 274.06 last 15.04) - open/close 4111 (time 1788842569.37 total 285.01 last 5.21) - open/close 4284 (time 1788842579.37 total 295.01 last 17.30) - open/close 4413 (time 1788842591.64 total 307.27 last 10.52) - open/close 4552 (time 1788842601.93 total 317.57 last 13.50) - open/close 4670 (time 1788842612.56 total 328.20 last 11.10) - open/close 4857 (time 1788842622.57 total 338.20 last 18.69) - open/close 5093 (time 1788842633.49 total 349.13 last 21.61) - open/close 5390 (time 1788842643.95 total 359.58 last 28.40) - open/close 5451 (time 1788842654.86 total 370.50 last 5.59) - open/close 5578 (time 1788842664.92 total 380.55 last 12.63) - open/close 5677 (time 1788842675.30 total 390.93 last 9.54) - open/close 5843 (time 1788842685.85 total 401.49 last 15.72) - open/close 5987 (time 1788842696.05 total 411.68 last 14.12) - open/close 6122 (time 1788842706.07 total 421.70 last 13.47) - open/close 6351 (time 1788842716.08 total 431.71 last 22.88) - open/close 6568 (time 1788842726.10 total 441.73 last 21.66) - open/close 6809 (time 1788842736.10 total 451.73 last 24.09) - open/close 7008 (time 1788842746.10 total 461.74 last 19.89) - open/close 7159 (time 1788842756.21 total 471.84 last 14.95) - open/close 7326 (time 1788842766.31 total 481.94 last 16.54) - open/close 7477 (time 1788842776.32 total 491.95 last 15.09) - open/close 7614 (time 1788842786.33 total 501.97 last 13.67) - open/close 7754 (time 1788842796.34 total 511.98 last 13.99) - open/close 7942 (time 1788842806.35 total 521.99 last 18.78) - open/close 8084 (time 1788842817.39 total 533.02 last 12.87) - open/close 8260 (time 1788842827.40 total 543.04 last 17.57) - open/close 8364 (time 1788842838.03 total 553.66 last 9.79) - open/close 8478 (time 1788842848.05 total 563.68 last 11.37) - open/close 8578 (time 1788842858.82 total 574.45 last 9.29) - open/close 8708 (time 1788842868.91 total 584.55 last 12.87) - open/close 8819 (time 1788842878.94 total 594.58 last 11.07) - open/close 9067 (time 1788842888.95 total 604.59 last 24.77) - open/close 9307 (time 1788842899.83 total 615.46 last 22.08) - open/close 9455 (time 1788842910.57 total 626.21 last 13.77) - open/close 9619 (time 1788842920.68 total 636.31 last 16.23) - open/close 9797 (time 1788842931.49 total 647.12 last 16.47) - open/close 9933 (time 1788842941.72 total 657.35 last 13.29) - open/close 10000 (time 1788842945.04 total 660.67 last 20.20) - open/close 10077 (time 1788842955.04 total 670.68 last 7.70) - open/close 10140 (time 1788842965.92 total 681.55 last 5.79) - open/close 10240 (time 1788842977.43 total 693.06 last 8.69) - open/close 10378 (time 1788842987.53 total 703.16 last 13.66) - open/close 10519 (time 1788842997.59 total 713.23 last 14.01) - open/close 10716 (time 1788843007.75 total 723.38 last 19.40) - open/close 10874 (time 1788843023.60 total 739.23 last 9.97) - open/close 11054 (time 1788843033.60 total 749.24 last 17.99) - open/close 11173 (time 1788843043.63 total 759.26 last 11.87) - open/close 11323 (time 1788843053.63 total 769.27 last 15.00) - open/close 11455 (time 1788843063.64 total 779.27 last 13.19) - open/close 11614 (time 1788843074.70 total 790.33 last 14.37) - open/close 11737 (time 1788843086.42 total 802.06 last 10.49) - open/close 11798 (time 1788843097.12 total 812.75 last 5.70) - open/close 11922 (time 1788843107.16 total 822.79 last 12.35) - open/close 12116 (time 1788843118.33 total 833.96 last 17.37) - open/close 12320 (time 1788843128.91 total 844.55 last 19.27) - open/close 12493 (time 1788843139.47 total 855.11 last 16.38) - open/close 12626 (time 1788843149.49 total 865.13 last 13.28) - open/close 12771 (time 1788843160.31 total 875.94 last 13.40) - open/close 12936 (time 1788843171.09 total 886.72 last 15.31) - open/close 13070 (time 1788843181.09 total 896.72 last 13.40) - open/close 13326 (time 1788843191.85 total 907.49 last 23.78) - open/close 13488 (time 1788843201.87 total 917.50 last 16.18) - open/close 13608 (time 1788843211.87 total 927.50 last 12.00) - open/close 13799 (time 1788843222.24 total 937.87 last 18.42) - open/close 13959 (time 1788843233.25 total 948.88 last 14.53) - open/close 14122 (time 1788843243.79 total 959.42 last 15.46) - open/close 14188 (time 1788843254.80 total 970.44 last 5.99) - open/close 14350 (time 1788843264.82 total 980.46 last 16.16) - open/close 14416 (time 1788843274.97 total 990.61 last 6.50) - open/close 14477 (time 1788843285.08 total 1000.71 last 6.04) - open/close 14649 (time 1788843296.08 total 1011.71 last 15.64) - open/close 14860 (time 1788843306.41 total 1022.04 last 20.43) - open/close 14928 (time 1788843317.28 total 1032.91 last 6.25) - open/close 15153 (time 1788843327.92 total 1043.56 last 21.13) - open/close 15312 (time 1788843337.93 total 1053.57 last 15.89) - open/close 15502 (time 1788843348.99 total 1064.62 last 17.19) - open/close 15575 (time 1788843359.71 total 1075.35 last 6.81) - open/close 15645 (time 1788843369.71 total 1085.35 last 7.00) - open/close 15752 (time 1788843379.74 total 1095.37 last 10.67) - open/close 15843 (time 1788843390.44 total 1106.08 last 8.50) - open/close 15936 (time 1788843400.44 total 1116.08 last 9.30) - open/close 16341 (time 1788843410.45 total 1126.08 last 40.47) - open/close 17069 (time 1788843420.47 total 1136.10 last 72.68) - open/close 17785 (time 1788843430.50 total 1146.13 last 71.37) - open/close 18436 (time 1788843440.51 total 1156.14 last 65.04) - open/close 19118 (time 1788843450.51 total 1166.14 last 68.20) - open/close 19838 (time 1788843460.52 total 1176.15 last 71.95) - open/close 20000 (time 1788843462.71 total 1178.34 last 73.84) - open/close 20742 (time 1788843472.72 total 1188.35 last 74.14) - open/close 21493 (time 1788843482.72 total 1198.35 last 75.07) - open/close 22382 (time 1788843492.73 total 1208.36 last 88.83) - open/close 23254 (time 1788843502.73 total 1218.37 last 87.16) - open/close 24028 (time 1788843512.74 total 1228.38 last 77.32) - open/close 24814 (time 1788843522.75 total 1238.38 last 78.55) - open/close 25573 (time 1788843532.76 total 1248.39 last 75.82) - open/close 26348 (time 1788843542.76 total 1258.40 last 77.46) - open/close 27069 (time 1788843552.77 total 1268.40 last 72.08) - open/close 27685 (time 1788843562.77 total 1278.41 last 61.56) - open/close 28369 (time 1788843572.79 total 1288.42 last 68.29) - open/close 28998 (time 1788843582.80 total 1298.43 last 62.87) - open/close 29650 (time 1788843592.80 total 1308.43 last 65.20) - open/close 30000 (time 1788843597.57 total 1313.20 last 73.29) - open/close 30591 (time 1788843607.59 total 1323.23 last 58.96) - open/close 31065 (time 1788843617.60 total 1333.24 last 47.37) - open/close 31670 (time 1788843627.61 total 1343.25 last 60.43) - open/close 32360 (time 1788843637.62 total 1353.26 last 68.94) - open/close 33140 (time 1788843647.62 total 1363.26 last 77.99) - open/close 33943 (time 1788843657.63 total 1373.26 last 80.23) - open/close 34720 (time 1788843667.65 total 1383.28 last 77.58) - open/close 35545 (time 1788843677.65 total 1393.28 last 82.46) - open/close 36257 (time 1788843687.66 total 1403.29 last 71.12) - open/close 36944 (time 1788843697.67 total 1413.31 last 68.62) - open/close 37608 (time 1788843707.67 total 1423.31 last 66.39) - open/close 38335 (time 1788843717.68 total 1433.31 last 72.69) - open/close 39071 (time 1788843727.68 total 1443.32 last 73.56) - open/close 39799 (time 1788843737.69 total 1453.33 last 72.72) - open/close 40000 (time 1788843740.59 total 1456.22 last 69.43) - open/close 40573 (time 1788843750.59 total 1466.22 last 57.30) - open/close 41146 (time 1788843760.62 total 1476.25 last 57.14) - open/close 41839 (time 1788843770.62 total 1486.25 last 69.27) - open/close 42570 (time 1788843780.63 total 1496.26 last 73.04) - open/close 43322 (time 1788843790.63 total 1506.27 last 75.16) - open/close 44099 (time 1788843800.63 total 1516.27 last 77.70) - open/close 44882 (time 1788843810.64 total 1526.27 last 78.27) - open/close 45587 (time 1788843820.66 total 1536.29 last 70.37) - open/close 46324 (time 1788843830.66 total 1546.29 last 73.69) - open/close 47045 (time 1788843840.66 total 1556.30 last 72.08) - open/close 47814 (time 1788843850.67 total 1566.30 last 76.86) - open/close 48663 (time 1788843860.69 total 1576.32 last 84.69) - open/close 49397 (time 1788843870.69 total 1586.33 last 73.40) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1589.73 seconds: 31.22 ops/second - unlinked 0 (time 1788843878 ; total 0 ; last 0) - unlinked 10000 (time 1788844027 ; total 149 ; last 149) - unlinked 20000 (time 1788844163 ; total 285 ; last 136) - unlinked 30000 (time 1788844292 ; total 414 ; last 129) - unlinked 40000 (time 1788844427 ; total 549 ; last 135) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 671 seconds: 73.965721 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: kmemleak disabled modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2