== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:28:37 (1788643717) start mds service on oleg350-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:19 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: kmemleak disabled oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg350-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 218 (time 1788643828.10 total 10.97 last 19.88) - open/close 407 (time 1788643838.60 total 21.47 last 17.99) - open/close 617 (time 1788643852.74 total 35.61 last 14.85) - open/close 788 (time 1788643863.24 total 46.10 last 16.29) - open/close 1019 (time 1788643874.31 total 57.17 last 20.87) - open/close 1207 (time 1788643884.37 total 67.23 last 18.69) - open/close 1359 (time 1788643894.74 total 77.60 last 14.66) - open/close 1550 (time 1788643905.57 total 88.43 last 17.64) - open/close 1767 (time 1788643916.96 total 99.83 last 19.05) - open/close 1961 (time 1788643926.97 total 109.84 last 19.38) - open/close 2236 (time 1788643938.00 total 120.87 last 24.93) - open/close 2449 (time 1788643948.38 total 131.25 last 20.51) - open/close 2606 (time 1788643958.62 total 141.49 last 15.33) - open/close 2833 (time 1788643969.11 total 151.98 last 21.64) - open/close 3029 (time 1788643980.04 total 162.91 last 17.94) - open/close 3269 (time 1788643990.06 total 172.92 last 23.96) - open/close 3427 (time 1788644000.40 total 183.27 last 15.27) - open/close 3601 (time 1788644010.83 total 193.70 last 16.68) - open/close 3798 (time 1788644021.67 total 204.53 last 18.18) - open/close 4077 (time 1788644031.74 total 214.61 last 27.69) - open/close 4239 (time 1788644042.53 total 225.40 last 15.01) - open/close 4331 (time 1788644052.53 total 235.40 last 9.20) - open/close 4571 (time 1788644065.55 total 248.42 last 18.44) - open/close 4756 (time 1788644075.56 total 258.42 last 18.49) - open/close 4932 (time 1788644086.04 total 268.90 last 16.79) - open/close 5148 (time 1788644096.45 total 279.32 last 20.73) - open/close 5405 (time 1788644106.50 total 289.36 last 25.59) - open/close 5668 (time 1788644117.19 total 300.06 last 24.59) - open/close 5855 (time 1788644127.37 total 310.23 last 18.38) - open/close 6093 (time 1788644137.56 total 320.42 last 23.36) - open/close 6325 (time 1788644148.49 total 331.36 last 21.21) - open/close 6585 (time 1788644158.50 total 341.36 last 25.99) - open/close 6805 (time 1788644169.03 total 351.90 last 20.88) - open/close 6969 (time 1788644179.49 total 362.35 last 15.69) - open/close 7038 (time 1788644189.50 total 372.37 last 6.89) - open/close 7176 (time 1788644199.69 total 382.56 last 13.55) - open/close 7325 (time 1788644210.12 total 392.99 last 14.28) - open/close 7568 (time 1788644222.80 total 405.67 last 19.16) - open/close 7715 (time 1788644235.15 total 418.02 last 11.90) - open/close 7813 (time 1788644245.52 total 428.38 last 9.46) - open/close 8001 (time 1788644255.53 total 438.39 last 18.78) - open/close 8138 (time 1788644266.45 total 449.32 last 12.54) - open/close 8240 (time 1788644277.53 total 460.39 last 9.21) - open/close 8392 (time 1788644287.54 total 470.40 last 15.18) - open/close 8628 (time 1788644297.54 total 480.41 last 23.60) - open/close 8869 (time 1788644307.54 total 490.41 last 24.09) - open/close 9044 (time 1788644317.57 total 500.43 last 17.46) - open/close 9179 (time 1788644327.72 total 510.58 last 13.30) - open/close 9327 (time 1788644337.72 total 520.59 last 14.79) - open/close 9489 (time 1788644348.44 total 531.30 last 15.12) - open/close 9625 (time 1788644358.99 total 541.85 last 12.89) - open/close 9707 (time 1788644369.81 total 552.68 last 7.57) - open/close 9826 (time 1788644379.86 total 562.73 last 11.84) - open/close 9969 (time 1788644390.14 total 573.01 last 13.91) - open/close 10220 (time 1788644400.48 total 583.34 last 24.29) - open/close 10466 (time 1788644410.49 total 593.35 last 24.57) - open/close 10716 (time 1788644420.70 total 603.56 last 24.49) - open/close 10879 (time 1788644430.86 total 613.73 last 16.04) - open/close 11089 (time 1788644441.94 total 624.81 last 18.96) - open/close 11347 (time 1788644451.95 total 634.81 last 25.79) - open/close 11490 (time 1788644461.96 total 644.82 last 14.28) - open/close 11694 (time 1788644471.98 total 654.85 last 20.36) - open/close 11927 (time 1788644482.39 total 665.26 last 22.38) - open/close 12204 (time 1788644492.75 total 675.62 last 26.73) - open/close 12428 (time 1788644502.81 total 685.68 last 22.27) - open/close 12694 (time 1788644512.82 total 695.69 last 26.58) - open/close 12801 (time 1788644523.41 total 706.28 last 10.10) - open/close 12941 (time 1788644533.42 total 716.29 last 13.98) - open/close 13128 (time 1788644544.15 total 727.02 last 17.44) - open/close 13287 (time 1788644554.54 total 737.41 last 15.30) - open/close 13436 (time 1788644565.76 total 748.63 last 13.27) - open/close 13671 (time 1788644576.74 total 759.60 last 21.42) - open/close 13875 (time 1788644587.74 total 770.60 last 18.55) - open/close 14061 (time 1788644597.84 total 780.71 last 18.40) - open/close 14165 (time 1788644607.95 total 790.81 last 10.29) - open/close 14351 (time 1788644618.78 total 801.65 last 17.17) - open/close 14417 (time 1788644628.82 total 811.68 last 6.57) - open/close 14606 (time 1788644638.82 total 821.69 last 18.90) - open/close 14838 (time 1788644648.87 total 831.74 last 23.08) - open/close 15087 (time 1788644658.87 total 841.74 last 24.89) - open/close 15360 (time 1788644668.88 total 851.74 last 27.29) - open/close 15549 (time 1788644679.53 total 862.40 last 17.74) - open/close 15814 (time 1788644690.13 total 873.00 last 25.01) - open/close 16065 (time 1788644700.68 total 883.54 last 23.80) - open/close 16314 (time 1788644710.68 total 893.54 last 24.90) - open/close 16542 (time 1788644721.62 total 904.48 last 20.84) - open/close 16780 (time 1788644731.62 total 914.49 last 23.78) - open/close 17007 (time 1788644741.86 total 924.72 last 22.19) - open/close 17203 (time 1788644751.88 total 934.75 last 19.55) - open/close 17370 (time 1788644761.89 total 944.76 last 16.68) - open/close 17573 (time 1788644772.83 total 955.70 last 18.56) - open/close 17699 (time 1788644782.85 total 965.72 last 12.57) - open/close 17828 (time 1788644792.87 total 975.73 last 12.88) - open/close 18049 (time 1788644802.87 total 985.74 last 22.09) - open/close 18298 (time 1788644812.88 total 995.75 last 24.88) - open/close 18852 (time 1788644822.88 total 1005.75 last 55.38) - open/close 19670 (time 1788644832.90 total 1015.76 last 81.68) - open/close 20000 (time 1788644837.26 total 1020.13 last 75.56) - open/close 20783 (time 1788644847.27 total 1030.14 last 78.25) - open/close 21548 (time 1788644857.27 total 1040.14 last 76.48) - open/close 22224 (time 1788644867.28 total 1050.14 last 67.57) - open/close 22888 (time 1788644877.28 total 1060.15 last 66.37) - open/close 23608 (time 1788644887.29 total 1070.16 last 71.94) - open/close 24364 (time 1788644897.29 total 1080.16 last 75.59) - open/close 25139 (time 1788644907.30 total 1090.17 last 77.44) - open/close 25900 (time 1788644917.31 total 1100.17 last 76.04) - open/close 26630 (time 1788644927.32 total 1110.18 last 72.92) - open/close 27397 (time 1788644937.32 total 1120.18 last 76.70) - open/close 28122 (time 1788644947.33 total 1130.20 last 72.41) - open/close 28882 (time 1788644957.34 total 1140.20 last 75.96) - open/close 29658 (time 1788644967.36 total 1150.22 last 77.44) - open/close 30000 (time 1788644971.94 total 1154.80 last 74.65) - open/close 30782 (time 1788644981.95 total 1164.82 last 78.07) - open/close 31558 (time 1788644991.97 total 1174.83 last 77.52) - open/close 32212 (time 1788645001.97 total 1184.84 last 65.36) - open/close 32901 (time 1788645011.98 total 1194.85 last 68.83) - open/close 33653 (time 1788645021.99 total 1204.86 last 75.14) - open/close 34368 (time 1788645032.00 total 1214.87 last 71.42) - open/close 34916 (time 1788645042.02 total 1224.88 last 54.70) - open/close 35536 (time 1788645052.03 total 1234.89 last 61.95) - open/close 36259 (time 1788645062.03 total 1244.90 last 72.25) - open/close 37045 (time 1788645072.04 total 1254.91 last 78.55) - open/close 37814 (time 1788645082.04 total 1264.91 last 76.88) - open/close 38572 (time 1788645092.04 total 1274.91 last 75.78) - open/close 39410 (time 1788645102.06 total 1284.92 last 83.71) - open/close 40000 (time 1788645110.74 total 1293.61 last 67.95) - open/close 40717 (time 1788645120.76 total 1303.62 last 71.58) - open/close 41391 (time 1788645130.77 total 1313.63 last 67.33) - open/close 42081 (time 1788645140.78 total 1323.64 last 68.93) - open/close 42799 (time 1788645150.78 total 1333.65 last 71.75) - open/close 43530 (time 1788645160.79 total 1343.65 last 73.06) - open/close 44298 (time 1788645170.79 total 1353.66 last 76.79) - open/close 45033 (time 1788645180.81 total 1363.68 last 73.34) - open/close 45807 (time 1788645190.82 total 1373.68 last 77.36) - open/close 46547 (time 1788645200.83 total 1383.69 last 73.92) - open/close 47258 (time 1788645210.83 total 1393.69 last 71.10) - open/close 47920 (time 1788645220.84 total 1403.70 last 66.13) - open/close 48488 (time 1788645230.84 total 1413.71 last 56.78) - open/close 49208 (time 1788645240.85 total 1423.72 last 71.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1429.12 seconds: 34.73 ops/second - unlinked 0 (time 1788645250 ; total 0 ; last 0) - unlinked 10000 (time 1788645401 ; total 151 ; last 151) - unlinked 20000 (time 1788645560 ; total 310 ; last 159) - unlinked 30000 (time 1788645707 ; total 457 ; last 147) - unlinked 40000 (time 1788645853 ; total 603 ; last 146) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 743 seconds: 66.798119 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: kmemleak disabled modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2