== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:38:50 (1788845930) start mds service on oleg311-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: kmemleak disabled oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg311-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg311-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg311-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1788846033.02 total 11.00 last 18.64) - open/close 390 (time 1788846045.36 total 23.34 last 14.99) - open/close 606 (time 1788846056.09 total 34.07 last 20.12) - open/close 850 (time 1788846066.72 total 44.70 last 22.96) - open/close 964 (time 1788846076.72 total 54.70 last 11.40) - open/close 1189 (time 1788846086.73 total 64.70 last 22.50) - open/close 1346 (time 1788846097.18 total 75.16 last 15.02) - open/close 1502 (time 1788846107.56 total 85.54 last 15.03) - open/close 1703 (time 1788846117.66 total 95.64 last 19.89) - open/close 1957 (time 1788846128.44 total 106.42 last 23.58) - open/close 2218 (time 1788846138.45 total 116.42 last 26.08) - open/close 2362 (time 1788846148.67 total 126.64 last 14.09) - open/close 2654 (time 1788846159.11 total 137.09 last 27.96) - open/close 2908 (time 1788846170.16 total 148.14 last 22.98) - open/close 3126 (time 1788846180.17 total 158.15 last 21.79) - open/close 3358 (time 1788846190.33 total 168.31 last 22.83) - open/close 3629 (time 1788846200.87 total 178.85 last 25.71) - open/close 3859 (time 1788846211.83 total 189.81 last 20.98) - open/close 3970 (time 1788846224.43 total 202.41 last 8.81) - open/close 4210 (time 1788846235.05 total 213.02 last 22.60) - open/close 4354 (time 1788846245.12 total 223.10 last 14.30) - open/close 4532 (time 1788846255.61 total 233.59 last 16.96) - open/close 4713 (time 1788846265.69 total 243.67 last 17.96) - open/close 4800 (time 1788846276.06 total 254.04 last 8.39) - open/close 4893 (time 1788846286.51 total 264.49 last 8.90) - open/close 5065 (time 1788846296.53 total 274.51 last 17.16) - open/close 5134 (time 1788846307.30 total 285.27 last 6.41) - open/close 5308 (time 1788846317.32 total 295.29 last 17.36) - open/close 5551 (time 1788846327.32 total 305.30 last 24.29) - open/close 5725 (time 1788846338.03 total 316.01 last 16.25) - open/close 5860 (time 1788846348.19 total 326.17 last 13.28) - open/close 6093 (time 1788846359.77 total 337.75 last 20.12) - open/close 6191 (time 1788846370.29 total 348.26 last 9.32) - open/close 6335 (time 1788846380.40 total 358.38 last 14.24) - open/close 6425 (time 1788846391.21 total 369.18 last 8.33) - open/close 6581 (time 1788846401.21 total 379.19 last 15.59) - open/close 6715 (time 1788846411.95 total 389.92 last 12.48) - open/close 6932 (time 1788846422.90 total 400.88 last 19.81) - open/close 7179 (time 1788846432.93 total 410.91 last 24.62) - open/close 7386 (time 1788846442.98 total 420.96 last 20.60) - open/close 7651 (time 1788846452.99 total 430.97 last 26.48) - open/close 7870 (time 1788846463.00 total 440.98 last 21.88) - open/close 8132 (time 1788846473.91 total 451.89 last 24.02) - open/close 8326 (time 1788846484.65 total 462.63 last 18.06) - open/close 8572 (time 1788846495.36 total 473.34 last 22.97) - open/close 8686 (time 1788846505.97 total 483.95 last 10.74) - open/close 8893 (time 1788846515.99 total 493.97 last 20.67) - open/close 9150 (time 1788846526.02 total 504.00 last 25.63) - open/close 9307 (time 1788846536.81 total 514.79 last 14.54) - open/close 9430 (time 1788846547.83 total 525.81 last 11.16) - open/close 9680 (time 1788846558.51 total 536.49 last 23.40) - open/close 9826 (time 1788846568.56 total 546.54 last 14.53) - open/close 10000 (time 1788846573.87 total 551.84 last 32.80) - open/close 10138 (time 1788846584.88 total 562.86 last 12.53) - open/close 10186 (time 1788846595.27 total 573.24 last 4.62) - open/close 10489 (time 1788846605.94 total 583.92 last 28.39) - open/close 10678 (time 1788846615.98 total 593.96 last 18.83) - open/close 10849 (time 1788846626.72 total 604.70 last 15.91) - open/close 11028 (time 1788846637.54 total 615.52 last 16.54) - open/close 11199 (time 1788846648.43 total 626.41 last 15.70) - open/close 11482 (time 1788846661.11 total 639.09 last 22.32) - open/close 11627 (time 1788846671.43 total 649.41 last 14.05) - open/close 11841 (time 1788846682.23 total 660.21 last 19.82) - open/close 12103 (time 1788846692.23 total 670.21 last 26.19) - open/close 12348 (time 1788846702.26 total 680.24 last 24.44) - open/close 12432 (time 1788846713.26 total 691.24 last 7.64) - open/close 12660 (time 1788846723.27 total 701.25 last 22.77) - open/close 12865 (time 1788846733.28 total 711.26 last 20.49) - open/close 13067 (time 1788846743.28 total 721.26 last 20.20) - open/close 13330 (time 1788846754.03 total 732.01 last 24.47) - open/close 13488 (time 1788846765.42 total 743.39 last 13.88) - open/close 13673 (time 1788846775.43 total 753.41 last 18.47) - open/close 13926 (time 1788846785.43 total 763.41 last 25.29) - open/close 14084 (time 1788846797.99 total 775.97 last 12.58) - open/close 14240 (time 1788846808.16 total 786.14 last 15.34) - open/close 14447 (time 1788846823.52 total 801.50 last 13.48) - open/close 14603 (time 1788846833.78 total 811.76 last 15.21) - open/close 14702 (time 1788846844.41 total 822.38 last 9.32) - open/close 14817 (time 1788846854.41 total 832.39 last 11.49) - open/close 14989 (time 1788846865.20 total 843.17 last 15.95) - open/close 15243 (time 1788846875.22 total 853.19 last 25.35) - open/close 15455 (time 1788846885.22 total 863.20 last 21.19) - open/close 15692 (time 1788846895.52 total 873.50 last 23.00) - open/close 15944 (time 1788846905.54 total 883.51 last 25.17) - open/close 16095 (time 1788846915.55 total 893.53 last 15.08) - open/close 16282 (time 1788846925.56 total 903.54 last 18.68) - open/close 16366 (time 1788846935.61 total 913.59 last 8.36) - open/close 16495 (time 1788846945.61 total 923.59 last 12.90) - open/close 16651 (time 1788846958.00 total 935.98 last 12.59) - open/close 16787 (time 1788846968.01 total 945.99 last 13.58) - open/close 16982 (time 1788846978.03 total 956.01 last 19.47) - open/close 17260 (time 1788846988.03 total 966.01 last 27.80) - open/close 17490 (time 1788846998.05 total 976.03 last 22.95) - open/close 17725 (time 1788847008.06 total 986.04 last 23.47) - open/close 17973 (time 1788847018.09 total 996.07 last 24.73) - open/close 18177 (time 1788847029.81 total 1007.79 last 17.41) - open/close 18419 (time 1788847039.97 total 1017.95 last 23.81) - open/close 18560 (time 1788847049.98 total 1027.96 last 14.09) - open/close 19263 (time 1788847059.98 total 1037.96 last 70.30) - open/close 19936 (time 1788847069.98 total 1047.96 last 67.27) - open/close 20662 (time 1788847079.99 total 1057.97 last 72.56) - open/close 21410 (time 1788847090.01 total 1067.99 last 74.66) - open/close 22004 (time 1788847100.01 total 1077.99 last 59.40) - open/close 22630 (time 1788847110.02 total 1088.00 last 62.53) - open/close 23424 (time 1788847120.02 total 1098.00 last 79.39) - open/close 24137 (time 1788847130.03 total 1108.00 last 71.26) - open/close 24810 (time 1788847140.03 total 1118.01 last 67.28) - open/close 25548 (time 1788847150.03 total 1128.01 last 73.76) - open/close 26206 (time 1788847160.05 total 1138.03 last 65.70) - open/close 26853 (time 1788847170.06 total 1148.04 last 64.64) - open/close 27486 (time 1788847180.07 total 1158.05 last 63.24) - open/close 28191 (time 1788847190.07 total 1168.05 last 70.46) - open/close 28939 (time 1788847200.08 total 1178.06 last 74.76) - open/close 29696 (time 1788847210.08 total 1188.06 last 75.66) - open/close 30000 (time 1788847213.58 total 1191.56 last 87.00) - open/close 30712 (time 1788847223.58 total 1201.56 last 71.17) - open/close 31492 (time 1788847233.59 total 1211.57 last 77.91) - open/close 32148 (time 1788847243.60 total 1221.58 last 65.56) - open/close 32832 (time 1788847253.61 total 1231.58 last 68.36) - open/close 33530 (time 1788847263.61 total 1241.59 last 69.77) - open/close 34231 (time 1788847273.63 total 1251.61 last 69.98) - open/close 35013 (time 1788847283.63 total 1261.61 last 78.20) - open/close 35697 (time 1788847293.64 total 1271.62 last 68.29) - open/close 36376 (time 1788847303.66 total 1281.64 last 67.76) - open/close 37044 (time 1788847313.67 total 1291.64 last 66.79) - open/close 37740 (time 1788847323.68 total 1301.65 last 69.53) - open/close 38486 (time 1788847333.69 total 1311.67 last 74.49) - open/close 39172 (time 1788847343.70 total 1321.68 last 68.54) - open/close 39924 (time 1788847353.72 total 1331.70 last 75.07) - open/close 40590 (time 1788847363.72 total 1341.70 last 66.58) - open/close 41299 (time 1788847373.73 total 1351.71 last 70.81) - open/close 41977 (time 1788847383.74 total 1361.71 last 67.78) - open/close 42737 (time 1788847393.74 total 1371.71 last 76.00) - open/close 43429 (time 1788847403.74 total 1381.72 last 69.16) - open/close 44189 (time 1788847413.76 total 1391.74 last 75.86) - open/close 44867 (time 1788847423.76 total 1401.74 last 67.78) - open/close 45571 (time 1788847433.77 total 1411.74 last 70.37) - open/close 46309 (time 1788847443.78 total 1421.76 last 73.66) - open/close 47077 (time 1788847453.79 total 1431.77 last 76.77) - open/close 47804 (time 1788847463.80 total 1441.77 last 72.64) - open/close 48525 (time 1788847473.80 total 1451.77 last 72.10) - open/close 49248 (time 1788847483.81 total 1461.79 last 72.21) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1467.05 seconds: 33.83 ops/second - unlinked 0 (time 1788847493 ; total 0 ; last 0) - unlinked 10000 (time 1788847635 ; total 142 ; last 142) - unlinked 20000 (time 1788847772 ; total 279 ; last 137) - unlinked 30000 (time 1788847929 ; total 436 ; last 157) - unlinked 40000 (time 1788848094 ; total 601 ; last 165) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 740 seconds: 67.068916 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local oleg311-server: kmemleak disabled modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2