== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:44:46 (1788752686) start mds service on oleg445-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: kmemleak disabled oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg445-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg445-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1788752796.59 total 10.94 last 11.97) - open/close 321 (time 1788752806.60 total 20.95 last 18.98) - open/close 476 (time 1788752817.26 total 31.60 last 14.55) - open/close 670 (time 1788752828.45 total 42.80 last 17.33) - open/close 831 (time 1788752839.02 total 53.37 last 15.23) - open/close 968 (time 1788752849.03 total 63.38 last 13.68) - open/close 1179 (time 1788752859.70 total 74.05 last 19.77) - open/close 1462 (time 1788752870.63 total 84.98 last 25.90) - open/close 1622 (time 1788752880.64 total 94.99 last 15.99) - open/close 1858 (time 1788752891.05 total 105.40 last 22.66) - open/close 2027 (time 1788752901.06 total 115.41 last 16.88) - open/close 2110 (time 1788752911.08 total 125.42 last 8.29) - open/close 2258 (time 1788752921.44 total 135.79 last 14.27) - open/close 2456 (time 1788752931.83 total 146.18 last 19.07) - open/close 2651 (time 1788752944.93 total 159.28 last 14.88) - open/close 2818 (time 1788752955.50 total 169.85 last 15.80) - open/close 3032 (time 1788752965.51 total 179.86 last 21.38) - open/close 3162 (time 1788752975.72 total 190.07 last 12.74) - open/close 3328 (time 1788752986.28 total 200.63 last 15.71) - open/close 3476 (time 1788752996.77 total 211.12 last 14.11) - open/close 3652 (time 1788753007.55 total 221.90 last 16.33) - open/close 3843 (time 1788753017.64 total 231.99 last 18.93) - open/close 4038 (time 1788753027.71 total 242.06 last 19.38) - open/close 4141 (time 1788753038.88 total 253.23 last 9.21) - open/close 4340 (time 1788753048.89 total 263.24 last 19.90) - open/close 4506 (time 1788753059.55 total 273.90 last 15.56) - open/close 4768 (time 1788753069.57 total 283.92 last 26.15) - open/close 4947 (time 1788753080.18 total 294.53 last 16.88) - open/close 5226 (time 1788753090.19 total 304.54 last 27.86) - open/close 5489 (time 1788753101.23 total 315.58 last 23.83) - open/close 5738 (time 1788753111.41 total 325.76 last 24.46) - open/close 6006 (time 1788753121.42 total 335.77 last 26.78) - open/close 6216 (time 1788753131.42 total 345.77 last 20.99) - open/close 6445 (time 1788753142.06 total 356.41 last 21.53) - open/close 6738 (time 1788753152.99 total 367.34 last 26.80) - open/close 7005 (time 1788753163.88 total 378.23 last 24.52) - open/close 7298 (time 1788753176.42 total 390.77 last 23.36) - open/close 7488 (time 1788753186.42 total 400.77 last 19.00) - open/close 7595 (time 1788753197.26 total 411.61 last 9.87) - open/close 7744 (time 1788753207.27 total 421.62 last 14.89) - open/close 7833 (time 1788753217.58 total 431.93 last 8.63) - open/close 8025 (time 1788753227.95 total 442.30 last 18.50) - open/close 8245 (time 1788753238.90 total 453.24 last 20.11) - open/close 8434 (time 1788753248.91 total 463.26 last 18.88) - open/close 8686 (time 1788753258.91 total 473.26 last 25.19) - open/close 8884 (time 1788753269.04 total 483.39 last 19.54) - open/close 9134 (time 1788753279.86 total 494.21 last 23.11) - open/close 9399 (time 1788753289.88 total 504.23 last 26.46) - open/close 9639 (time 1788753300.48 total 514.83 last 22.64) - open/close 9832 (time 1788753310.49 total 524.84 last 19.27) - open/close 10000 (time 1788753316.53 total 530.88 last 27.85) - open/close 10174 (time 1788753326.69 total 541.04 last 17.11) - open/close 10374 (time 1788753336.81 total 551.16 last 19.78) - open/close 10562 (time 1788753346.81 total 561.16 last 18.80) - open/close 10773 (time 1788753356.81 total 571.16 last 21.09) - open/close 10983 (time 1788753367.09 total 581.44 last 20.44) - open/close 11145 (time 1788753377.52 total 591.86 last 15.53) - open/close 11270 (time 1788753387.83 total 602.18 last 12.11) - open/close 11333 (time 1788753398.52 total 612.87 last 5.90) - open/close 11457 (time 1788753410.35 total 624.69 last 10.49) - open/close 11659 (time 1788753420.60 total 634.95 last 19.70) - open/close 11734 (time 1788753431.53 total 645.88 last 6.86) - open/close 11938 (time 1788753441.54 total 655.89 last 20.38) - open/close 12149 (time 1788753451.81 total 666.16 last 20.54) - open/close 12331 (time 1788753461.81 total 676.16 last 18.20) - open/close 12573 (time 1788753472.49 total 686.84 last 22.65) - open/close 12743 (time 1788753483.30 total 697.65 last 15.73) - open/close 12939 (time 1788753493.76 total 708.11 last 18.75) - open/close 13127 (time 1788753503.86 total 718.21 last 18.61) - open/close 13329 (time 1788753514.85 total 729.20 last 18.38) - open/close 13449 (time 1788753524.85 total 739.20 last 11.99) - open/close 13623 (time 1788753534.86 total 749.21 last 17.38) - open/close 13770 (time 1788753545.27 total 759.62 last 14.13) - open/close 14044 (time 1788753555.97 total 770.32 last 25.61) - open/close 14247 (time 1788753566.16 total 780.51 last 19.92) - open/close 14449 (time 1788753576.38 total 790.73 last 19.77) - open/close 14512 (time 1788753587.11 total 801.46 last 5.87) - open/close 14638 (time 1788753597.12 total 811.46 last 12.60) - open/close 14824 (time 1788753607.12 total 821.47 last 18.58) - open/close 14892 (time 1788753617.91 total 832.26 last 6.30) - open/close 15032 (time 1788753628.02 total 842.37 last 13.84) - open/close 15216 (time 1788753638.12 total 852.47 last 18.22) - open/close 15381 (time 1788753648.13 total 862.48 last 16.48) - open/close 15499 (time 1788753658.15 total 872.50 last 11.78) - open/close 15753 (time 1788753669.10 total 883.45 last 23.21) - open/close 15902 (time 1788753679.10 total 893.45 last 14.90) - open/close 16003 (time 1788753690.50 total 904.85 last 8.86) - open/close 16103 (time 1788753703.42 total 917.77 last 7.74) - open/close 16231 (time 1788753713.44 total 927.79 last 12.78) - open/close 16417 (time 1788753723.82 total 938.17 last 17.92) - open/close 16712 (time 1788753736.96 total 951.31 last 22.45) - open/close 16954 (time 1788753747.75 total 962.10 last 22.42) - open/close 17172 (time 1788753758.39 total 972.74 last 20.50) - open/close 17270 (time 1788753770.24 total 984.59 last 8.27) - open/close 17403 (time 1788753780.29 total 994.64 last 13.23) - open/close 17520 (time 1788753790.31 total 1004.66 last 11.68) - open/close 17711 (time 1788753801.19 total 1015.54 last 17.55) - open/close 17779 (time 1788753812.20 total 1026.55 last 6.18) - open/close 17840 (time 1788753823.28 total 1037.63 last 5.50) - open/close 17952 (time 1788753833.30 total 1047.64 last 11.19) - open/close 18592 (time 1788753843.30 total 1057.65 last 63.98) - open/close 19279 (time 1788753853.30 total 1067.65 last 68.66) - open/close 19923 (time 1788753863.30 total 1077.65 last 64.39) - open/close 20581 (time 1788753873.33 total 1087.68 last 65.66) - open/close 21208 (time 1788753883.34 total 1097.69 last 62.62) - open/close 21855 (time 1788753893.35 total 1107.70 last 64.62) - open/close 22535 (time 1788753903.35 total 1117.70 last 68.00) - open/close 23306 (time 1788753913.36 total 1127.70 last 77.07) - open/close 23936 (time 1788753923.36 total 1137.71 last 62.97) - open/close 24512 (time 1788753933.37 total 1147.72 last 57.55) - open/close 25092 (time 1788753943.38 total 1157.73 last 57.93) - open/close 25817 (time 1788753953.39 total 1167.74 last 72.45) - open/close 26435 (time 1788753963.39 total 1177.74 last 61.77) - open/close 27100 (time 1788753973.39 total 1187.74 last 66.49) - open/close 27750 (time 1788753983.40 total 1197.75 last 64.95) - open/close 28388 (time 1788753993.41 total 1207.76 last 63.73) - open/close 29049 (time 1788754003.45 total 1217.79 last 65.88) - open/close 29647 (time 1788754013.46 total 1227.81 last 59.70) - open/close 30000 (time 1788754018.79 total 1233.14 last 66.21) - open/close 30666 (time 1788754028.80 total 1243.15 last 66.54) - open/close 31346 (time 1788754038.82 total 1253.17 last 67.91) - open/close 31903 (time 1788754048.82 total 1263.17 last 55.68) - open/close 32582 (time 1788754058.83 total 1273.18 last 67.83) - open/close 33222 (time 1788754068.83 total 1283.18 last 63.99) - open/close 33845 (time 1788754078.83 total 1293.18 last 62.29) - open/close 34563 (time 1788754088.83 total 1303.18 last 71.79) - open/close 35230 (time 1788754098.84 total 1313.19 last 66.62) - open/close 35812 (time 1788754108.85 total 1323.20 last 58.19) - open/close 36553 (time 1788754118.87 total 1333.22 last 73.93) - open/close 37173 (time 1788754128.88 total 1343.23 last 61.95) - open/close 37771 (time 1788754138.88 total 1353.23 last 59.78) - open/close 38487 (time 1788754148.88 total 1363.23 last 71.58) - open/close 39167 (time 1788754158.90 total 1373.25 last 67.90) - open/close 39871 (time 1788754168.90 total 1383.25 last 70.38) - open/close 40638 (time 1788754178.91 total 1393.26 last 76.64) - open/close 41310 (time 1788754188.92 total 1403.27 last 67.14) - open/close 41983 (time 1788754198.93 total 1413.28 last 67.24) - open/close 42692 (time 1788754208.94 total 1423.29 last 70.82) - open/close 43432 (time 1788754218.94 total 1433.29 last 73.96) - open/close 44116 (time 1788754228.95 total 1443.30 last 68.34) - open/close 44781 (time 1788754238.97 total 1453.32 last 66.39) - open/close 45527 (time 1788754248.98 total 1463.33 last 74.55) - open/close 46208 (time 1788754258.99 total 1473.34 last 68.03) - open/close 46899 (time 1788754268.99 total 1483.34 last 69.08) - open/close 47587 (time 1788754278.99 total 1493.34 last 68.77) - open/close 48258 (time 1788754289.01 total 1503.36 last 67.01) - open/close 48996 (time 1788754299.01 total 1513.36 last 73.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1522.98 seconds: 32.59 ops/second - unlinked 0 (time 1788754314 ; total 0 ; last 0) - unlinked 10000 (time 1788754462 ; total 148 ; last 148) - unlinked 20000 (time 1788754602 ; total 288 ; last 140) - unlinked 30000 (time 1788754744 ; total 430 ; last 142) - unlinked 40000 (time 1788754897 ; total 583 ; last 153) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 726 seconds: 68.362259 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: kmemleak disabled modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2