== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:22:09 (1789780929) start mds service on oleg211-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 18 15:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: kmemleak disabled oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg211-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg211-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg211-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 95 (time 1789781040.47 total 10.07 last 9.43) - open/close 310 (time 1789781050.50 total 20.10 last 21.43) - open/close 483 (time 1789781061.39 total 31.00 last 15.88) - open/close 666 (time 1789781071.40 total 41.00 last 18.29) - open/close 832 (time 1789781082.07 total 51.67 last 15.55) - open/close 1068 (time 1789781092.55 total 62.15 last 22.53) - open/close 1323 (time 1789781103.21 total 72.82 last 23.91) - open/close 1462 (time 1789781113.22 total 82.83 last 13.88) - open/close 1621 (time 1789781124.17 total 93.77 last 14.53) - open/close 1858 (time 1789781136.78 total 106.39 last 18.79) - open/close 2069 (time 1789781147.04 total 116.64 last 20.58) - open/close 2272 (time 1789781159.17 total 128.77 last 16.74) - open/close 2503 (time 1789781169.35 total 138.95 last 22.69) - open/close 2687 (time 1789781179.88 total 149.48 last 17.48) - open/close 2931 (time 1789781190.56 total 160.16 last 22.85) - open/close 3080 (time 1789781200.91 total 170.52 last 14.39) - open/close 3205 (time 1789781210.93 total 180.54 last 12.48) - open/close 3286 (time 1789781221.18 total 190.78 last 7.91) - open/close 3516 (time 1789781231.32 total 200.92 last 22.68) - open/close 3637 (time 1789781242.12 total 211.72 last 11.21) - open/close 3813 (time 1789781253.05 total 222.66 last 16.09) - open/close 3953 (time 1789781263.07 total 232.67 last 13.98) - open/close 4214 (time 1789781273.19 total 242.79 last 25.79) - open/close 4409 (time 1789781284.03 total 253.64 last 17.98) - open/close 4553 (time 1789781294.04 total 263.64 last 14.40) - open/close 4780 (time 1789781304.19 total 273.80 last 22.35) - open/close 4887 (time 1789781314.96 total 284.56 last 9.94) - open/close 5065 (time 1789781326.53 total 296.13 last 15.38) - open/close 5163 (time 1789781336.54 total 306.14 last 9.79) - open/close 5252 (time 1789781346.56 total 316.17 last 8.88) - open/close 5397 (time 1789781357.40 total 327.01 last 13.37) - open/close 5568 (time 1789781367.41 total 337.02 last 17.08) - open/close 5776 (time 1789781377.41 total 347.02 last 20.80) - open/close 5982 (time 1789781388.24 total 357.85 last 19.03) - open/close 6229 (time 1789781398.25 total 367.86 last 24.67) - open/close 6379 (time 1789781408.38 total 377.98 last 14.81) - open/close 6542 (time 1789781421.30 total 390.90 last 12.62) - open/close 6755 (time 1789781431.30 total 400.90 last 21.30) - open/close 6952 (time 1789781441.30 total 410.91 last 19.69) - open/close 7143 (time 1789781451.32 total 420.92 last 19.07) - open/close 7292 (time 1789781464.40 total 434.00 last 11.39) - open/close 7468 (time 1789781474.41 total 444.02 last 17.58) - open/close 7603 (time 1789781484.54 total 454.15 last 13.32) - open/close 7698 (time 1789781495.23 total 464.84 last 8.89) - open/close 7753 (time 1789781505.26 total 474.86 last 5.49) - open/close 7955 (time 1789781516.30 total 485.90 last 18.30) - open/close 8137 (time 1789781526.67 total 496.27 last 17.54) - open/close 8276 (time 1789781537.41 total 507.01 last 12.95) - open/close 8448 (time 1789781550.28 total 519.88 last 13.36) - open/close 8612 (time 1789781560.68 total 530.28 last 15.77) - open/close 8677 (time 1789781570.70 total 540.30 last 6.48) - open/close 8801 (time 1789781581.64 total 551.24 last 11.34) - open/close 8889 (time 1789781592.33 total 561.93 last 8.23) - open/close 8941 (time 1789781603.40 total 573.01 last 4.69) - open/close 9108 (time 1789781614.09 total 583.70 last 15.62) - open/close 9299 (time 1789781624.75 total 594.35 last 17.93) - open/close 9497 (time 1789781635.65 total 605.25 last 18.16) - open/close 9749 (time 1789781648.09 total 617.69 last 20.26) - open/close 9929 (time 1789781658.59 total 628.19 last 17.14) - open/close 10000 (time 1789781662.17 total 631.77 last 19.84) - open/close 10017 (time 1789781673.87 total 643.48 last 1.45) - open/close 10185 (time 1789781684.28 total 653.88 last 16.15) - open/close 10355 (time 1789781694.30 total 663.90 last 16.97) - open/close 10485 (time 1789781704.54 total 674.15 last 12.69) - open/close 10592 (time 1789781715.58 total 685.18 last 9.70) - open/close 10808 (time 1789781725.59 total 695.19 last 21.58) - open/close 10975 (time 1789781735.61 total 705.21 last 16.67) - open/close 11056 (time 1789781745.77 total 715.37 last 7.97) - open/close 11190 (time 1789781758.41 total 728.01 last 10.60) - open/close 11393 (time 1789781768.77 total 738.38 last 19.59) - open/close 11483 (time 1789781778.85 total 748.45 last 8.93) - open/close 11591 (time 1789781789.14 total 758.74 last 10.50) - open/close 11731 (time 1789781800.17 total 769.77 last 12.69) - open/close 11930 (time 1789781810.99 total 780.59 last 18.39) - open/close 12110 (time 1789781822.05 total 791.66 last 16.27) - open/close 12135 (time 1789781832.93 total 802.53 last 2.30) - open/close 12322 (time 1789781842.96 total 812.56 last 18.64) - open/close 12485 (time 1789781853.98 total 823.59 last 14.79) - open/close 12642 (time 1789781863.99 total 833.59 last 15.69) - open/close 12824 (time 1789781876.33 total 845.93 last 14.75) - open/close 12949 (time 1789781887.17 total 856.78 last 11.52) - open/close 13094 (time 1789781900.04 total 869.64 last 11.27) - open/close 13207 (time 1789781910.06 total 879.66 last 11.28) - open/close 13394 (time 1789781920.41 total 890.01 last 18.07) - open/close 13512 (time 1789781930.45 total 900.05 last 11.76) - open/close 13640 (time 1789781941.50 total 911.11 last 11.58) - open/close 13815 (time 1789781951.52 total 921.12 last 17.48) - open/close 13996 (time 1789781961.53 total 931.13 last 18.08) - open/close 14106 (time 1789781971.55 total 941.15 last 10.98) - open/close 14267 (time 1789781981.97 total 951.57 last 15.46) - open/close 14419 (time 1789781991.98 total 961.58 last 15.18) - open/close 14572 (time 1789782002.90 total 972.50 last 14.01) - open/close 14661 (time 1789782012.91 total 982.51 last 8.89) - open/close 14734 (time 1789782023.10 total 992.71 last 7.16) - open/close 14988 (time 1789782033.74 total 1003.35 last 23.88) - open/close 15111 (time 1789782043.90 total 1013.51 last 12.10) - open/close 15278 (time 1789782053.92 total 1023.52 last 16.68) - open/close 15490 (time 1789782063.94 total 1033.54 last 21.15) - open/close 15657 (time 1789782076.92 total 1046.53 last 12.86) - open/close 15782 (time 1789782086.96 total 1056.56 last 12.45) - open/close 15901 (time 1789782096.98 total 1066.59 last 11.87) - open/close 16016 (time 1789782107.56 total 1077.16 last 10.87) - open/close 16239 (time 1789782117.58 total 1087.18 last 22.26) - open/close 16763 (time 1789782127.58 total 1097.19 last 52.37) - open/close 17744 (time 1789782137.58 total 1107.19 last 98.07) - open/close 18336 (time 1789782147.60 total 1117.21 last 59.10) - open/close 19044 (time 1789782157.61 total 1127.21 last 70.74) - open/close 19697 (time 1789782167.64 total 1137.24 last 65.12) - open/close 20000 (time 1789782171.87 total 1141.47 last 71.59) - open/close 20744 (time 1789782181.88 total 1151.49 last 74.32) - open/close 21429 (time 1789782191.89 total 1161.49 last 68.46) - open/close 22256 (time 1789782201.90 total 1171.50 last 82.62) - open/close 22811 (time 1789782211.90 total 1181.51 last 55.46) - open/close 23466 (time 1789782221.91 total 1191.51 last 65.49) - open/close 24246 (time 1789782231.91 total 1201.52 last 77.94) - open/close 24894 (time 1789782241.93 total 1211.53 last 64.70) - open/close 25437 (time 1789782251.94 total 1221.54 last 54.25) - open/close 26073 (time 1789782261.94 total 1231.55 last 63.57) - open/close 26770 (time 1789782271.95 total 1241.55 last 69.67) - open/close 27519 (time 1789782281.95 total 1251.56 last 74.86) - open/close 28355 (time 1789782291.96 total 1261.57 last 83.53) - open/close 29040 (time 1789782301.97 total 1271.58 last 68.42) - open/close 29756 (time 1789782311.97 total 1281.58 last 71.60) - open/close 30000 (time 1789782316.53 total 1286.14 last 53.53) - open/close 30737 (time 1789782326.54 total 1296.14 last 73.67) - open/close 31495 (time 1789782336.55 total 1306.15 last 75.71) - open/close 32319 (time 1789782346.55 total 1316.16 last 82.35) - open/close 33052 (time 1789782356.56 total 1326.16 last 73.25) - open/close 33808 (time 1789782366.57 total 1336.17 last 75.56) - open/close 34694 (time 1789782376.57 total 1346.18 last 88.53) - open/close 35433 (time 1789782386.58 total 1356.19 last 73.84) - open/close 36121 (time 1789782396.59 total 1366.19 last 68.77) - open/close 36903 (time 1789782406.60 total 1376.20 last 78.11) - open/close 37778 (time 1789782416.60 total 1386.20 last 87.49) - open/close 38688 (time 1789782426.61 total 1396.21 last 90.93) - open/close 39461 (time 1789782436.61 total 1406.21 last 77.27) - open/close 40000 (time 1789782444.49 total 1414.10 last 68.35) - open/close 40556 (time 1789782454.50 total 1424.10 last 55.56) - open/close 41104 (time 1789782464.53 total 1434.13 last 54.65) - open/close 41696 (time 1789782474.53 total 1444.14 last 59.18) - open/close 42298 (time 1789782484.54 total 1454.14 last 60.16) - open/close 42896 (time 1789782494.56 total 1464.16 last 59.71) - open/close 43492 (time 1789782504.56 total 1474.16 last 59.58) - open/close 44110 (time 1789782514.57 total 1484.17 last 61.72) - open/close 44737 (time 1789782524.58 total 1494.18 last 62.67) - open/close 45398 (time 1789782534.58 total 1504.18 last 66.08) - open/close 46109 (time 1789782544.59 total 1514.20 last 71.00) - open/close 46747 (time 1789782554.60 total 1524.21 last 63.73) - open/close 47334 (time 1789782564.61 total 1534.21 last 58.67) - open/close 47930 (time 1789782574.61 total 1544.21 last 59.59) - open/close 48424 (time 1789782584.61 total 1554.22 last 49.38) - open/close 49088 (time 1789782594.63 total 1564.23 last 66.32) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1573.59 seconds: 31.54 ops/second - unlinked 0 (time 1789782611 ; total 0 ; last 0) - unlinked 10000 (time 1789782793 ; total 182 ; last 182) - unlinked 20000 (time 1789782931 ; total 320 ; last 138) - unlinked 30000 (time 1789783076 ; total 465 ; last 145) - unlinked 40000 (time 1789783226 ; total 615 ; last 150) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 749 seconds: 66.263016 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local oleg211-server: kmemleak disabled modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2