== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:47:08 (1788443228) start mds service on oleg156-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: kmemleak disabled oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions oleg156-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg156-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg156-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg156-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg156-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 90 (time 1788443344.00 total 10.43 last 8.63) - open/close 189 (time 1788443354.62 total 21.06 last 9.32) - open/close 364 (time 1788443365.23 total 31.66 last 16.50) - open/close 645 (time 1788443375.24 total 41.67 last 28.08) - open/close 839 (time 1788443385.64 total 52.07 last 18.65) - open/close 980 (time 1788443398.51 total 64.94 last 10.96) - open/close 1096 (time 1788443408.51 total 74.94 last 11.60) - open/close 1239 (time 1788443419.11 total 85.54 last 13.49) - open/close 1364 (time 1788443429.17 total 95.61 last 12.42) - open/close 1531 (time 1788443439.96 total 106.40 last 15.48) - open/close 1766 (time 1788443452.78 total 119.21 last 18.34) - open/close 1922 (time 1788443463.02 total 129.46 last 15.22) - open/close 2145 (time 1788443473.03 total 139.47 last 22.29) - open/close 2346 (time 1788443483.05 total 149.49 last 20.06) - open/close 2426 (time 1788443493.08 total 159.52 last 7.97) - open/close 2696 (time 1788443505.65 total 172.09 last 21.49) - open/close 2872 (time 1788443516.13 total 182.56 last 16.80) - open/close 3042 (time 1788443526.63 total 193.07 last 16.19) - open/close 3287 (time 1788443537.39 total 203.82 last 22.78) - open/close 3448 (time 1788443547.89 total 214.33 last 15.33) - open/close 3675 (time 1788443558.71 total 225.14 last 20.99) - open/close 3915 (time 1788443569.26 total 235.70 last 22.73) - open/close 4157 (time 1788443579.27 total 245.71 last 24.18) - open/close 4382 (time 1788443589.96 total 256.39 last 21.05) - open/close 4510 (time 1788443599.98 total 266.42 last 12.77) - open/close 4744 (time 1788443612.65 total 279.09 last 18.47) - open/close 4847 (time 1788443623.59 total 290.03 last 9.41) - open/close 5001 (time 1788443634.59 total 301.03 last 14.01) - open/close 5127 (time 1788443646.19 total 312.62 last 10.86) - open/close 5379 (time 1788443656.93 total 323.37 last 23.46) - open/close 5507 (time 1788443668.66 total 335.10 last 10.91) - open/close 5769 (time 1788443678.67 total 345.10 last 26.18) - open/close 5888 (time 1788443689.25 total 355.69 last 11.24) - open/close 6112 (time 1788443700.06 total 366.50 last 20.73) - open/close 6212 (time 1788443711.01 total 377.45 last 9.13) - open/close 6395 (time 1788443721.55 total 387.98 last 17.37) - open/close 6625 (time 1788443731.59 total 398.03 last 22.89) - open/close 6832 (time 1788443742.63 total 409.07 last 18.76) - open/close 6984 (time 1788443752.93 total 419.37 last 14.75) - open/close 7150 (time 1788443763.88 total 430.31 last 15.17) - open/close 7263 (time 1788443773.88 total 440.32 last 11.29) - open/close 7519 (time 1788443783.89 total 450.33 last 25.57) - open/close 7787 (time 1788443793.91 total 460.34 last 26.77) - open/close 7998 (time 1788443804.89 total 471.33 last 19.21) - open/close 8246 (time 1788443815.45 total 481.89 last 23.49) - open/close 8492 (time 1788443826.14 total 492.57 last 23.02) - open/close 8766 (time 1788443837.13 total 503.56 last 24.93) - open/close 9045 (time 1788443847.19 total 513.63 last 27.72) - open/close 9285 (time 1788443858.39 total 524.83 last 21.43) - open/close 9547 (time 1788443869.05 total 535.49 last 24.57) - open/close 9825 (time 1788443879.32 total 545.76 last 27.07) - open/close 10000 (time 1788443888.62 total 555.06 last 18.82) - open/close 10237 (time 1788443898.63 total 565.07 last 23.67) - open/close 10434 (time 1788443908.64 total 575.08 last 19.69) - open/close 10627 (time 1788443918.66 total 585.09 last 19.26) - open/close 10874 (time 1788443929.38 total 595.81 last 23.04) - open/close 11112 (time 1788443940.25 total 606.69 last 21.89) - open/close 11332 (time 1788443951.04 total 617.48 last 20.38) - open/close 11407 (time 1788443961.63 total 628.07 last 7.08) - open/close 11597 (time 1788443971.64 total 638.08 last 18.98) - open/close 11810 (time 1788443981.64 total 648.08 last 21.30) - open/close 11959 (time 1788443991.66 total 658.09 last 14.88) - open/close 12062 (time 1788444001.79 total 668.23 last 10.16) - open/close 12208 (time 1788444012.80 total 679.24 last 13.26) - open/close 12340 (time 1788444023.19 total 689.63 last 12.71) - open/close 12469 (time 1788444035.11 total 701.54 last 10.83) - open/close 12646 (time 1788444045.87 total 712.30 last 16.45) - open/close 12756 (time 1788444056.29 total 722.73 last 10.55) - open/close 12936 (time 1788444066.31 total 732.75 last 17.96) - open/close 13092 (time 1788444077.03 total 743.47 last 14.56) - open/close 13246 (time 1788444087.05 total 753.48 last 15.37) - open/close 13410 (time 1788444097.06 total 763.50 last 16.37) - open/close 13483 (time 1788444107.11 total 773.55 last 7.27) - open/close 13626 (time 1788444117.63 total 784.07 last 13.59) - open/close 13795 (time 1788444128.09 total 794.53 last 16.16) - open/close 13963 (time 1788444139.12 total 805.56 last 15.23) - open/close 14084 (time 1788444150.10 total 816.54 last 11.02) - open/close 14373 (time 1788444160.11 total 826.55 last 28.87) - open/close 14676 (time 1788444170.11 total 836.55 last 30.29) - open/close 14797 (time 1788444180.94 total 847.38 last 11.18) - open/close 14953 (time 1788444191.78 total 858.22 last 14.39) - open/close 15085 (time 1788444201.94 total 868.38 last 12.99) - open/close 15244 (time 1788444212.99 total 879.42 last 14.40) - open/close 15423 (time 1788444223.00 total 889.44 last 17.87) - open/close 15555 (time 1788444233.38 total 899.82 last 12.72) - open/close 15737 (time 1788444244.32 total 910.76 last 16.63) - open/close 15940 (time 1788444254.33 total 920.76 last 20.29) - open/close 15974 (time 1788444264.33 total 930.77 last 3.40) - open/close 16178 (time 1788444274.35 total 940.78 last 20.37) - open/close 16368 (time 1788444284.41 total 950.85 last 18.88) - open/close 16598 (time 1788444294.96 total 961.40 last 21.80) - open/close 16752 (time 1788444305.46 total 971.90 last 14.67) - open/close 16825 (time 1788444315.96 total 982.40 last 6.95) - open/close 16979 (time 1788444326.02 total 992.45 last 15.31) - open/close 17187 (time 1788444336.16 total 1002.59 last 20.52) - open/close 17335 (time 1788444346.17 total 1012.61 last 14.78) - open/close 17589 (time 1788444357.18 total 1023.62 last 23.07) - open/close 17804 (time 1788444367.21 total 1033.64 last 21.44) - open/close 17973 (time 1788444377.84 total 1044.27 last 15.90) - open/close 18259 (time 1788444387.85 total 1054.29 last 28.55) - open/close 18545 (time 1788444397.85 total 1064.29 last 28.60) - open/close 19361 (time 1788444407.86 total 1074.29 last 81.59) - open/close 20000 (time 1788444415.76 total 1082.20 last 80.82) - open/close 20764 (time 1788444425.77 total 1092.20 last 76.36) - open/close 21353 (time 1788444435.77 total 1102.21 last 58.86) - open/close 22113 (time 1788444445.77 total 1112.21 last 75.99) - open/close 23010 (time 1788444455.78 total 1122.22 last 89.66) - open/close 23765 (time 1788444465.78 total 1132.22 last 75.46) - open/close 24620 (time 1788444475.79 total 1142.23 last 85.45) - open/close 25460 (time 1788444485.80 total 1152.23 last 83.94) - open/close 26243 (time 1788444495.81 total 1162.25 last 78.20) - open/close 27017 (time 1788444505.81 total 1172.25 last 77.39) - open/close 27842 (time 1788444515.82 total 1182.25 last 82.47) - open/close 28619 (time 1788444525.82 total 1192.25 last 77.69) - open/close 29381 (time 1788444535.82 total 1202.26 last 76.18) - open/close 30000 (time 1788444545.01 total 1211.45 last 67.31) - open/close 30731 (time 1788444555.02 total 1221.45 last 73.07) - open/close 31529 (time 1788444565.02 total 1231.46 last 79.77) - open/close 32282 (time 1788444575.03 total 1241.47 last 75.22) - open/close 33060 (time 1788444585.04 total 1251.47 last 77.77) - open/close 33830 (time 1788444595.04 total 1261.47 last 77.00) - open/close 34521 (time 1788444605.05 total 1271.49 last 69.00) - open/close 35209 (time 1788444615.06 total 1281.50 last 68.75) - open/close 35982 (time 1788444625.07 total 1291.50 last 77.25) - open/close 36820 (time 1788444635.07 total 1301.51 last 83.72) - open/close 37561 (time 1788444645.08 total 1311.52 last 74.04) - open/close 38292 (time 1788444655.09 total 1321.53 last 73.05) - open/close 39169 (time 1788444665.09 total 1331.53 last 87.68) - open/close 39871 (time 1788444675.09 total 1341.53 last 70.18) - open/close 40500 (time 1788444685.10 total 1351.54 last 62.85) - open/close 41096 (time 1788444695.11 total 1361.54 last 59.57) - open/close 41785 (time 1788444705.12 total 1371.56 last 68.82) - open/close 42504 (time 1788444715.12 total 1381.56 last 71.86) - open/close 43141 (time 1788444725.13 total 1391.56 last 63.69) - open/close 43790 (time 1788444735.14 total 1401.58 last 64.82) - open/close 44493 (time 1788444745.14 total 1411.58 last 70.28) - open/close 45228 (time 1788444755.15 total 1421.58 last 73.46) - open/close 45995 (time 1788444765.17 total 1431.60 last 76.54) - open/close 46716 (time 1788444775.18 total 1441.62 last 72.01) - open/close 47406 (time 1788444785.20 total 1451.64 last 68.88) - open/close 48163 (time 1788444795.20 total 1461.64 last 75.69) - open/close 49081 (time 1788444805.21 total 1471.65 last 91.70) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1478.09 seconds: 33.58 ops/second - unlinked 0 (time 1788444818 ; total 0 ; last 0) - unlinked 10000 (time 1788444959 ; total 141 ; last 141) - unlinked 20000 (time 1788445096 ; total 278 ; last 137) - unlinked 30000 (time 1788445232 ; total 414 ; last 136) - unlinked 40000 (time 1788445371 ; total 553 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 679 seconds: 73.094254 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg156-server unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: kmemleak disabled modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2