== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:06:18 (1788923178) start mds service on oleg158-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 8 20:20 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: kmemleak disabled oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg158-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg158-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1788923300.92 total 10.02 last 18.16) - open/close 264 (time 1788923313.01 total 22.11 last 6.78) - open/close 427 (time 1788923323.56 total 32.66 last 15.46) - open/close 524 (time 1788923334.58 total 43.68 last 8.80) - open/close 752 (time 1788923344.58 total 53.68 last 22.79) - open/close 916 (time 1788923354.89 total 63.99 last 15.92) - open/close 1138 (time 1788923365.75 total 74.85 last 20.44) - open/close 1363 (time 1788923376.32 total 85.42 last 21.29) - open/close 1471 (time 1788923386.35 total 95.45 last 10.76) - open/close 1674 (time 1788923396.35 total 105.45 last 20.29) - open/close 1822 (time 1788923407.22 total 116.32 last 13.61) - open/close 2065 (time 1788923417.22 total 126.32 last 24.30) - open/close 2187 (time 1788923430.25 total 139.35 last 9.37) - open/close 2294 (time 1788923442.04 total 151.14 last 9.08) - open/close 2434 (time 1788923452.07 total 161.17 last 13.96) - open/close 2551 (time 1788923462.18 total 171.28 last 11.57) - open/close 2639 (time 1788923472.19 total 181.29 last 8.79) - open/close 2772 (time 1788923482.21 total 191.31 last 13.28) - open/close 2988 (time 1788923492.37 total 201.47 last 21.25) - open/close 3074 (time 1788923502.51 total 211.61 last 8.49) - open/close 3226 (time 1788923512.67 total 221.76 last 14.96) - open/close 3486 (time 1788923522.67 total 231.77 last 25.99) - open/close 3559 (time 1788923532.71 total 241.81 last 7.27) - open/close 3799 (time 1788923542.72 total 251.82 last 23.99) - open/close 4017 (time 1788923553.51 total 262.60 last 20.21) - open/close 4117 (time 1788923563.93 total 273.03 last 9.59) - open/close 4319 (time 1788923574.09 total 283.18 last 19.90) - open/close 4483 (time 1788923584.09 total 293.18 last 16.40) - open/close 4655 (time 1788923595.03 total 304.13 last 15.71) - open/close 4750 (time 1788923605.03 total 314.13 last 9.50) - open/close 4855 (time 1788923615.86 total 324.96 last 9.69) - open/close 4954 (time 1788923626.76 total 335.85 last 9.09) - open/close 5062 (time 1788923637.35 total 346.45 last 10.20) - open/close 5246 (time 1788923647.36 total 356.46 last 18.38) - open/close 5372 (time 1788923658.27 total 367.37 last 11.55) - open/close 5500 (time 1788923668.85 total 377.95 last 12.10) - open/close 5640 (time 1788923678.87 total 387.97 last 13.97) - open/close 5811 (time 1788923688.89 total 397.99 last 17.07) - open/close 6034 (time 1788923701.61 total 410.71 last 17.53) - open/close 6233 (time 1788923711.65 total 420.75 last 19.83) - open/close 6302 (time 1788923721.66 total 430.75 last 6.89) - open/close 6497 (time 1788923731.86 total 440.96 last 19.11) - open/close 6684 (time 1788923742.14 total 451.24 last 18.18) - open/close 6800 (time 1788923752.16 total 461.26 last 11.57) - open/close 6988 (time 1788923762.58 total 471.68 last 18.04) - open/close 7059 (time 1788923773.60 total 482.70 last 6.44) - open/close 7207 (time 1788923783.61 total 492.71 last 14.79) - open/close 7471 (time 1788923793.62 total 502.72 last 26.38) - open/close 7585 (time 1788923804.66 total 513.76 last 10.32) - open/close 7750 (time 1788923814.76 total 523.86 last 16.33) - open/close 7957 (time 1788923825.24 total 534.34 last 19.74) - open/close 8216 (time 1788923835.25 total 544.35 last 25.88) - open/close 8390 (time 1788923845.92 total 555.02 last 16.31) - open/close 8554 (time 1788923857.20 total 566.30 last 14.54) - open/close 8751 (time 1788923867.20 total 576.30 last 19.70) - open/close 8868 (time 1788923877.48 total 586.58 last 11.39) - open/close 9108 (time 1788923888.30 total 597.40 last 22.17) - open/close 9187 (time 1788923898.56 total 607.66 last 7.70) - open/close 9286 (time 1788923908.96 total 618.06 last 9.52) - open/close 9495 (time 1788923919.00 total 628.10 last 20.81) - open/close 9681 (time 1788923929.02 total 638.12 last 18.58) - open/close 9824 (time 1788923939.04 total 648.14 last 14.27) - open/close 9995 (time 1788923949.68 total 658.78 last 16.08) - open/close 10199 (time 1788923959.73 total 668.82 last 20.30) - open/close 10334 (time 1788923970.45 total 679.55 last 12.58) - open/close 10435 (time 1788923981.43 total 690.53 last 9.20) - open/close 10617 (time 1788923991.85 total 700.95 last 17.46) - open/close 10781 (time 1788924002.54 total 711.64 last 15.35) - open/close 10920 (time 1788924012.54 total 721.64 last 13.90) - open/close 11108 (time 1788924023.41 total 732.51 last 17.29) - open/close 11248 (time 1788924034.34 total 743.44 last 12.81) - open/close 11400 (time 1788924044.91 total 754.01 last 14.39) - open/close 11611 (time 1788924055.46 total 764.56 last 19.99) - open/close 11767 (time 1788924065.78 total 774.88 last 15.11) - open/close 11984 (time 1788924076.16 total 785.26 last 20.91) - open/close 12153 (time 1788924086.56 total 795.66 last 16.25) - open/close 12281 (time 1788924096.96 total 806.06 last 12.31) - open/close 12425 (time 1788924107.27 total 816.37 last 13.96) - open/close 12718 (time 1788924119.98 total 829.08 last 23.05) - open/close 12859 (time 1788924130.27 total 839.37 last 13.70) - open/close 13027 (time 1788924141.02 total 850.12 last 15.63) - open/close 13116 (time 1788924151.15 total 860.25 last 8.79) - open/close 13236 (time 1788924161.75 total 870.85 last 11.32) - open/close 13459 (time 1788924171.93 total 881.03 last 21.90) - open/close 13673 (time 1788924182.56 total 891.66 last 20.13) - open/close 13867 (time 1788924193.00 total 902.10 last 18.59) - open/close 14002 (time 1788924203.00 total 912.10 last 13.49) - open/close 14187 (time 1788924213.21 total 922.31 last 18.12) - open/close 14435 (time 1788924225.12 total 934.22 last 20.82) - open/close 14697 (time 1788924235.13 total 944.23 last 26.18) - open/close 14885 (time 1788924245.61 total 954.70 last 17.95) - open/close 15165 (time 1788924255.61 total 964.71 last 27.98) - open/close 15344 (time 1788924266.05 total 975.15 last 17.14) - open/close 15537 (time 1788924277.09 total 986.19 last 17.49) - open/close 15787 (time 1788924287.15 total 996.25 last 24.86) - open/close 15968 (time 1788924297.77 total 1006.87 last 17.04) - open/close 16105 (time 1788924308.51 total 1017.61 last 12.76) - open/close 16360 (time 1788924319.02 total 1028.12 last 24.26) - open/close 16641 (time 1788924329.02 total 1038.12 last 28.08) - open/close 16782 (time 1788924340.59 total 1049.69 last 12.19) - open/close 16936 (time 1788924350.61 total 1059.71 last 15.37) - open/close 17002 (time 1788924361.55 total 1070.65 last 6.03) - open/close 17223 (time 1788924373.85 total 1082.95 last 17.96) - open/close 17449 (time 1788924383.86 total 1092.96 last 22.57) - open/close 17564 (time 1788924393.96 total 1103.06 last 11.39) - open/close 17837 (time 1788924404.65 total 1113.75 last 25.53) - open/close 18196 (time 1788924414.65 total 1123.75 last 35.90) - open/close 19122 (time 1788924424.66 total 1133.76 last 92.57) - open/close 19970 (time 1788924434.66 total 1143.76 last 84.74) - open/close 20708 (time 1788924444.67 total 1153.76 last 73.79) - open/close 21510 (time 1788924454.67 total 1163.77 last 80.13) - open/close 22274 (time 1788924464.68 total 1173.78 last 76.37) - open/close 22983 (time 1788924474.68 total 1183.78 last 70.88) - open/close 24050 (time 1788924484.69 total 1193.79 last 106.58) - open/close 24884 (time 1788924494.69 total 1203.79 last 83.38) - open/close 25824 (time 1788924504.70 total 1213.80 last 93.94) - open/close 26742 (time 1788924514.70 total 1223.80 last 91.78) - open/close 27679 (time 1788924524.71 total 1233.81 last 93.66) - open/close 28482 (time 1788924534.71 total 1243.81 last 80.25) - open/close 29355 (time 1788924544.72 total 1253.82 last 87.25) - open/close 29979 (time 1788924554.74 total 1263.84 last 62.26) - open/close 30636 (time 1788924564.74 total 1273.84 last 65.69) - open/close 31331 (time 1788924574.76 total 1283.86 last 69.35) - open/close 32207 (time 1788924584.76 total 1293.86 last 87.59) - open/close 33069 (time 1788924594.77 total 1303.87 last 86.14) - open/close 33654 (time 1788924604.77 total 1313.87 last 58.49) - open/close 34570 (time 1788924614.78 total 1323.88 last 91.53) - open/close 35336 (time 1788924624.79 total 1333.89 last 76.53) - open/close 36143 (time 1788924634.80 total 1343.90 last 80.60) - open/close 37038 (time 1788924644.80 total 1353.90 last 89.49) - open/close 37863 (time 1788924654.81 total 1363.91 last 82.47) - open/close 38543 (time 1788924664.82 total 1373.92 last 67.88) - open/close 39045 (time 1788924674.84 total 1383.94 last 50.13) - open/close 39724 (time 1788924684.84 total 1393.94 last 67.87) - open/close 40000 (time 1788924688.06 total 1397.16 last 85.83) - open/close 40800 (time 1788924698.06 total 1407.16 last 79.98) - open/close 41640 (time 1788924708.07 total 1417.17 last 83.93) - open/close 42265 (time 1788924718.09 total 1427.19 last 62.37) - open/close 43133 (time 1788924728.10 total 1437.19 last 86.76) - open/close 43940 (time 1788924738.11 total 1447.21 last 80.61) - open/close 44786 (time 1788924748.12 total 1457.22 last 84.48) - open/close 45706 (time 1788924758.13 total 1467.23 last 91.90) - open/close 46486 (time 1788924768.15 total 1477.25 last 77.83) - open/close 47415 (time 1788924778.16 total 1487.26 last 92.85) - open/close 48248 (time 1788924788.16 total 1497.26 last 83.30) - open/close 49099 (time 1788924798.16 total 1507.26 last 85.10) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1513.76 seconds: 32.79 ops/second - unlinked 0 (time 1788924808 ; total 0 ; last 0) - unlinked 10000 (time 1788924922 ; total 114 ; last 114) - unlinked 20000 (time 1788925035 ; total 227 ; last 113) - unlinked 30000 (time 1788925105 ; total 297 ; last 70) - unlinked 40000 (time 1788925163 ; total 355 ; last 58) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 428 seconds: 115.960281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: kmemleak disabled modules unloaded. pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2