== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:10:08 (1785316208) start mds service on oleg431-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg431-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg431-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 144 (time 1785316346.80 total 11.74 last 12.27) - open/close 279 (time 1785316357.85 total 22.79 last 12.22) - open/close 422 (time 1785316370.27 total 35.21 last 11.52) - open/close 547 (time 1785316381.34 total 46.28 last 11.29) - open/close 697 (time 1785316391.86 total 56.79 last 14.27) - open/close 787 (time 1785316402.19 total 67.12 last 8.71) - open/close 859 (time 1785316413.00 total 77.94 last 6.66) - open/close 953 (time 1785316423.57 total 88.50 last 8.90) - open/close 1153 (time 1785316433.99 total 98.93 last 19.18) - open/close 1284 (time 1785316444.25 total 109.18 last 12.78) - open/close 1512 (time 1785316454.27 total 119.21 last 22.75) - open/close 1603 (time 1785316464.92 total 129.86 last 8.54) - open/close 1709 (time 1785316475.98 total 140.92 last 9.58) - open/close 1805 (time 1785316486.86 total 151.80 last 8.83) - open/close 1927 (time 1785316496.88 total 161.82 last 12.17) - open/close 2021 (time 1785316506.89 total 171.83 last 9.39) - open/close 2152 (time 1785316516.91 total 181.85 last 13.07) - open/close 2298 (time 1785316527.63 total 192.57 last 13.62) - open/close 2457 (time 1785316537.65 total 202.59 last 15.86) - open/close 2544 (time 1785316548.19 total 213.12 last 8.26) - open/close 2613 (time 1785316558.19 total 223.13 last 6.89) - open/close 2731 (time 1785316569.24 total 234.18 last 10.68) - open/close 2859 (time 1785316580.11 total 245.04 last 11.78) - open/close 2988 (time 1785316591.00 total 255.94 last 11.83) - open/close 3112 (time 1785316601.75 total 266.68 last 11.54) - open/close 3163 (time 1785316612.24 total 277.18 last 4.86) - open/close 3245 (time 1785316622.61 total 287.54 last 7.91) - open/close 3333 (time 1785316632.64 total 297.58 last 8.77) - open/close 3472 (time 1785316642.65 total 307.58 last 13.89) - open/close 3543 (time 1785316652.68 total 317.62 last 7.08) - open/close 3710 (time 1785316663.75 total 328.69 last 15.09) - open/close 3873 (time 1785316674.64 total 339.58 last 14.97) - open/close 3981 (time 1785316684.65 total 349.58 last 10.80) - open/close 4096 (time 1785316694.66 total 359.59 last 11.49) - open/close 4186 (time 1785316704.66 total 369.59 last 9.00) - open/close 4335 (time 1785316714.66 total 379.60 last 14.90) - open/close 4402 (time 1785316725.40 total 390.33 last 6.24) - open/close 4572 (time 1785316735.82 total 400.76 last 16.31) - open/close 4779 (time 1785316746.22 total 411.15 last 19.91) - open/close 4878 (time 1785316756.43 total 421.37 last 9.69) - open/close 4989 (time 1785316767.25 total 432.18 last 10.27) - open/close 5116 (time 1785316777.25 total 442.18 last 12.70) - open/close 5301 (time 1785316787.39 total 452.33 last 18.23) - open/close 5462 (time 1785316797.73 total 462.66 last 15.58) - open/close 5579 (time 1785316808.78 total 473.71 last 10.59) - open/close 5639 (time 1785316819.41 total 484.35 last 5.64) - open/close 5764 (time 1785316829.44 total 494.38 last 12.46) - open/close 5943 (time 1785316839.47 total 504.40 last 17.86) - open/close 6119 (time 1785316849.81 total 514.74 last 17.02) - open/close 6238 (time 1785316860.69 total 525.63 last 10.94) - open/close 6389 (time 1785316870.72 total 535.66 last 15.05) - open/close 6523 (time 1785316880.74 total 545.68 last 13.37) - open/close 6686 (time 1785316891.73 total 556.67 last 14.83) - open/close 6879 (time 1785316901.81 total 566.74 last 19.15) - open/close 6995 (time 1785316911.83 total 576.77 last 11.57) - open/close 7113 (time 1785316921.88 total 586.82 last 11.74) - open/close 7239 (time 1785316932.49 total 597.43 last 11.87) - open/close 7406 (time 1785316942.63 total 607.56 last 16.48) - open/close 7487 (time 1785316952.91 total 617.85 last 7.88) - open/close 7597 (time 1785316962.92 total 627.85 last 10.99) - open/close 7703 (time 1785316972.95 total 637.89 last 10.57) - open/close 7779 (time 1785316983.08 total 648.01 last 7.50) - open/close 7873 (time 1785316993.30 total 658.24 last 9.19) - open/close 8102 (time 1785317003.67 total 668.60 last 22.09) - open/close 8247 (time 1785317014.54 total 679.48 last 13.33) - open/close 8400 (time 1785317024.58 total 689.52 last 15.24) - open/close 8503 (time 1785317035.03 total 699.97 last 9.85) - open/close 8628 (time 1785317045.42 total 710.36 last 12.03) - open/close 8770 (time 1785317056.47 total 721.40 last 12.86) - open/close 8901 (time 1785317066.48 total 731.41 last 13.08) - open/close 8990 (time 1785317076.69 total 741.62 last 8.72) - open/close 9127 (time 1785317087.74 total 752.68 last 12.40) - open/close 9289 (time 1785317098.31 total 763.25 last 15.32) - open/close 9414 (time 1785317108.87 total 773.81 last 11.84) - open/close 9592 (time 1785317118.87 total 783.81 last 17.79) - open/close 9739 (time 1785317129.68 total 794.62 last 13.60) - open/close 9875 (time 1785317139.74 total 804.68 last 13.52) - open/close 10000 (time 1785317144.65 total 809.59 last 25.45) - open/close 10049 (time 1785317155.03 total 819.96 last 4.72) - open/close 10113 (time 1785317165.08 total 830.01 last 6.37) - open/close 10154 (time 1785317175.67 total 840.60 last 3.87) - open/close 10293 (time 1785317185.67 total 850.60 last 13.90) - open/close 10422 (time 1785317196.30 total 861.24 last 12.13) - open/close 10473 (time 1785317206.81 total 871.75 last 4.85) - open/close 10591 (time 1785317217.70 total 882.64 last 10.84) - open/close 10662 (time 1785317229.99 total 894.92 last 5.78) - open/close 10779 (time 1785317240.53 total 905.46 last 11.10) - open/close 10842 (time 1785317250.53 total 915.47 last 6.30) - open/close 10984 (time 1785317261.26 total 926.19 last 13.24) - open/close 11183 (time 1785317272.30 total 937.24 last 18.01) - open/close 11318 (time 1785317282.32 total 947.25 last 13.48) - open/close 11442 (time 1785317292.34 total 957.27 last 12.37) - open/close 11554 (time 1785317302.89 total 967.82 last 10.62) - open/close 11687 (time 1785317313.19 total 978.12 last 12.91) - open/close 11781 (time 1785317323.53 total 988.47 last 9.09) - open/close 11932 (time 1785317333.54 total 998.48 last 15.09) - open/close 12108 (time 1785317343.91 total 1008.84 last 16.98) - open/close 12161 (time 1785317354.45 total 1019.39 last 5.02) - open/close 12262 (time 1785317367.19 total 1032.13 last 7.93) - open/close 12375 (time 1785317377.55 total 1042.49 last 10.91) - open/close 12452 (time 1785317387.93 total 1052.87 last 7.42) - open/close 12624 (time 1785317398.01 total 1062.95 last 17.06) - open/close 12764 (time 1785317408.04 total 1072.97 last 13.96) - open/close 12904 (time 1785317418.04 total 1082.98 last 13.99) - open/close 13042 (time 1785317429.24 total 1094.17 last 12.33) - open/close 13136 (time 1785317440.40 total 1105.34 last 8.42) - open/close 13291 (time 1785317450.43 total 1115.36 last 15.46) - open/close 13442 (time 1785317460.44 total 1125.38 last 15.08) - open/close 13612 (time 1785317470.44 total 1135.38 last 16.99) - open/close 13724 (time 1785317480.46 total 1145.40 last 11.18) - open/close 13915 (time 1785317490.48 total 1155.41 last 19.07) - open/close 14050 (time 1785317501.52 total 1166.45 last 12.23) - open/close 14168 (time 1785317512.15 total 1177.08 last 11.10) - open/close 14284 (time 1785317523.12 total 1188.05 last 10.58) - open/close 14420 (time 1785317535.59 total 1200.53 last 10.90) - open/close 14493 (time 1785317546.25 total 1211.19 last 6.85) - open/close 14624 (time 1785317556.90 total 1221.83 last 12.31) - open/close 14807 (time 1785317567.37 total 1232.31 last 17.47) - open/close 14947 (time 1785317577.37 total 1242.31 last 14.00) - open/close 15050 (time 1785317588.25 total 1253.19 last 9.47) - open/close 15412 (time 1785317598.27 total 1263.20 last 36.14) - open/close 16052 (time 1785317608.30 total 1273.23 last 63.80) - open/close 17114 (time 1785317618.30 total 1283.24 last 106.15) - open/close 17716 (time 1785317628.32 total 1293.25 last 60.12) - open/close 18184 (time 1785317638.32 total 1303.25 last 46.78) - open/close 18943 (time 1785317648.36 total 1313.30 last 75.56) - open/close 19843 (time 1785317658.36 total 1323.30 last 89.99) - open/close 20447 (time 1785317668.38 total 1333.32 last 60.28) - open/close 21173 (time 1785317678.38 total 1343.32 last 72.60) - open/close 22433 (time 1785317688.39 total 1353.32 last 125.96) - open/close 23070 (time 1785317698.39 total 1363.33 last 63.68) - open/close 23623 (time 1785317708.41 total 1373.35 last 55.18) - open/close 24200 (time 1785317718.43 total 1383.37 last 57.58) - open/close 24813 (time 1785317728.45 total 1393.38 last 61.21) - open/close 25378 (time 1785317738.47 total 1403.41 last 56.37) - open/close 26197 (time 1785317748.47 total 1413.41 last 81.89) - open/close 26944 (time 1785317758.47 total 1423.41 last 74.69) - open/close 27585 (time 1785317768.50 total 1433.44 last 63.90) - open/close 28191 (time 1785317778.50 total 1443.44 last 60.59) - open/close 28649 (time 1785317788.54 total 1453.47 last 45.65) - open/close 29198 (time 1785317798.54 total 1463.48 last 54.86) - open/close 29846 (time 1785317808.55 total 1473.49 last 64.77) - open/close 30572 (time 1785317818.59 total 1483.53 last 72.30) - open/close 31053 (time 1785317828.61 total 1493.54 last 48.02) - open/close 31564 (time 1785317838.62 total 1503.56 last 51.01) - open/close 32114 (time 1785317848.63 total 1513.57 last 54.95) - open/close 32585 (time 1785317858.65 total 1523.59 last 47.02) - open/close 33103 (time 1785317868.65 total 1533.59 last 51.79) - open/close 33809 (time 1785317878.65 total 1543.59 last 70.58) - open/close 34813 (time 1785317888.66 total 1553.60 last 100.34) - open/close 35568 (time 1785317898.68 total 1563.62 last 75.34) - open/close 35974 (time 1785317908.68 total 1573.62 last 40.59) - open/close 36408 (time 1785317918.71 total 1583.64 last 43.30) - open/close 37117 (time 1785317928.73 total 1593.67 last 70.73) - open/close 38317 (time 1785317938.74 total 1603.67 last 119.96) - open/close 38867 (time 1785317948.74 total 1613.68 last 54.96) - open/close 39365 (time 1785317958.76 total 1623.69 last 49.72) - open/close 39813 (time 1785317968.76 total 1633.70 last 44.78) - open/close 40000 (time 1785317974.69 total 1639.63 last 31.55) - open/close 40521 (time 1785317984.71 total 1649.65 last 52.00) - open/close 41055 (time 1785317994.73 total 1659.67 last 53.27) - open/close 41695 (time 1785318004.75 total 1669.69 last 63.88) - open/close 42291 (time 1785318014.77 total 1679.70 last 59.52) - open/close 42843 (time 1785318024.77 total 1689.70 last 55.19) - open/close 43744 (time 1785318034.77 total 1699.71 last 90.04) - open/close 44611 (time 1785318044.79 total 1709.72 last 86.61) - open/close 45187 (time 1785318054.80 total 1719.73 last 57.53) - open/close 45774 (time 1785318064.81 total 1729.74 last 58.65) - open/close 46524 (time 1785318074.81 total 1739.75 last 74.97) - open/close 47202 (time 1785318084.82 total 1749.75 last 67.75) - open/close 48129 (time 1785318094.82 total 1759.76 last 92.67) - open/close 49174 (time 1785318104.83 total 1769.76 last 104.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1775.41 seconds: 27.95 ops/second - unlinked 0 (time 1785318115 ; total 0 ; last 0) - unlinked 10000 (time 1785318264 ; total 149 ; last 149) - unlinked 20000 (time 1785318413 ; total 298 ; last 149) - unlinked 30000 (time 1785318559 ; total 444 ; last 146) - unlinked 40000 (time 1785318688 ; total 573 ; last 129) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 700 seconds: 70.901428 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg431-server unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: modules unloaded. pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2