== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:25:35 (1785399935) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg234-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1785400081.80 total 10.39 last 10.10) - open/close 191 (time 1785400092.86 total 21.45 last 7.78) - open/close 250 (time 1785400103.83 total 32.42 last 5.38) - open/close 338 (time 1785400114.91 total 43.50 last 7.94) - open/close 480 (time 1785400125.13 total 53.72 last 13.90) - open/close 660 (time 1785400135.17 total 63.76 last 17.92) - open/close 824 (time 1785400146.05 total 74.64 last 15.07) - open/close 952 (time 1785400156.50 total 85.09 last 12.26) - open/close 1108 (time 1785400166.98 total 95.57 last 14.88) - open/close 1218 (time 1785400178.08 total 106.67 last 9.91) - open/close 1293 (time 1785400188.24 total 116.83 last 7.38) - open/close 1508 (time 1785400199.24 total 127.83 last 19.54) - open/close 1595 (time 1785400209.93 total 138.52 last 8.14) - open/close 1739 (time 1785400220.24 total 148.83 last 13.97) - open/close 1902 (time 1785400231.05 total 159.64 last 15.08) - open/close 2074 (time 1785400241.61 total 170.20 last 16.29) - open/close 2314 (time 1785400251.98 total 180.57 last 23.13) - open/close 2445 (time 1785400262.53 total 191.12 last 12.42) - open/close 2626 (time 1785400272.54 total 201.13 last 18.09) - open/close 2754 (time 1785400282.82 total 211.41 last 12.45) - open/close 2917 (time 1785400293.27 total 221.86 last 15.60) - open/close 3062 (time 1785400303.38 total 231.97 last 14.35) - open/close 3191 (time 1785400314.06 total 242.65 last 12.08) - open/close 3420 (time 1785400324.55 total 253.14 last 21.83) - open/close 3512 (time 1785400334.55 total 263.14 last 9.20) - open/close 3643 (time 1785400344.56 total 273.15 last 13.09) - open/close 3798 (time 1785400354.56 total 283.15 last 15.50) - open/close 3936 (time 1785400366.03 total 294.62 last 12.03) - open/close 4064 (time 1785400377.05 total 305.64 last 11.61) - open/close 4159 (time 1785400387.08 total 315.67 last 9.47) - open/close 4313 (time 1785400397.08 total 325.67 last 15.39) - open/close 4460 (time 1785400407.64 total 336.23 last 13.92) - open/close 4568 (time 1785400418.38 total 346.97 last 10.06) - open/close 4683 (time 1785400428.54 total 357.13 last 11.32) - open/close 4885 (time 1785400439.33 total 367.92 last 18.73) - open/close 5010 (time 1785400449.46 total 378.05 last 12.34) - open/close 5175 (time 1785400459.48 total 388.07 last 16.46) - open/close 5360 (time 1785400469.48 total 398.07 last 18.49) - open/close 5536 (time 1785400479.69 total 408.28 last 17.25) - open/close 5746 (time 1785400490.05 total 418.64 last 20.27) - open/close 5985 (time 1785400501.47 total 430.06 last 20.92) - open/close 6104 (time 1785400512.23 total 440.82 last 11.06) - open/close 6216 (time 1785400522.25 total 450.84 last 11.18) - open/close 6307 (time 1785400532.61 total 461.20 last 8.78) - open/close 6370 (time 1785400543.66 total 472.25 last 5.71) - open/close 6498 (time 1785400553.67 total 482.26 last 12.78) - open/close 6653 (time 1785400563.68 total 492.27 last 15.49) - open/close 6766 (time 1785400573.68 total 502.27 last 11.30) - open/close 6875 (time 1785400583.69 total 512.28 last 10.89) - open/close 6995 (time 1785400593.92 total 522.51 last 11.73) - open/close 7103 (time 1785400604.31 total 532.90 last 10.40) - open/close 7190 (time 1785400614.93 total 543.52 last 8.19) - open/close 7297 (time 1785400624.97 total 553.56 last 10.65) - open/close 7406 (time 1785400634.98 total 563.57 last 10.89) - open/close 7536 (time 1785400645.33 total 573.92 last 12.56) - open/close 7604 (time 1785400655.36 total 583.95 last 6.78) - open/close 7727 (time 1785400665.54 total 594.13 last 12.08) - open/close 7881 (time 1785400675.56 total 604.14 last 15.38) - open/close 8060 (time 1785400686.42 total 615.01 last 16.47) - open/close 8126 (time 1785400696.45 total 625.04 last 6.58) - open/close 8273 (time 1785400706.78 total 635.37 last 14.23) - open/close 8355 (time 1785400716.80 total 645.39 last 8.18) - open/close 8503 (time 1785400727.62 total 656.21 last 13.68) - open/close 8626 (time 1785400738.19 total 666.78 last 11.64) - open/close 8688 (time 1785400748.88 total 677.47 last 5.80) - open/close 8870 (time 1785400762.07 total 690.66 last 13.80) - open/close 8977 (time 1785400772.87 total 701.46 last 9.91) - open/close 9082 (time 1785400783.44 total 712.02 last 9.94) - open/close 9277 (time 1785400793.44 total 722.03 last 19.50) - open/close 9427 (time 1785400804.23 total 732.82 last 13.90) - open/close 9540 (time 1785400814.53 total 743.12 last 10.97) - open/close 9702 (time 1785400824.55 total 753.14 last 16.17) - open/close 9791 (time 1785400834.58 total 763.17 last 8.88) - open/close 9891 (time 1785400844.59 total 773.18 last 9.98) - open/close 10000 (time 1785400851.60 total 780.19 last 15.55) - open/close 10065 (time 1785400862.57 total 791.16 last 5.93) - open/close 10136 (time 1785400872.57 total 801.16 last 7.10) - open/close 10256 (time 1785400882.59 total 811.18 last 11.98) - open/close 10361 (time 1785400893.26 total 821.85 last 9.84) - open/close 10472 (time 1785400903.36 total 831.95 last 10.99) - open/close 10619 (time 1785400913.46 total 842.05 last 14.56) - open/close 10810 (time 1785400923.73 total 852.32 last 18.59) - open/close 10887 (time 1785400933.97 total 862.56 last 7.53) - open/close 10986 (time 1785400944.20 total 872.79 last 9.67) - open/close 11118 (time 1785400954.21 total 882.80 last 13.18) - open/close 11274 (time 1785400964.48 total 893.07 last 15.19) - open/close 11455 (time 1785400974.55 total 903.14 last 17.98) - open/close 11552 (time 1785400984.56 total 913.15 last 9.69) - open/close 11697 (time 1785400994.56 total 923.15 last 14.49) - open/close 11810 (time 1785401005.16 total 933.75 last 10.67) - open/close 11983 (time 1785401015.24 total 943.83 last 17.17) - open/close 12160 (time 1785401026.13 total 954.72 last 16.24) - open/close 12269 (time 1785401037.18 total 965.77 last 9.86) - open/close 12372 (time 1785401047.19 total 975.78 last 10.29) - open/close 12474 (time 1785401057.31 total 985.90 last 10.09) - open/close 12641 (time 1785401069.67 total 998.25 last 13.51) - open/close 12839 (time 1785401080.41 total 1009.00 last 18.42) - open/close 12993 (time 1785401090.43 total 1019.02 last 15.38) - open/close 13095 (time 1785401101.03 total 1029.62 last 9.62) - open/close 13215 (time 1785401113.29 total 1041.88 last 9.79) - open/close 13352 (time 1785401124.26 total 1052.85 last 12.49) - open/close 13525 (time 1785401134.77 total 1063.36 last 16.46) - open/close 13607 (time 1785401145.50 total 1074.09 last 7.64) - open/close 13738 (time 1785401156.36 total 1084.95 last 12.06) - open/close 13846 (time 1785401166.37 total 1094.96 last 10.80) - open/close 13949 (time 1785401176.37 total 1104.96 last 10.30) - open/close 14033 (time 1785401186.82 total 1115.41 last 8.04) - open/close 14129 (time 1785401196.83 total 1125.42 last 9.59) - open/close 14247 (time 1785401207.27 total 1135.86 last 11.30) - open/close 14379 (time 1785401218.30 total 1146.89 last 11.97) - open/close 14534 (time 1785401228.43 total 1157.02 last 15.31) - open/close 14718 (time 1785401238.44 total 1167.03 last 18.38) - open/close 14857 (time 1785401249.36 total 1177.95 last 12.73) - open/close 15002 (time 1785401259.39 total 1187.98 last 14.46) - open/close 15135 (time 1785401269.39 total 1197.98 last 13.30) - open/close 15271 (time 1785401279.75 total 1208.34 last 13.13) - open/close 15389 (time 1785401289.77 total 1218.36 last 11.78) - open/close 15517 (time 1785401300.63 total 1229.22 last 11.78) - open/close 15732 (time 1785401311.46 total 1240.05 last 19.86) - open/close 15996 (time 1785401322.25 total 1250.84 last 24.45) - open/close 16114 (time 1785401332.50 total 1261.09 last 11.52) - open/close 16219 (time 1785401343.32 total 1271.91 last 9.71) - open/close 16367 (time 1785401353.62 total 1282.21 last 14.36) - open/close 16471 (time 1785401363.64 total 1292.23 last 10.39) - open/close 16596 (time 1785401374.24 total 1302.83 last 11.79) - open/close 16753 (time 1785401384.25 total 1312.84 last 15.69) - open/close 16921 (time 1785401394.39 total 1322.98 last 16.58) - open/close 17034 (time 1785401404.39 total 1332.98 last 11.30) - open/close 17151 (time 1785401415.37 total 1343.96 last 10.65) - open/close 17258 (time 1785401426.20 total 1354.79 last 9.88) - open/close 17360 (time 1785401436.82 total 1365.41 last 9.60) - open/close 17835 (time 1785401446.82 total 1375.41 last 47.48) - open/close 18318 (time 1785401456.83 total 1385.42 last 48.26) - open/close 19140 (time 1785401466.84 total 1395.43 last 82.16) - open/close 19829 (time 1785401476.85 total 1405.44 last 68.81) - open/close 20000 (time 1785401480.38 total 1408.97 last 48.46) - open/close 20445 (time 1785401490.39 total 1418.98 last 44.46) - open/close 20865 (time 1785401500.39 total 1428.98 last 42.00) - open/close 21380 (time 1785401510.40 total 1438.99 last 51.43) - open/close 22087 (time 1785401520.42 total 1449.01 last 70.58) - open/close 22749 (time 1785401530.43 total 1459.02 last 66.12) - open/close 23427 (time 1785401540.44 total 1469.03 last 67.73) - open/close 23862 (time 1785401550.48 total 1479.07 last 43.31) - open/close 24136 (time 1785401560.73 total 1489.32 last 26.75) - open/close 24477 (time 1785401570.74 total 1499.33 last 34.06) - open/close 24848 (time 1785401580.75 total 1509.34 last 37.07) - open/close 25296 (time 1785401590.76 total 1519.35 last 44.75) - open/close 25699 (time 1785401600.78 total 1529.37 last 40.22) - open/close 26215 (time 1785401610.78 total 1539.37 last 51.58) - open/close 27005 (time 1785401620.81 total 1549.40 last 78.74) - open/close 27543 (time 1785401630.82 total 1559.41 last 53.75) - open/close 28218 (time 1785401640.84 total 1569.43 last 67.38) - open/close 28901 (time 1785401650.85 total 1579.44 last 68.24) - open/close 29469 (time 1785401660.89 total 1589.48 last 56.56) - open/close 30000 (time 1785401668.12 total 1596.71 last 73.46) - open/close 30744 (time 1785401678.12 total 1606.71 last 74.36) - open/close 31536 (time 1785401688.13 total 1616.72 last 79.13) - open/close 32067 (time 1785401698.14 total 1626.73 last 53.08) - open/close 32981 (time 1785401708.14 total 1636.73 last 91.33) - open/close 33614 (time 1785401718.16 total 1646.75 last 63.18) - open/close 34261 (time 1785401728.16 total 1656.75 last 64.69) - open/close 34973 (time 1785401738.19 total 1666.78 last 71.04) - open/close 35369 (time 1785401748.21 total 1676.80 last 39.53) - open/close 35878 (time 1785401758.22 total 1686.81 last 50.82) - open/close 36344 (time 1785401768.23 total 1696.82 last 46.55) - open/close 36896 (time 1785401778.24 total 1706.83 last 55.15) - open/close 37536 (time 1785401788.26 total 1716.85 last 63.91) - open/close 38115 (time 1785401798.26 total 1726.85 last 57.86) - open/close 38684 (time 1785401808.28 total 1736.87 last 56.83) - open/close 39457 (time 1785401818.28 total 1746.87 last 77.27) - open/close 39962 (time 1785401828.29 total 1756.87 last 50.47) - open/close 40494 (time 1785401838.30 total 1766.89 last 53.14) - open/close 40944 (time 1785401848.33 total 1776.92 last 44.84) - open/close 41623 (time 1785401858.35 total 1786.94 last 67.75) - open/close 42042 (time 1785401868.38 total 1796.97 last 41.80) - open/close 42544 (time 1785401878.39 total 1806.98 last 50.16) - open/close 43112 (time 1785401888.43 total 1817.02 last 56.57) - open/close 43613 (time 1785401898.43 total 1827.02 last 50.08) - open/close 44199 (time 1785401908.43 total 1837.02 last 58.59) - open/close 44751 (time 1785401918.46 total 1847.05 last 55.06) - open/close 45235 (time 1785401928.47 total 1857.06 last 48.36) - open/close 45741 (time 1785401938.47 total 1867.06 last 50.58) - open/close 46269 (time 1785401948.49 total 1877.08 last 52.69) - open/close 46872 (time 1785401958.52 total 1887.11 last 60.15) - open/close 47306 (time 1785401968.54 total 1897.13 last 43.27) - open/close 48030 (time 1785401978.54 total 1907.13 last 72.40) - open/close 48470 (time 1785401988.59 total 1917.18 last 43.78) - open/close 48855 (time 1785401998.60 total 1927.19 last 38.49) - open/close 49277 (time 1785402008.61 total 1937.20 last 42.14) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1944.21 seconds: 25.53 ops/second - unlinked 0 (time 1785402023 ; total 0 ; last 0) - unlinked 10000 (time 1785402201 ; total 178 ; last 178) - unlinked 20000 (time 1785402366 ; total 343 ; last 165) - unlinked 30000 (time 1785402508 ; total 485 ; last 142) - unlinked 40000 (time 1785402692 ; total 669 ; last 184) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 808 seconds: 61.424503 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2