== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:17:59 (1786022279) start mds service on oleg442-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg442-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg442-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 93 (time 1786022425.72 total 10.03 last 9.27) - open/close 221 (time 1786022436.13 total 20.44 last 12.29) - open/close 290 (time 1786022446.26 total 30.57 last 6.81) - open/close 471 (time 1786022457.11 total 41.43 last 16.67) - open/close 656 (time 1786022467.87 total 52.19 last 17.19) - open/close 727 (time 1786022477.89 total 62.20 last 7.09) - open/close 889 (time 1786022488.96 total 73.27 last 14.64) - open/close 1027 (time 1786022499.00 total 83.31 last 13.74) - open/close 1067 (time 1786022509.29 total 93.60 last 3.89) - open/close 1218 (time 1786022519.30 total 103.61 last 15.08) - open/close 1323 (time 1786022530.29 total 114.60 last 9.56) - open/close 1496 (time 1786022540.91 total 125.23 last 16.28) - open/close 1591 (time 1786022551.30 total 135.62 last 9.14) - open/close 1729 (time 1786022561.64 total 145.95 last 13.35) - open/close 1840 (time 1786022572.60 total 156.92 last 10.13) - open/close 1990 (time 1786022583.47 total 167.78 last 13.81) - open/close 2150 (time 1786022593.48 total 177.79 last 15.99) - open/close 2257 (time 1786022605.29 total 189.60 last 9.06) - open/close 2283 (time 1786022616.18 total 200.49 last 2.39) - open/close 2369 (time 1786022627.11 total 211.43 last 7.87) - open/close 2512 (time 1786022637.42 total 221.73 last 13.87) - open/close 2658 (time 1786022647.45 total 231.76 last 14.56) - open/close 2774 (time 1786022658.31 total 242.63 last 10.68) - open/close 2888 (time 1786022669.22 total 253.54 last 10.45) - open/close 2987 (time 1786022679.80 total 264.11 last 9.36) - open/close 3051 (time 1786022689.80 total 274.12 last 6.40) - open/close 3174 (time 1786022700.46 total 284.77 last 11.54) - open/close 3332 (time 1786022711.09 total 295.40 last 14.87) - open/close 3521 (time 1786022721.11 total 305.42 last 18.86) - open/close 3632 (time 1786022731.13 total 315.45 last 11.07) - open/close 3729 (time 1786022741.18 total 325.49 last 9.66) - open/close 3880 (time 1786022751.89 total 336.20 last 14.10) - open/close 4041 (time 1786022762.70 total 347.01 last 14.90) - open/close 4211 (time 1786022772.70 total 357.01 last 16.99) - open/close 4307 (time 1786022782.91 total 367.22 last 9.41) - open/close 4447 (time 1786022793.33 total 377.64 last 13.43) - open/close 4539 (time 1786022803.97 total 388.28 last 8.65) - open/close 4652 (time 1786022814.00 total 398.31 last 11.27) - open/close 4759 (time 1786022824.48 total 408.79 last 10.21) - open/close 4923 (time 1786022834.49 total 418.80 last 16.37) - open/close 5044 (time 1786022845.54 total 429.85 last 10.95) - open/close 5165 (time 1786022855.74 total 440.05 last 11.86) - open/close 5297 (time 1786022865.74 total 450.06 last 13.19) - open/close 5420 (time 1786022876.14 total 460.45 last 11.83) - open/close 5535 (time 1786022886.59 total 470.90 last 11.01) - open/close 5731 (time 1786022896.88 total 481.19 last 19.05) - open/close 5883 (time 1786022906.88 total 491.19 last 15.19) - open/close 5997 (time 1786022916.89 total 501.20 last 11.39) - open/close 6094 (time 1786022927.66 total 511.97 last 9.00) - open/close 6214 (time 1786022938.09 total 522.40 last 11.51) - open/close 6341 (time 1786022948.99 total 533.30 last 11.65) - open/close 6478 (time 1786022959.54 total 543.85 last 12.99) - open/close 6666 (time 1786022969.62 total 553.94 last 18.64) - open/close 6735 (time 1786022979.85 total 564.16 last 6.75) - open/close 6800 (time 1786022989.87 total 574.18 last 6.49) - open/close 6908 (time 1786022999.89 total 584.20 last 10.77) - open/close 7043 (time 1786023010.61 total 594.92 last 12.59) - open/close 7163 (time 1786023020.62 total 604.93 last 11.99) - open/close 7254 (time 1786023033.25 total 617.56 last 7.21) - open/close 7393 (time 1786023043.90 total 628.21 last 13.05) - open/close 7464 (time 1786023054.46 total 638.77 last 6.72) - open/close 7592 (time 1786023065.42 total 649.74 last 11.68) - open/close 7705 (time 1786023075.44 total 659.75 last 11.28) - open/close 7882 (time 1786023085.44 total 669.75 last 17.70) - open/close 7969 (time 1786023098.25 total 682.56 last 6.79) - open/close 8063 (time 1786023108.89 total 693.21 last 8.83) - open/close 8135 (time 1786023118.91 total 703.22 last 7.19) - open/close 8238 (time 1786023128.99 total 713.30 last 10.22) - open/close 8385 (time 1786023139.02 total 723.33 last 14.65) - open/close 8538 (time 1786023149.43 total 733.74 last 14.71) - open/close 8644 (time 1786023160.29 total 744.61 last 9.75) - open/close 8714 (time 1786023170.67 total 754.98 last 6.75) - open/close 8847 (time 1786023181.68 total 765.99 last 12.08) - open/close 8936 (time 1786023192.61 total 776.92 last 8.14) - open/close 9069 (time 1786023203.00 total 787.31 last 12.80) - open/close 9127 (time 1786023214.06 total 798.37 last 5.25) - open/close 9226 (time 1786023224.50 total 808.82 last 9.48) - open/close 9416 (time 1786023235.52 total 819.83 last 17.25) - open/close 9579 (time 1786023245.52 total 829.83 last 16.30) - open/close 9668 (time 1786023255.54 total 839.85 last 8.89) - open/close 9790 (time 1786023266.01 total 850.32 last 11.65) - open/close 9879 (time 1786023276.57 total 860.88 last 8.43) - open/close 9976 (time 1786023287.09 total 871.40 last 9.22) - open/close 10000 (time 1786023289.80 total 874.11 last 8.86) - open/close 10129 (time 1786023300.07 total 884.39 last 12.55) - open/close 10266 (time 1786023311.08 total 895.39 last 12.45) - open/close 10400 (time 1786023321.09 total 905.41 last 13.38) - open/close 10535 (time 1786023332.07 total 916.38 last 12.30) - open/close 10695 (time 1786023342.09 total 926.41 last 15.96) - open/close 10858 (time 1786023352.11 total 936.42 last 16.28) - open/close 10914 (time 1786023362.13 total 946.45 last 5.58) - open/close 11036 (time 1786023372.16 total 956.47 last 12.16) - open/close 11169 (time 1786023382.49 total 966.80 last 12.88) - open/close 11286 (time 1786023393.46 total 977.77 last 10.66) - open/close 11375 (time 1786023404.53 total 988.84 last 8.04) - open/close 11488 (time 1786023414.58 total 998.89 last 11.24) - open/close 11636 (time 1786023424.65 total 1008.96 last 14.71) - open/close 11719 (time 1786023435.18 total 1019.49 last 7.88) - open/close 11829 (time 1786023445.22 total 1029.53 last 10.96) - open/close 12018 (time 1786023456.16 total 1040.47 last 17.27) - open/close 12188 (time 1786023466.80 total 1051.11 last 15.98) - open/close 12293 (time 1786023476.82 total 1061.13 last 10.48) - open/close 12433 (time 1786023487.49 total 1071.80 last 13.12) - open/close 12577 (time 1786023497.96 total 1082.27 last 13.76) - open/close 12700 (time 1786023508.08 total 1092.39 last 12.15) - open/close 12800 (time 1786023518.08 total 1102.39 last 10.00) - open/close 12933 (time 1786023528.08 total 1112.40 last 13.30) - open/close 13043 (time 1786023538.99 total 1123.30 last 10.09) - open/close 13149 (time 1786023549.86 total 1134.17 last 9.75) - open/close 13260 (time 1786023560.81 total 1145.12 last 10.13) - open/close 13423 (time 1786023570.82 total 1155.13 last 16.29) - open/close 13492 (time 1786023580.85 total 1165.16 last 6.88) - open/close 13631 (time 1786023590.85 total 1175.16 last 13.89) - open/close 13731 (time 1786023601.26 total 1185.57 last 9.61) - open/close 13910 (time 1786023611.26 total 1195.57 last 17.90) - open/close 14058 (time 1786023621.99 total 1206.30 last 13.79) - open/close 14203 (time 1786023632.35 total 1216.66 last 14.00) - open/close 14299 (time 1786023643.18 total 1227.50 last 8.86) - open/close 14387 (time 1786023653.94 total 1238.25 last 8.18) - open/close 14538 (time 1786023665.57 total 1249.88 last 12.98) - open/close 14689 (time 1786023675.58 total 1259.89 last 15.09) - open/close 14812 (time 1786023686.05 total 1270.36 last 11.75) - open/close 14938 (time 1786023696.15 total 1280.47 last 12.47) - open/close 15005 (time 1786023707.20 total 1291.51 last 6.06) - open/close 15070 (time 1786023717.20 total 1301.52 last 6.50) - open/close 15179 (time 1786023727.23 total 1311.54 last 10.88) - open/close 15306 (time 1786023737.89 total 1322.21 last 11.90) - open/close 15449 (time 1786023748.22 total 1332.53 last 13.84) - open/close 15543 (time 1786023758.69 total 1343.00 last 8.98) - open/close 15659 (time 1786023769.95 total 1354.27 last 10.30) - open/close 15812 (time 1786023779.96 total 1364.27 last 15.29) - open/close 16243 (time 1786023789.96 total 1374.28 last 43.08) - open/close 16654 (time 1786023799.97 total 1384.28 last 41.07) - open/close 17008 (time 1786023809.99 total 1394.31 last 35.32) - open/close 17584 (time 1786023820.00 total 1404.31 last 57.55) - open/close 18078 (time 1786023830.02 total 1414.33 last 49.32) - open/close 18583 (time 1786023840.03 total 1424.34 last 50.44) - open/close 19065 (time 1786023850.07 total 1434.38 last 48.00) - open/close 19806 (time 1786023860.07 total 1444.39 last 74.08) - open/close 20000 (time 1786023864.17 total 1448.48 last 47.42) - open/close 20546 (time 1786023874.17 total 1458.49 last 54.56) - open/close 21021 (time 1786023884.19 total 1468.50 last 47.41) - open/close 21335 (time 1786023894.20 total 1478.51 last 31.38) - open/close 21818 (time 1786023904.22 total 1488.53 last 48.20) - open/close 22234 (time 1786023914.23 total 1498.54 last 41.57) - open/close 22764 (time 1786023924.25 total 1508.56 last 52.89) - open/close 23510 (time 1786023934.27 total 1518.58 last 74.46) - open/close 24056 (time 1786023944.27 total 1528.58 last 54.59) - open/close 24489 (time 1786023954.30 total 1538.61 last 43.17) - open/close 24995 (time 1786023964.30 total 1548.61 last 50.59) - open/close 25435 (time 1786023974.31 total 1558.62 last 43.95) - open/close 25833 (time 1786023984.32 total 1568.63 last 39.78) - open/close 26313 (time 1786023994.32 total 1578.63 last 47.98) - open/close 27036 (time 1786024004.33 total 1588.64 last 72.22) - open/close 27726 (time 1786024014.33 total 1598.64 last 68.98) - open/close 28149 (time 1786024024.35 total 1608.66 last 42.22) - open/close 28763 (time 1786024034.36 total 1618.67 last 61.39) - open/close 29356 (time 1786024044.36 total 1628.68 last 59.25) - open/close 29923 (time 1786024054.37 total 1638.68 last 56.69) - open/close 30445 (time 1786024064.38 total 1648.69 last 52.13) - open/close 31045 (time 1786024074.38 total 1658.69 last 59.97) - open/close 31593 (time 1786024084.39 total 1668.70 last 54.77) - open/close 32039 (time 1786024094.43 total 1678.75 last 44.39) - open/close 32427 (time 1786024104.45 total 1688.76 last 38.75) - open/close 32918 (time 1786024114.45 total 1698.76 last 49.08) - open/close 33369 (time 1786024124.46 total 1708.77 last 45.08) - open/close 33937 (time 1786024134.47 total 1718.78 last 56.73) - open/close 34547 (time 1786024144.48 total 1728.79 last 60.96) - open/close 35203 (time 1786024154.49 total 1738.80 last 65.53) - open/close 36112 (time 1786024164.49 total 1748.81 last 90.83) - open/close 36655 (time 1786024174.51 total 1758.82 last 54.24) - open/close 37254 (time 1786024184.51 total 1768.82 last 59.87) - open/close 37901 (time 1786024194.52 total 1778.83 last 64.64) - open/close 38423 (time 1786024204.52 total 1788.84 last 52.18) - open/close 38876 (time 1786024214.53 total 1798.84 last 45.27) - open/close 39278 (time 1786024224.55 total 1808.86 last 40.11) - open/close 39777 (time 1786024234.56 total 1818.87 last 49.86) - open/close 40000 (time 1786024237.18 total 1821.49 last 85.13) - open/close 40538 (time 1786024247.20 total 1831.51 last 53.69) - open/close 40948 (time 1786024257.22 total 1841.53 last 40.93) - open/close 41559 (time 1786024267.24 total 1851.56 last 60.94) - open/close 42048 (time 1786024277.25 total 1861.56 last 48.88) - open/close 42865 (time 1786024287.26 total 1871.57 last 81.61) - open/close 43329 (time 1786024297.27 total 1881.58 last 46.37) - open/close 44116 (time 1786024307.27 total 1891.58 last 78.68) - open/close 44760 (time 1786024317.28 total 1901.59 last 64.37) - open/close 45179 (time 1786024327.29 total 1911.60 last 41.84) - open/close 45830 (time 1786024337.30 total 1921.61 last 65.05) - open/close 46349 (time 1786024347.30 total 1931.61 last 51.88) - open/close 46828 (time 1786024357.31 total 1941.63 last 47.84) - open/close 47239 (time 1786024367.32 total 1951.63 last 41.07) - open/close 47624 (time 1786024377.35 total 1961.66 last 38.40) - open/close 47958 (time 1786024387.37 total 1971.69 last 33.31) - open/close 48432 (time 1786024397.39 total 1981.70 last 47.34) - open/close 49013 (time 1786024407.39 total 1991.70 last 58.07) - open/close 49403 (time 1786024417.41 total 2001.72 last 38.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 2005.20 seconds: 24.75 ops/second - unlinked 0 (time 1786024427 ; total 1 ; last 1) - unlinked 10000 (time 1786024617 ; total 191 ; last 190) - unlinked 20000 (time 1786024795 ; total 369 ; last 178) - unlinked 30000 (time 1786025000 ; total 574 ; last 205) - unlinked 40000 (time 1786025155 ; total 729 ; last 155) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 931 seconds: 53.309345 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: modules unloaded. pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2