== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:03:16 (1785398596) start mds service on oleg128-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg128-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg128-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg128-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 123 (time 1785398745.74 total 10.40 last 11.82) - open/close 235 (time 1785398755.76 total 20.42 last 11.18) - open/close 440 (time 1785398765.78 total 30.45 last 20.45) - open/close 511 (time 1785398776.13 total 40.79 last 6.86) - open/close 715 (time 1785398786.95 total 51.61 last 18.85) - open/close 921 (time 1785398797.69 total 62.35 last 19.18) - open/close 1010 (time 1785398807.87 total 72.53 last 8.74) - open/close 1065 (time 1785398818.56 total 83.22 last 5.15) - open/close 1191 (time 1785398829.49 total 94.15 last 11.53) - open/close 1283 (time 1785398839.87 total 104.54 last 8.86) - open/close 1383 (time 1785398850.18 total 114.84 last 9.70) - open/close 1470 (time 1785398860.40 total 125.06 last 8.51) - open/close 1580 (time 1785398871.10 total 135.76 last 10.28) - open/close 1647 (time 1785398881.65 total 146.31 last 6.35) - open/close 1765 (time 1785398892.67 total 157.33 last 10.71) - open/close 1909 (time 1785398903.16 total 167.83 last 13.72) - open/close 2048 (time 1785398914.07 total 178.73 last 12.75) - open/close 2167 (time 1785398924.17 total 188.83 last 11.78) - open/close 2315 (time 1785398934.30 total 198.96 last 14.61) - open/close 2452 (time 1785398944.96 total 209.62 last 12.86) - open/close 2576 (time 1785398955.19 total 219.85 last 12.12) - open/close 2650 (time 1785398965.19 total 229.86 last 7.39) - open/close 2751 (time 1785398975.21 total 239.87 last 10.09) - open/close 2927 (time 1785398985.80 total 250.46 last 16.62) - open/close 3060 (time 1785398996.02 total 260.68 last 13.01) - open/close 3142 (time 1785399006.02 total 270.68 last 8.20) - open/close 3316 (time 1785399016.25 total 280.92 last 17.00) - open/close 3421 (time 1785399026.95 total 291.61 last 9.82) - open/close 3533 (time 1785399037.81 total 302.47 last 10.31) - open/close 3695 (time 1785399048.05 total 312.71 last 15.83) - open/close 3758 (time 1785399058.11 total 322.77 last 6.26) - open/close 3882 (time 1785399068.54 total 333.20 last 11.89) - open/close 4029 (time 1785399079.28 total 343.95 last 13.68) - open/close 4150 (time 1785399089.30 total 353.96 last 12.09) - open/close 4266 (time 1785399099.42 total 364.08 last 11.46) - open/close 4475 (time 1785399110.34 total 375.00 last 19.14) - open/close 4551 (time 1785399120.78 total 385.45 last 7.28) - open/close 4684 (time 1785399131.65 total 396.31 last 12.24) - open/close 4826 (time 1785399142.52 total 407.18 last 13.07) - open/close 4942 (time 1785399153.36 total 418.02 last 10.70) - open/close 5180 (time 1785399163.76 total 428.43 last 22.88) - open/close 5363 (time 1785399173.78 total 438.44 last 18.27) - open/close 5549 (time 1785399183.95 total 448.61 last 18.28) - open/close 5654 (time 1785399194.77 total 459.43 last 9.71) - open/close 5791 (time 1785399205.11 total 469.77 last 13.25) - open/close 5912 (time 1785399215.72 total 480.39 last 11.40) - open/close 6008 (time 1785399225.78 total 490.44 last 9.55) - open/close 6264 (time 1785399235.78 total 500.44 last 25.59) - open/close 6408 (time 1785399246.22 total 510.89 last 13.79) - open/close 6529 (time 1785399257.03 total 521.70 last 11.19) - open/close 6625 (time 1785399267.91 total 532.57 last 8.83) - open/close 6771 (time 1785399278.38 total 543.04 last 13.94) - open/close 6927 (time 1785399288.38 total 553.05 last 15.59) - open/close 7038 (time 1785399298.48 total 563.14 last 10.99) - open/close 7201 (time 1785399309.17 total 573.83 last 15.25) - open/close 7276 (time 1785399319.55 total 584.21 last 7.22) - open/close 7385 (time 1785399329.56 total 594.22 last 10.89) - open/close 7537 (time 1785399340.60 total 605.27 last 13.76) - open/close 7645 (time 1785399351.59 total 616.25 last 9.83) - open/close 7734 (time 1785399364.26 total 628.92 last 7.02) - open/close 7853 (time 1785399374.98 total 639.64 last 11.10) - open/close 8032 (time 1785399385.96 total 650.63 last 16.29) - open/close 8123 (time 1785399397.04 total 661.70 last 8.22) - open/close 8281 (time 1785399407.07 total 671.73 last 15.75) - open/close 8418 (time 1785399417.61 total 682.27 last 13.00) - open/close 8522 (time 1785399427.77 total 692.43 last 10.24) - open/close 8597 (time 1785399438.18 total 702.84 last 7.21) - open/close 8736 (time 1785399448.18 total 712.84 last 13.90) - open/close 8835 (time 1785399458.55 total 723.21 last 9.54) - open/close 8949 (time 1785399469.06 total 733.72 last 10.85) - open/close 9027 (time 1785399479.06 total 743.72 last 7.80) - open/close 9213 (time 1785399490.13 total 754.80 last 16.80) - open/close 9329 (time 1785399500.15 total 764.81 last 11.58) - open/close 9454 (time 1785399510.27 total 774.93 last 12.35) - open/close 9581 (time 1785399520.30 total 784.96 last 12.66) - open/close 9666 (time 1785399531.26 total 795.93 last 7.75) - open/close 9819 (time 1785399542.13 total 806.80 last 14.08) - open/close 9892 (time 1785399553.14 total 817.81 last 6.63) - open/close 9997 (time 1785399564.08 total 828.74 last 9.60) - open/close 10146 (time 1785399574.51 total 839.17 last 14.29) - open/close 10263 (time 1785399584.52 total 849.18 last 11.69) - open/close 10385 (time 1785399597.25 total 861.91 last 9.59) - open/close 10585 (time 1785399608.18 total 872.84 last 18.29) - open/close 10794 (time 1785399619.02 total 883.68 last 19.28) - open/close 10856 (time 1785399629.96 total 894.62 last 5.67) - open/close 10983 (time 1785399639.98 total 904.64 last 12.67) - open/close 11078 (time 1785399649.99 total 914.65 last 9.49) - open/close 11196 (time 1785399660.33 total 924.99 last 11.41) - open/close 11323 (time 1785399670.39 total 935.06 last 12.62) - open/close 11428 (time 1785399681.45 total 946.11 last 9.50) - open/close 11556 (time 1785399692.02 total 956.69 last 12.11) - open/close 11701 (time 1785399702.04 total 966.71 last 14.47) - open/close 11892 (time 1785399712.95 total 977.61 last 17.51) - open/close 12002 (time 1785399723.71 total 988.37 last 10.23) - open/close 12077 (time 1785399733.75 total 998.41 last 7.47) - open/close 12135 (time 1785399744.50 total 1009.16 last 5.39) - open/close 12265 (time 1785399755.05 total 1019.71 last 12.32) - open/close 12392 (time 1785399765.06 total 1029.73 last 12.68) - open/close 12545 (time 1785399776.88 total 1041.54 last 12.95) - open/close 12725 (time 1785399786.89 total 1051.55 last 17.97) - open/close 12908 (time 1785399797.59 total 1062.26 last 17.10) - open/close 13067 (time 1785399809.54 total 1074.21 last 13.30) - open/close 13271 (time 1785399819.57 total 1084.23 last 20.35) - open/close 13447 (time 1785399830.10 total 1094.76 last 16.71) - open/close 13570 (time 1785399841.16 total 1105.82 last 11.12) - open/close 13703 (time 1785399853.25 total 1117.92 last 11.00) - open/close 13748 (time 1785399865.86 total 1130.52 last 3.57) - open/close 13944 (time 1785399875.86 total 1140.52 last 19.59) - open/close 14152 (time 1785399886.85 total 1151.51 last 18.92) - open/close 14299 (time 1785399897.36 total 1162.02 last 13.99) - open/close 14468 (time 1785399907.94 total 1172.60 last 15.98) - open/close 14687 (time 1785399917.96 total 1182.62 last 21.84) - open/close 14811 (time 1785399927.98 total 1192.64 last 12.38) - open/close 14896 (time 1785399937.99 total 1202.65 last 8.49) - open/close 15030 (time 1785399948.67 total 1213.34 last 12.54) - open/close 15284 (time 1785399958.69 total 1223.35 last 25.36) - open/close 15943 (time 1785399968.69 total 1233.35 last 65.90) - open/close 16633 (time 1785399978.69 total 1243.36 last 68.97) - open/close 17334 (time 1785399988.70 total 1253.36 last 70.05) - open/close 17830 (time 1785399998.71 total 1263.37 last 49.56) - open/close 18249 (time 1785400008.72 total 1273.39 last 41.84) - open/close 18844 (time 1785400018.73 total 1283.39 last 59.48) - open/close 19441 (time 1785400028.74 total 1293.41 last 59.60) - open/close 20000 (time 1785400037.89 total 1302.55 last 61.11) - open/close 20570 (time 1785400047.89 total 1312.56 last 56.98) - open/close 21159 (time 1785400057.90 total 1322.56 last 58.89) - open/close 21785 (time 1785400067.91 total 1332.57 last 62.51) - open/close 22411 (time 1785400077.91 total 1342.58 last 62.59) - open/close 23073 (time 1785400087.92 total 1352.58 last 66.17) - open/close 23901 (time 1785400097.94 total 1362.60 last 82.62) - open/close 24266 (time 1785400107.96 total 1372.62 last 36.44) - open/close 25071 (time 1785400117.96 total 1382.62 last 80.48) - open/close 25678 (time 1785400127.96 total 1392.62 last 60.68) - open/close 26513 (time 1785400137.97 total 1402.63 last 83.45) - open/close 27127 (time 1785400147.97 total 1412.64 last 61.36) - open/close 27760 (time 1785400157.99 total 1422.65 last 63.22) - open/close 28505 (time 1785400168.00 total 1432.66 last 74.43) - open/close 29028 (time 1785400178.00 total 1442.66 last 52.29) - open/close 29472 (time 1785400188.01 total 1452.67 last 44.35) - open/close 30000 (time 1785400195.61 total 1460.27 last 69.46) - open/close 30669 (time 1785400205.62 total 1470.28 last 66.83) - open/close 31265 (time 1785400215.62 total 1480.29 last 59.58) - open/close 31993 (time 1785400225.63 total 1490.29 last 72.75) - open/close 32554 (time 1785400235.67 total 1500.33 last 55.89) - open/close 32974 (time 1785400245.70 total 1510.36 last 41.87) - open/close 33391 (time 1785400255.70 total 1520.36 last 41.69) - open/close 33802 (time 1785400265.72 total 1530.38 last 41.03) - open/close 34529 (time 1785400275.73 total 1540.39 last 72.64) - open/close 35001 (time 1785400285.73 total 1550.39 last 47.18) - open/close 35478 (time 1785400295.74 total 1560.41 last 47.64) - open/close 35905 (time 1785400305.80 total 1570.47 last 42.44) - open/close 36355 (time 1785400315.81 total 1580.47 last 44.97) - open/close 36874 (time 1785400325.81 total 1590.47 last 51.89) - open/close 37360 (time 1785400335.81 total 1600.48 last 48.59) - open/close 37765 (time 1785400345.82 total 1610.48 last 40.47) - open/close 38466 (time 1785400355.83 total 1620.49 last 70.04) - open/close 39291 (time 1785400365.83 total 1630.49 last 82.49) - open/close 39753 (time 1785400375.85 total 1640.51 last 46.12) - open/close 40000 (time 1785400380.71 total 1645.38 last 50.75) - open/close 40504 (time 1785400390.74 total 1655.40 last 50.28) - open/close 40970 (time 1785400400.76 total 1665.42 last 46.50) - open/close 41386 (time 1785400410.78 total 1675.44 last 41.53) - open/close 41802 (time 1785400420.78 total 1685.45 last 41.57) - open/close 42314 (time 1785400430.80 total 1695.46 last 51.12) - open/close 42888 (time 1785400440.81 total 1705.47 last 57.34) - open/close 43585 (time 1785400450.82 total 1715.48 last 69.62) - open/close 44249 (time 1785400460.84 total 1725.50 last 66.25) - open/close 44740 (time 1785400470.84 total 1735.51 last 49.09) - open/close 45202 (time 1785400480.84 total 1745.51 last 46.20) - open/close 45851 (time 1785400490.85 total 1755.51 last 64.87) - open/close 46665 (time 1785400500.85 total 1765.51 last 81.39) - open/close 47319 (time 1785400510.86 total 1775.52 last 65.35) - open/close 47816 (time 1785400520.86 total 1785.52 last 49.70) - open/close 48304 (time 1785400530.87 total 1795.53 last 48.73) - open/close 48815 (time 1785400540.88 total 1805.54 last 51.05) - open/close 49298 (time 1785400550.88 total 1815.55 last 48.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1823.37 seconds: 27.22 ops/second - unlinked 0 (time 1785400566 ; total 0 ; last 0) - unlinked 10000 (time 1785400755 ; total 189 ; last 189) - unlinked 20000 (time 1785400954 ; total 388 ; last 199) - unlinked 30000 (time 1785401129 ; total 563 ; last 175) - unlinked 40000 (time 1785401280 ; total 714 ; last 151) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 872 seconds: 56.916283 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server unloading modules via unload_modules_local on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing unload_modules_local oleg128-server: modules unloaded. pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2