== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:03:31 (1785780211) start mds service on oleg401-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg401-server: detected 4 online CPUs by sysfs oleg401-server: MODOPTS_LIBCFS= oleg401-server: Force libcfs to create 2 CPU partitions oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg401-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg401-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg401-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 108 (time 1785780347.93 total 10.02 last 10.78) - open/close 257 (time 1785780359.00 total 21.08 last 13.47) - open/close 397 (time 1785780369.20 total 31.28 last 13.72) - open/close 551 (time 1785780379.75 total 41.83 last 14.60) - open/close 666 (time 1785780390.05 total 52.13 last 11.16) - open/close 827 (time 1785780400.05 total 62.14 last 16.10) - open/close 971 (time 1785780410.52 total 72.60 last 13.76) - open/close 1178 (time 1785780420.53 total 82.61 last 20.68) - open/close 1361 (time 1785780430.54 total 92.62 last 18.28) - open/close 1529 (time 1785780441.12 total 103.20 last 15.88) - open/close 1720 (time 1785780451.94 total 114.02 last 17.66) - open/close 1846 (time 1785780462.40 total 124.48 last 12.04) - open/close 1995 (time 1785780472.40 total 134.48 last 14.90) - open/close 2149 (time 1785780482.41 total 144.50 last 15.38) - open/close 2224 (time 1785780492.54 total 154.62 last 7.41) - open/close 2370 (time 1785780502.74 total 164.83 last 14.31) - open/close 2481 (time 1785780513.71 total 175.79 last 10.12) - open/close 2579 (time 1785780523.77 total 185.85 last 9.74) - open/close 2692 (time 1785780534.69 total 196.77 last 10.35) - open/close 2895 (time 1785780545.23 total 207.32 last 19.25) - open/close 3051 (time 1785780555.24 total 217.32 last 15.59) - open/close 3215 (time 1785780565.29 total 227.38 last 16.31) - open/close 3387 (time 1785780575.31 total 237.40 last 17.16) - open/close 3508 (time 1785780585.33 total 247.42 last 12.08) - open/close 3702 (time 1785780595.34 total 257.42 last 19.39) - open/close 3779 (time 1785780605.49 total 267.57 last 7.59) - open/close 3940 (time 1785780615.50 total 277.58 last 16.08) - open/close 4091 (time 1785780625.65 total 287.74 last 14.87) - open/close 4248 (time 1785780635.83 total 297.91 last 15.43) - open/close 4363 (time 1785780645.85 total 307.93 last 11.48) - open/close 4440 (time 1785780655.88 total 317.97 last 7.67) - open/close 4581 (time 1785780666.48 total 328.56 last 13.31) - open/close 4739 (time 1785780677.41 total 339.50 last 14.45) - open/close 4835 (time 1785780688.12 total 350.20 last 8.97) - open/close 4943 (time 1785780699.18 total 361.26 last 9.77) - open/close 4997 (time 1785780709.79 total 371.88 last 5.09) - open/close 5063 (time 1785780720.67 total 382.76 last 6.07) - open/close 5128 (time 1785780731.43 total 393.51 last 6.04) - open/close 5407 (time 1785780741.44 total 403.52 last 27.86) - open/close 5533 (time 1785780751.53 total 413.61 last 12.49) - open/close 5697 (time 1785780761.55 total 423.64 last 16.35) - open/close 5820 (time 1785780772.10 total 434.18 last 11.67) - open/close 5978 (time 1785780782.10 total 444.18 last 15.79) - open/close 6124 (time 1785780792.11 total 454.19 last 14.59) - open/close 6306 (time 1785780802.53 total 464.61 last 17.46) - open/close 6464 (time 1785780812.55 total 474.63 last 15.77) - open/close 6594 (time 1785780823.28 total 485.36 last 12.12) - open/close 6722 (time 1785780833.29 total 495.37 last 12.79) - open/close 6841 (time 1785780843.30 total 505.38 last 11.88) - open/close 7047 (time 1785780854.25 total 516.33 last 18.82) - open/close 7232 (time 1785780864.25 total 526.33 last 18.49) - open/close 7430 (time 1785780874.28 total 536.37 last 19.74) - open/close 7503 (time 1785780884.46 total 546.55 last 7.17) - open/close 7644 (time 1785780894.47 total 556.55 last 14.09) - open/close 7765 (time 1785780904.47 total 566.56 last 12.10) - open/close 7907 (time 1785780914.48 total 576.57 last 14.19) - open/close 8055 (time 1785780924.51 total 586.59 last 14.76) - open/close 8173 (time 1785780935.35 total 597.43 last 10.89) - open/close 8252 (time 1785780945.35 total 607.44 last 7.90) - open/close 8323 (time 1785780955.44 total 617.52 last 7.04) - open/close 8424 (time 1785780965.45 total 627.53 last 10.09) - open/close 8564 (time 1785780976.11 total 638.20 last 13.13) - open/close 8697 (time 1785780986.30 total 648.38 last 13.06) - open/close 8849 (time 1785780996.61 total 658.69 last 14.74) - open/close 9001 (time 1785781006.78 total 668.86 last 14.95) - open/close 9214 (time 1785781016.80 total 678.88 last 21.25) - open/close 9331 (time 1785781026.80 total 688.89 last 11.69) - open/close 9457 (time 1785781037.17 total 699.26 last 12.15) - open/close 9633 (time 1785781047.18 total 709.27 last 17.58) - open/close 9779 (time 1785781058.03 total 720.11 last 13.46) - open/close 9856 (time 1785781068.24 total 730.32 last 7.54) - open/close 9975 (time 1785781079.04 total 741.12 last 11.02) - open/close 10070 (time 1785781089.74 total 751.82 last 8.88) - open/close 10217 (time 1785781099.76 total 761.85 last 14.66) - open/close 10370 (time 1785781111.67 total 773.76 last 12.85) - open/close 10566 (time 1785781122.75 total 784.83 last 17.69) - open/close 10636 (time 1785781132.99 total 795.07 last 6.84) - open/close 10795 (time 1785781143.97 total 806.06 last 14.48) - open/close 10972 (time 1785781154.17 total 816.25 last 17.36) - open/close 11099 (time 1785781164.59 total 826.68 last 12.18) - open/close 11230 (time 1785781174.62 total 836.71 last 13.06) - open/close 11339 (time 1785781184.63 total 846.71 last 10.89) - open/close 11513 (time 1785781197.03 total 859.12 last 14.03) - open/close 11640 (time 1785781207.60 total 869.68 last 12.02) - open/close 11808 (time 1785781217.64 total 879.72 last 16.74) - open/close 11962 (time 1785781227.81 total 889.89 last 15.14) - open/close 12098 (time 1785781237.82 total 899.90 last 13.59) - open/close 12213 (time 1785781247.84 total 909.92 last 11.48) - open/close 12325 (time 1785781257.87 total 919.96 last 11.16) - open/close 12466 (time 1785781267.87 total 929.96 last 14.10) - open/close 12564 (time 1785781278.65 total 940.73 last 9.10) - open/close 12725 (time 1785781289.40 total 951.48 last 14.97) - open/close 12804 (time 1785781300.00 total 962.09 last 7.45) - open/close 12972 (time 1785781310.25 total 972.33 last 16.39) - open/close 13103 (time 1785781320.27 total 982.35 last 13.08) - open/close 13280 (time 1785781331.00 total 993.09 last 16.48) - open/close 13400 (time 1785781341.01 total 1003.09 last 12.00) - open/close 13604 (time 1785781352.03 total 1014.12 last 18.50) - open/close 13748 (time 1785781362.08 total 1024.16 last 14.34) - open/close 13894 (time 1785781373.06 total 1035.14 last 13.30) - open/close 14004 (time 1785781384.05 total 1046.13 last 10.01) - open/close 14123 (time 1785781394.37 total 1056.45 last 11.53) - open/close 14329 (time 1785781405.06 total 1067.14 last 19.27) - open/close 14498 (time 1785781416.07 total 1078.15 last 15.35) - open/close 14657 (time 1785781426.41 total 1088.49 last 15.38) - open/close 14769 (time 1785781436.52 total 1098.60 last 11.08) - open/close 14884 (time 1785781446.64 total 1108.72 last 11.36) - open/close 15001 (time 1785781456.65 total 1118.73 last 11.69) - open/close 15107 (time 1785781466.66 total 1128.74 last 10.59) - open/close 15240 (time 1785781477.41 total 1139.50 last 12.37) - open/close 15327 (time 1785781487.90 total 1149.99 last 8.29) - open/close 15429 (time 1785781499.62 total 1161.70 last 8.71) - open/close 15543 (time 1785781509.99 total 1172.07 last 10.99) - open/close 15673 (time 1785781520.89 total 1182.98 last 11.92) - open/close 15808 (time 1785781530.97 total 1193.05 last 13.40) - open/close 15931 (time 1785781541.30 total 1203.38 last 11.91) - open/close 16030 (time 1785781551.91 total 1213.99 last 9.33) - open/close 16254 (time 1785781562.24 total 1224.32 last 21.69) - open/close 16304 (time 1785781573.11 total 1235.19 last 4.60) - open/close 16362 (time 1785781583.84 total 1245.93 last 5.40) - open/close 16500 (time 1785781593.86 total 1255.95 last 13.77) - open/close 16663 (time 1785781604.13 total 1266.21 last 15.89) - open/close 16782 (time 1785781614.13 total 1276.21 last 11.90) - open/close 16871 (time 1785781624.87 total 1286.96 last 8.28) - open/close 16990 (time 1785781636.65 total 1298.73 last 10.11) - open/close 17118 (time 1785781646.65 total 1308.74 last 12.79) - open/close 17294 (time 1785781656.66 total 1318.74 last 17.59) - open/close 17445 (time 1785781666.66 total 1328.74 last 15.09) - open/close 17586 (time 1785781676.68 total 1338.76 last 14.08) - open/close 17734 (time 1785781686.72 total 1348.80 last 14.74) - open/close 17907 (time 1785781697.39 total 1359.47 last 16.21) - open/close 18033 (time 1785781707.69 total 1369.77 last 12.23) - open/close 18404 (time 1785781717.71 total 1379.79 last 37.02) - open/close 18920 (time 1785781727.73 total 1389.81 last 51.53) - open/close 19516 (time 1785781737.73 total 1399.81 last 59.60) - open/close 20000 (time 1785781746.50 total 1408.58 last 55.15) - open/close 20468 (time 1785781756.51 total 1418.59 last 46.78) - open/close 20944 (time 1785781766.53 total 1428.62 last 47.47) - open/close 21313 (time 1785781776.54 total 1438.63 last 36.86) - open/close 21899 (time 1785781786.55 total 1448.64 last 58.54) - open/close 22638 (time 1785781796.58 total 1458.66 last 73.72) - open/close 23137 (time 1785781806.58 total 1468.66 last 49.90) - open/close 23952 (time 1785781816.58 total 1478.66 last 81.49) - open/close 24743 (time 1785781826.58 total 1488.67 last 79.06) - open/close 25694 (time 1785781836.61 total 1498.69 last 94.90) - open/close 26104 (time 1785781846.61 total 1508.70 last 40.97) - open/close 26650 (time 1785781856.62 total 1518.71 last 54.54) - open/close 27338 (time 1785781866.62 total 1528.71 last 68.80) - open/close 28179 (time 1785781876.63 total 1538.72 last 84.00) - open/close 28951 (time 1785781886.64 total 1548.72 last 77.19) - open/close 29437 (time 1785781896.64 total 1558.72 last 48.59) - open/close 30000 (time 1785781904.90 total 1566.98 last 68.13) - open/close 30677 (time 1785781914.91 total 1577.00 last 67.62) - open/close 31127 (time 1785781924.94 total 1587.02 last 44.88) - open/close 31710 (time 1785781934.95 total 1597.03 last 58.26) - open/close 32366 (time 1785781944.99 total 1607.08 last 65.29) - open/close 32994 (time 1785781955.01 total 1617.10 last 62.67) - open/close 33716 (time 1785781965.05 total 1627.13 last 71.96) - open/close 34302 (time 1785781975.06 total 1637.14 last 58.55) - open/close 34780 (time 1785781985.07 total 1647.15 last 47.74) - open/close 35202 (time 1785781995.08 total 1657.16 last 42.17) - open/close 35553 (time 1785782005.10 total 1667.19 last 35.01) - open/close 35930 (time 1785782015.12 total 1677.20 last 37.64) - open/close 36437 (time 1785782025.12 total 1687.20 last 50.69) - open/close 36983 (time 1785782035.12 total 1697.20 last 54.59) - open/close 37740 (time 1785782045.15 total 1707.23 last 75.51) - open/close 38442 (time 1785782055.18 total 1717.26 last 70.00) - open/close 38890 (time 1785782065.19 total 1727.28 last 44.72) - open/close 39518 (time 1785782075.21 total 1737.30 last 62.67) - open/close 40000 (time 1785782080.65 total 1742.74 last 88.58) - open/close 40767 (time 1785782090.66 total 1752.74 last 76.65) - open/close 41210 (time 1785782100.66 total 1762.75 last 44.28) - open/close 41655 (time 1785782110.70 total 1772.78 last 44.35) - open/close 42277 (time 1785782120.70 total 1782.78 last 62.18) - open/close 42729 (time 1785782130.73 total 1792.81 last 45.07) - open/close 43270 (time 1785782140.75 total 1802.83 last 53.99) - open/close 44126 (time 1785782150.76 total 1812.84 last 85.54) - open/close 44817 (time 1785782160.76 total 1822.84 last 69.09) - open/close 45406 (time 1785782170.78 total 1832.86 last 58.78) - open/close 46147 (time 1785782180.79 total 1842.87 last 74.00) - open/close 46806 (time 1785782190.79 total 1852.87 last 65.90) - open/close 47378 (time 1785782200.82 total 1862.91 last 57.02) - open/close 47815 (time 1785782210.82 total 1872.91 last 43.70) - open/close 48480 (time 1785782220.83 total 1882.92 last 66.44) - open/close 48927 (time 1785782230.85 total 1892.93 last 44.63) - open/close 49437 (time 1785782240.89 total 1902.97 last 50.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1907.93 seconds: 26.01 ops/second - unlinked 0 (time 1785782252 ; total 0 ; last 0) - unlinked 10000 (time 1785782404 ; total 152 ; last 152) - unlinked 20000 (time 1785782575 ; total 323 ; last 171) - unlinked 30000 (time 1785782732 ; total 480 ; last 157) - unlinked 40000 (time 1785782905 ; total 653 ; last 173) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 804 seconds: 61.730099 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg401-server unloading modules via unload_modules_local on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local oleg401-server: modules unloaded. pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2