== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:05:13 (1785938713) start mds service on oleg237-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg237-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 118 (time 1785938855.55 total 13.04 last 9.05) - open/close 217 (time 1785938868.03 total 25.52 last 7.94) - open/close 341 (time 1785938878.85 total 36.34 last 11.46) - open/close 540 (time 1785938889.55 total 47.03 last 18.60) - open/close 671 (time 1785938900.12 total 57.61 last 12.39) - open/close 798 (time 1785938910.99 total 68.48 last 11.68) - open/close 954 (time 1785938920.99 total 78.48 last 15.60) - open/close 1176 (time 1785938931.78 total 89.27 last 20.58) - open/close 1301 (time 1785938941.79 total 99.28 last 12.49) - open/close 1422 (time 1785938951.79 total 109.28 last 12.10) - open/close 1572 (time 1785938962.18 total 119.67 last 14.44) - open/close 1777 (time 1785938973.15 total 130.63 last 18.69) - open/close 2000 (time 1785938983.98 total 141.46 last 20.59) - open/close 2253 (time 1785938994.68 total 152.17 last 23.63) - open/close 2318 (time 1785939004.70 total 162.18 last 6.49) - open/close 2418 (time 1785939015.68 total 173.17 last 9.11) - open/close 2602 (time 1785939027.64 total 185.13 last 15.38) - open/close 2811 (time 1785939038.26 total 195.75 last 19.68) - open/close 2971 (time 1785939049.01 total 206.50 last 14.88) - open/close 3128 (time 1785939059.52 total 217.01 last 14.94) - open/close 3296 (time 1785939069.53 total 227.02 last 16.79) - open/close 3429 (time 1785939079.54 total 237.03 last 13.28) - open/close 3609 (time 1785939090.39 total 247.87 last 16.60) - open/close 3824 (time 1785939101.07 total 258.55 last 20.13) - open/close 3910 (time 1785939111.29 total 268.77 last 8.42) - open/close 3980 (time 1785939121.31 total 278.79 last 6.99) - open/close 4142 (time 1785939131.54 total 289.03 last 15.82) - open/close 4288 (time 1785939142.28 total 299.76 last 13.60) - open/close 4431 (time 1785939152.29 total 309.78 last 14.28) - open/close 4547 (time 1785939162.29 total 319.78 last 11.60) - open/close 4699 (time 1785939172.63 total 330.12 last 14.70) - open/close 4870 (time 1785939183.04 total 340.52 last 16.43) - open/close 5019 (time 1785939193.55 total 351.04 last 14.17) - open/close 5187 (time 1785939203.67 total 361.16 last 16.61) - open/close 5345 (time 1785939213.83 total 371.32 last 15.54) - open/close 5516 (time 1785939224.78 total 382.27 last 15.62) - open/close 5596 (time 1785939235.77 total 393.26 last 7.28) - open/close 5670 (time 1785939245.80 total 403.29 last 7.38) - open/close 5783 (time 1785939255.81 total 413.30 last 11.29) - open/close 5939 (time 1785939266.82 total 424.31 last 14.17) - open/close 6045 (time 1785939276.83 total 434.32 last 10.59) - open/close 6094 (time 1785939287.49 total 444.98 last 4.60) - open/close 6148 (time 1785939298.20 total 455.69 last 5.04) - open/close 6255 (time 1785939308.20 total 465.69 last 10.69) - open/close 6429 (time 1785939319.03 total 476.52 last 16.07) - open/close 6640 (time 1785939329.76 total 487.25 last 19.66) - open/close 6827 (time 1785939339.77 total 497.26 last 18.69) - open/close 6970 (time 1785939349.77 total 507.26 last 14.29) - open/close 7044 (time 1785939359.88 total 517.37 last 7.32) - open/close 7206 (time 1785939370.62 total 528.11 last 15.08) - open/close 7336 (time 1785939380.65 total 538.14 last 12.96) - open/close 7488 (time 1785939391.14 total 548.63 last 14.49) - open/close 7635 (time 1785939401.22 total 558.71 last 14.57) - open/close 7805 (time 1785939411.36 total 568.85 last 16.77) - open/close 8003 (time 1785939422.23 total 579.72 last 18.22) - open/close 8117 (time 1785939432.23 total 589.72 last 11.39) - open/close 8228 (time 1785939443.28 total 600.77 last 10.04) - open/close 8368 (time 1785939453.56 total 611.05 last 13.62) - open/close 8562 (time 1785939463.69 total 621.18 last 19.15) - open/close 8641 (time 1785939473.92 total 631.41 last 7.72) - open/close 8767 (time 1785939484.48 total 641.97 last 11.93) - open/close 8854 (time 1785939494.80 total 652.29 last 8.43) - open/close 8981 (time 1785939504.82 total 662.31 last 12.68) - open/close 9119 (time 1785939514.82 total 672.31 last 13.80) - open/close 9254 (time 1785939525.44 total 682.93 last 12.71) - open/close 9305 (time 1785939536.07 total 693.56 last 4.80) - open/close 9383 (time 1785939548.96 total 706.45 last 6.05) - open/close 9537 (time 1785939559.61 total 717.10 last 14.45) - open/close 9619 (time 1785939570.77 total 728.26 last 7.35) - open/close 9728 (time 1785939581.39 total 738.87 last 10.27) - open/close 9821 (time 1785939592.07 total 749.56 last 8.70) - open/close 9952 (time 1785939602.87 total 760.36 last 12.14) - open/close 10000 (time 1785939606.34 total 763.83 last 13.83) - open/close 10171 (time 1785939616.45 total 773.94 last 16.91) - open/close 10268 (time 1785939627.55 total 785.04 last 8.73) - open/close 10391 (time 1785939637.56 total 795.04 last 12.30) - open/close 10485 (time 1785939647.59 total 805.08 last 9.36) - open/close 10707 (time 1785939657.62 total 815.11 last 22.14) - open/close 10851 (time 1785939668.29 total 825.78 last 13.49) - open/close 10950 (time 1785939678.47 total 835.96 last 9.73) - open/close 11024 (time 1785939690.39 total 847.88 last 6.21) - open/close 11185 (time 1785939701.42 total 858.91 last 14.59) - open/close 11334 (time 1785939711.77 total 869.26 last 14.39) - open/close 11401 (time 1785939722.71 total 880.20 last 6.13) - open/close 11544 (time 1785939733.26 total 890.74 last 13.56) - open/close 11680 (time 1785939743.31 total 900.80 last 13.53) - open/close 11905 (time 1785939753.33 total 910.82 last 22.45) - open/close 12109 (time 1785939764.01 total 921.50 last 19.10) - open/close 12184 (time 1785939774.01 total 931.50 last 7.50) - open/close 12293 (time 1785939784.21 total 941.70 last 10.69) - open/close 12451 (time 1785939794.53 total 952.02 last 15.32) - open/close 12575 (time 1785939805.56 total 963.04 last 11.25) - open/close 12765 (time 1785939816.00 total 973.48 last 18.20) - open/close 12846 (time 1785939826.14 total 983.63 last 7.98) - open/close 12987 (time 1785939836.41 total 993.89 last 13.74) - open/close 13133 (time 1785939846.42 total 1003.91 last 14.58) - open/close 13283 (time 1785939856.77 total 1014.25 last 14.50) - open/close 13445 (time 1785939866.94 total 1024.42 last 15.93) - open/close 13604 (time 1785939879.18 total 1036.67 last 12.98) - open/close 13707 (time 1785939891.15 total 1048.64 last 8.61) - open/close 13851 (time 1785939902.14 total 1059.63 last 13.10) - open/close 13964 (time 1785939912.15 total 1069.64 last 11.30) - open/close 14044 (time 1785939924.83 total 1082.32 last 6.31) - open/close 14203 (time 1785939934.92 total 1092.41 last 15.76) - open/close 14355 (time 1785939945.34 total 1102.83 last 14.59) - open/close 14541 (time 1785939955.86 total 1113.35 last 17.68) - open/close 14654 (time 1785939966.01 total 1123.50 last 11.13) - open/close 14784 (time 1785939976.72 total 1134.21 last 12.14) - open/close 14945 (time 1785939986.75 total 1144.24 last 16.05) - open/close 15098 (time 1785939996.75 total 1154.24 last 15.30) - open/close 15235 (time 1785940006.91 total 1164.39 last 13.49) - open/close 15301 (time 1785940017.86 total 1175.34 last 6.03) - open/close 15435 (time 1785940027.86 total 1185.35 last 13.40) - open/close 15564 (time 1785940040.03 total 1197.52 last 10.60) - open/close 15782 (time 1785940050.04 total 1207.52 last 21.79) - open/close 15879 (time 1785940061.05 total 1218.54 last 8.81) - open/close 16043 (time 1785940073.89 total 1231.37 last 12.77) - open/close 16258 (time 1785940085.19 total 1242.68 last 19.02) - open/close 16369 (time 1785940095.21 total 1252.69 last 11.09) - open/close 16584 (time 1785940106.01 total 1263.49 last 19.91) - open/close 16813 (time 1785940116.99 total 1274.48 last 20.84) - open/close 16959 (time 1785940127.00 total 1284.49 last 14.58) - open/close 17066 (time 1785940137.99 total 1295.48 last 9.74) - open/close 17423 (time 1785940147.99 total 1305.48 last 35.70) - open/close 17778 (time 1785940158.03 total 1315.52 last 35.35) - open/close 18418 (time 1785940168.05 total 1325.54 last 63.88) - open/close 19054 (time 1785940178.08 total 1335.57 last 63.40) - open/close 19808 (time 1785940188.09 total 1345.57 last 75.38) - open/close 20000 (time 1785940190.66 total 1348.15 last 74.63) - open/close 20690 (time 1785940200.68 total 1358.16 last 68.88) - open/close 21681 (time 1785940210.68 total 1368.17 last 99.04) - open/close 22230 (time 1785940220.68 total 1378.17 last 54.90) - open/close 23180 (time 1785940230.69 total 1388.18 last 94.95) - open/close 23749 (time 1785940240.69 total 1398.18 last 56.90) - open/close 24560 (time 1785940250.70 total 1408.19 last 81.01) - open/close 25210 (time 1785940260.71 total 1418.20 last 64.95) - open/close 25857 (time 1785940270.71 total 1428.20 last 64.69) - open/close 26550 (time 1785940280.71 total 1438.20 last 69.29) - open/close 27053 (time 1785940290.72 total 1448.21 last 50.26) - open/close 27678 (time 1785940300.72 total 1458.21 last 62.50) - open/close 28207 (time 1785940310.74 total 1468.22 last 52.80) - open/close 28870 (time 1785940320.75 total 1478.24 last 66.20) - open/close 29622 (time 1785940330.76 total 1488.24 last 75.17) - open/close 30000 (time 1785940337.17 total 1494.66 last 58.95) - open/close 30548 (time 1785940347.19 total 1504.68 last 54.67) - open/close 31143 (time 1785940357.21 total 1514.69 last 59.42) - open/close 31943 (time 1785940367.21 total 1524.69 last 80.00) - open/close 32515 (time 1785940377.22 total 1534.71 last 57.11) - open/close 33306 (time 1785940387.22 total 1544.71 last 79.09) - open/close 34002 (time 1785940397.24 total 1554.73 last 69.49) - open/close 34456 (time 1785940407.24 total 1564.73 last 45.38) - open/close 35158 (time 1785940417.27 total 1574.76 last 70.00) - open/close 35683 (time 1785940427.29 total 1584.77 last 52.43) - open/close 36637 (time 1785940437.29 total 1594.78 last 95.36) - open/close 37169 (time 1785940447.31 total 1604.80 last 53.10) - open/close 37711 (time 1785940457.32 total 1614.80 last 54.16) - open/close 38159 (time 1785940467.36 total 1624.85 last 44.61) - open/close 38525 (time 1785940477.36 total 1634.85 last 36.59) - open/close 38940 (time 1785940487.40 total 1644.89 last 41.33) - open/close 39451 (time 1785940497.42 total 1654.90 last 51.03) - open/close 39903 (time 1785940507.42 total 1664.91 last 45.19) - open/close 40378 (time 1785940517.44 total 1674.92 last 47.41) - open/close 40963 (time 1785940527.45 total 1684.93 last 58.44) - open/close 41540 (time 1785940537.47 total 1694.95 last 57.59) - open/close 42270 (time 1785940547.47 total 1704.96 last 72.96) - open/close 42795 (time 1785940557.49 total 1714.98 last 52.41) - open/close 43276 (time 1785940567.51 total 1724.99 last 48.01) - open/close 43965 (time 1785940577.53 total 1735.01 last 68.77) - open/close 44536 (time 1785940587.54 total 1745.02 last 57.04) - open/close 45060 (time 1785940597.54 total 1755.03 last 52.36) - open/close 45591 (time 1785940607.55 total 1765.04 last 53.06) - open/close 46143 (time 1785940617.55 total 1775.04 last 55.19) - open/close 46827 (time 1785940627.56 total 1785.04 last 68.37) - open/close 47429 (time 1785940637.57 total 1795.06 last 60.10) - open/close 48281 (time 1785940647.62 total 1805.10 last 84.83) - open/close 48885 (time 1785940657.64 total 1815.13 last 60.27) - open/close 49369 (time 1785940667.66 total 1825.15 last 48.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1829.15 seconds: 27.13 ops/second - unlinked 0 (time 1785940676 ; total 0 ; last 0) - unlinked 10000 (time 1785940827 ; total 151 ; last 151) - unlinked 20000 (time 1785941006 ; total 330 ; last 179) - unlinked 30000 (time 1785941165 ; total 489 ; last 159) - unlinked 40000 (time 1785941301 ; total 625 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 769 seconds: 64.539665 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2