== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:44:24 (1786952664) start mds service on oleg418-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg418-server: detected 4 online CPUs by sysfs oleg418-server: MODOPTS_LIBCFS= oleg418-server: Force libcfs to create 2 CPU partitions oleg418-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg418-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg418-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg418-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 107 (time 1786952796.42 total 11.05 last 9.68) - open/close 232 (time 1786952806.43 total 21.06 last 12.48) - open/close 392 (time 1786952818.97 total 33.60 last 12.76) - open/close 539 (time 1786952829.00 total 43.64 last 14.65) - open/close 852 (time 1786952839.00 total 53.64 last 31.30) - open/close 1126 (time 1786952849.30 total 63.93 last 26.60) - open/close 1359 (time 1786952860.26 total 74.89 last 21.27) - open/close 1613 (time 1786952870.28 total 84.91 last 25.34) - open/close 1774 (time 1786952880.84 total 95.47 last 15.24) - open/close 1972 (time 1786952891.09 total 105.72 last 19.33) - open/close 2205 (time 1786952902.10 total 116.74 last 21.15) - open/close 2521 (time 1786952912.92 total 127.55 last 29.22) - open/close 2754 (time 1786952922.96 total 137.59 last 23.21) - open/close 2979 (time 1786952933.22 total 147.85 last 21.94) - open/close 3132 (time 1786952943.22 total 157.85 last 15.30) - open/close 3430 (time 1786952953.24 total 167.87 last 29.72) - open/close 3686 (time 1786952964.23 total 178.86 last 23.30) - open/close 3922 (time 1786952974.34 total 188.97 last 23.34) - open/close 4164 (time 1786952984.36 total 198.99 last 24.16) - open/close 4332 (time 1786952995.03 total 209.66 last 15.75) - open/close 4458 (time 1786953005.03 total 219.66 last 12.59) - open/close 4612 (time 1786953015.70 total 230.33 last 14.43) - open/close 4849 (time 1786953025.72 total 240.35 last 23.67) - open/close 5015 (time 1786953035.73 total 250.36 last 16.58) - open/close 5105 (time 1786953046.37 total 261.00 last 8.45) - open/close 5211 (time 1786953056.38 total 271.01 last 10.60) - open/close 5340 (time 1786953066.38 total 281.01 last 12.89) - open/close 5547 (time 1786953077.42 total 292.05 last 18.76) - open/close 5696 (time 1786953087.50 total 302.13 last 14.78) - open/close 5836 (time 1786953097.52 total 312.15 last 13.97) - open/close 6091 (time 1786953107.55 total 322.18 last 25.43) - open/close 6286 (time 1786953117.90 total 332.53 last 18.83) - open/close 6490 (time 1786953127.92 total 342.55 last 20.37) - open/close 6602 (time 1786953138.83 total 353.46 last 10.27) - open/close 6883 (time 1786953148.85 total 363.48 last 28.03) - open/close 6989 (time 1786953161.50 total 376.13 last 8.38) - open/close 7218 (time 1786953171.52 total 386.15 last 22.85) - open/close 7353 (time 1786953181.99 total 396.63 last 12.89) - open/close 7456 (time 1786953192.40 total 407.03 last 9.90) - open/close 7645 (time 1786953202.41 total 417.04 last 18.89) - open/close 7880 (time 1786953213.28 total 427.91 last 21.63) - open/close 8031 (time 1786953224.66 total 439.29 last 13.26) - open/close 8252 (time 1786953235.10 total 449.73 last 21.17) - open/close 8486 (time 1786953245.79 total 460.42 last 21.89) - open/close 8731 (time 1786953256.72 total 471.35 last 22.42) - open/close 8892 (time 1786953266.74 total 481.37 last 16.07) - open/close 9051 (time 1786953277.45 total 492.09 last 14.84) - open/close 9174 (time 1786953288.14 total 502.77 last 11.51) - open/close 9345 (time 1786953299.17 total 513.80 last 15.51) - open/close 9432 (time 1786953309.39 total 524.02 last 8.52) - open/close 9627 (time 1786953319.41 total 534.04 last 19.46) - open/close 9863 (time 1786953329.41 total 544.04 last 23.59) - open/close 10000 (time 1786953334.80 total 549.43 last 25.43) - open/close 10178 (time 1786953344.91 total 559.54 last 17.60) - open/close 10350 (time 1786953354.91 total 569.54 last 17.19) - open/close 10458 (time 1786953365.45 total 580.08 last 10.24) - open/close 10655 (time 1786953376.39 total 591.02 last 18.01) - open/close 10851 (time 1786953386.40 total 601.04 last 19.58) - open/close 11035 (time 1786953396.42 total 611.05 last 18.38) - open/close 11162 (time 1786953406.62 total 621.25 last 12.45) - open/close 11252 (time 1786953416.63 total 631.26 last 8.99) - open/close 11410 (time 1786953426.92 total 641.55 last 15.35) - open/close 11481 (time 1786953436.92 total 651.55 last 7.10) - open/close 11553 (time 1786953446.94 total 661.57 last 7.19) - open/close 11726 (time 1786953456.99 total 671.62 last 17.21) - open/close 11884 (time 1786953467.42 total 682.05 last 15.15) - open/close 12077 (time 1786953477.66 total 692.29 last 18.86) - open/close 12144 (time 1786953487.77 total 702.40 last 6.63) - open/close 12275 (time 1786953498.39 total 713.02 last 12.33) - open/close 12500 (time 1786953508.70 total 723.33 last 21.82) - open/close 12615 (time 1786953519.64 total 734.27 last 10.51) - open/close 12714 (time 1786953530.39 total 745.02 last 9.21) - open/close 12843 (time 1786953540.41 total 755.04 last 12.87) - open/close 13079 (time 1786953550.41 total 765.04 last 23.59) - open/close 13226 (time 1786953560.44 total 775.07 last 14.66) - open/close 13369 (time 1786953570.45 total 785.08 last 14.29) - open/close 13467 (time 1786953580.46 total 795.09 last 9.79) - open/close 13602 (time 1786953590.46 total 805.09 last 13.49) - open/close 13705 (time 1786953601.13 total 815.76 last 9.66) - open/close 13817 (time 1786953611.83 total 826.46 last 10.47) - open/close 13996 (time 1786953623.76 total 838.39 last 15.00) - open/close 14140 (time 1786953634.87 total 849.50 last 12.97) - open/close 14248 (time 1786953645.90 total 860.53 last 9.79) - open/close 14507 (time 1786953655.92 total 870.55 last 25.85) - open/close 14739 (time 1786953665.93 total 880.56 last 23.16) - open/close 14992 (time 1786953675.95 total 890.58 last 25.26) - open/close 15045 (time 1786953686.49 total 901.12 last 5.03) - open/close 15295 (time 1786953697.43 total 912.06 last 22.86) - open/close 15538 (time 1786953708.05 total 922.68 last 22.87) - open/close 15789 (time 1786953718.07 total 932.70 last 25.06) - open/close 15970 (time 1786953728.08 total 942.71 last 18.08) - open/close 16153 (time 1786953738.22 total 952.85 last 18.05) - open/close 16296 (time 1786953749.11 total 963.74 last 13.13) - open/close 16373 (time 1786953759.96 total 974.59 last 7.10) - open/close 16472 (time 1786953770.07 total 984.70 last 9.80) - open/close 16609 (time 1786953780.88 total 995.51 last 12.67) - open/close 16811 (time 1786953790.89 total 1005.52 last 20.18) - open/close 16973 (time 1786953800.89 total 1015.52 last 16.19) - open/close 17052 (time 1786953810.91 total 1025.54 last 7.89) - open/close 17119 (time 1786953821.01 total 1035.64 last 6.63) - open/close 17209 (time 1786953831.06 total 1045.69 last 8.96) - open/close 17403 (time 1786953843.73 total 1058.36 last 15.31) - open/close 17480 (time 1786953854.23 total 1068.86 last 7.33) - open/close 17754 (time 1786953864.24 total 1078.87 last 27.38) - open/close 18348 (time 1786953874.24 total 1088.87 last 59.38) - open/close 18991 (time 1786953884.25 total 1098.88 last 64.23) - open/close 19663 (time 1786953894.26 total 1108.89 last 67.14) - open/close 20000 (time 1786953899.28 total 1113.91 last 67.21) - open/close 20679 (time 1786953909.28 total 1123.91 last 67.88) - open/close 21311 (time 1786953919.29 total 1133.92 last 63.15) - open/close 21863 (time 1786953929.29 total 1143.92 last 55.17) - open/close 22601 (time 1786953939.30 total 1153.93 last 73.76) - open/close 23266 (time 1786953949.30 total 1163.93 last 66.47) - open/close 23790 (time 1786953959.31 total 1173.94 last 52.37) - open/close 24347 (time 1786953969.32 total 1183.95 last 55.63) - open/close 25019 (time 1786953979.32 total 1193.96 last 67.18) - open/close 25481 (time 1786953989.33 total 1203.96 last 46.16) - open/close 25987 (time 1786953999.35 total 1213.98 last 50.51) - open/close 26476 (time 1786954009.35 total 1223.98 last 48.88) - open/close 27115 (time 1786954019.36 total 1233.99 last 63.88) - open/close 27703 (time 1786954029.38 total 1244.01 last 58.66) - open/close 28262 (time 1786954039.38 total 1254.01 last 55.90) - open/close 28823 (time 1786954049.38 total 1264.01 last 56.09) - open/close 29506 (time 1786954059.39 total 1274.02 last 68.27) - open/close 30000 (time 1786954065.78 total 1280.41 last 77.22) - open/close 30510 (time 1786954075.80 total 1290.43 last 50.91) - open/close 30995 (time 1786954085.81 total 1300.44 last 48.49) - open/close 31576 (time 1786954095.81 total 1310.44 last 58.05) - open/close 32173 (time 1786954105.82 total 1320.45 last 59.67) - open/close 32700 (time 1786954115.85 total 1330.48 last 52.54) - open/close 33259 (time 1786954125.85 total 1340.48 last 55.89) - open/close 33811 (time 1786954135.85 total 1350.48 last 55.20) - open/close 34431 (time 1786954145.86 total 1360.49 last 61.97) - open/close 35112 (time 1786954155.87 total 1370.50 last 68.02) - open/close 35675 (time 1786954165.88 total 1380.51 last 56.21) - open/close 36302 (time 1786954175.89 total 1390.52 last 62.66) - open/close 36911 (time 1786954185.91 total 1400.54 last 60.77) - open/close 37497 (time 1786954195.92 total 1410.55 last 58.53) - open/close 38007 (time 1786954205.94 total 1420.57 last 50.91) - open/close 38603 (time 1786954215.95 total 1430.58 last 59.53) - open/close 39152 (time 1786954225.98 total 1440.61 last 54.75) - open/close 39795 (time 1786954235.99 total 1450.62 last 64.22) - open/close 40000 (time 1786954239.94 total 1454.57 last 51.98) - open/close 40589 (time 1786954249.95 total 1464.59 last 58.79) - open/close 41155 (time 1786954259.97 total 1474.60 last 56.54) - open/close 41733 (time 1786954269.97 total 1484.60 last 57.76) - open/close 42406 (time 1786954279.98 total 1494.61 last 67.27) - open/close 42986 (time 1786954289.98 total 1504.61 last 57.98) - open/close 43565 (time 1786954299.98 total 1514.61 last 57.89) - open/close 44245 (time 1786954309.99 total 1524.62 last 67.96) - open/close 44889 (time 1786954319.99 total 1534.62 last 64.35) - open/close 45651 (time 1786954330.00 total 1544.63 last 76.14) - open/close 46404 (time 1786954340.01 total 1554.64 last 75.27) - open/close 46944 (time 1786954350.01 total 1564.64 last 53.98) - open/close 47634 (time 1786954360.01 total 1574.64 last 69.00) - open/close 48207 (time 1786954370.02 total 1584.65 last 57.28) - open/close 48777 (time 1786954380.02 total 1594.65 last 56.98) - open/close 49344 (time 1786954390.03 total 1604.66 last 56.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1609.89 seconds: 30.83 ops/second - unlinked 0 (time 1786954400 ; total 0 ; last 0) - unlinked 10000 (time 1786954562 ; total 162 ; last 162) - unlinked 20000 (time 1786954706 ; total 306 ; last 144) - unlinked 30000 (time 1786954850 ; total 450 ; last 144) - unlinked 40000 (time 1786955001 ; total 601 ; last 151) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 746 seconds: 66.529488 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules via unload_modules_local on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local oleg418-server: modules unloaded. pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2