== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:48:46 (1782222526) start mds service on oleg358-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg358-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 122 (time 1782222635.58 total 10.46 last 11.67) - open/close 393 (time 1782222646.46 total 21.33 last 24.92) - open/close 650 (time 1782222656.46 total 31.33 last 25.70) - open/close 903 (time 1782222667.51 total 42.38 last 22.90) - open/close 1144 (time 1782222677.63 total 52.51 last 23.80) - open/close 1357 (time 1782222690.07 total 64.95 last 17.13) - open/close 1497 (time 1782222700.47 total 75.34 last 13.47) - open/close 1677 (time 1782222710.47 total 85.35 last 17.99) - open/close 1867 (time 1782222720.49 total 95.37 last 18.97) - open/close 2033 (time 1782222730.92 total 105.80 last 15.91) - open/close 2286 (time 1782222741.62 total 116.50 last 23.65) - open/close 2449 (time 1782222751.93 total 126.80 last 15.82) - open/close 2744 (time 1782222762.54 total 137.42 last 27.80) - open/close 2990 (time 1782222772.54 total 147.42 last 24.59) - open/close 3209 (time 1782222783.08 total 157.96 last 20.78) - open/close 3430 (time 1782222795.84 total 170.72 last 17.32) - open/close 3646 (time 1782222806.31 total 181.19 last 20.63) - open/close 3899 (time 1782222816.31 total 191.19 last 25.29) - open/close 4108 (time 1782222826.32 total 201.20 last 20.89) - open/close 4362 (time 1782222836.32 total 211.20 last 25.40) - open/close 4595 (time 1782222846.88 total 221.76 last 22.06) - open/close 4783 (time 1782222856.89 total 231.76 last 18.80) - open/close 4941 (time 1782222868.48 total 243.35 last 13.63) - open/close 5063 (time 1782222879.48 total 254.35 last 11.09) - open/close 5295 (time 1782222890.40 total 265.28 last 21.23) - open/close 5543 (time 1782222900.41 total 275.28 last 24.79) - open/close 5777 (time 1782222913.08 total 287.96 last 18.46) - open/close 6037 (time 1782222924.01 total 298.89 last 23.79) - open/close 6275 (time 1782222934.38 total 309.26 last 22.96) - open/close 6455 (time 1782222944.41 total 319.29 last 17.95) - open/close 6642 (time 1782222955.04 total 329.92 last 17.59) - open/close 6879 (time 1782222965.04 total 339.92 last 23.70) - open/close 7104 (time 1782222975.27 total 350.15 last 21.99) - open/close 7352 (time 1782222985.28 total 360.16 last 24.78) - open/close 7478 (time 1782222995.30 total 370.17 last 12.58) - open/close 7642 (time 1782223005.55 total 380.43 last 15.99) - open/close 7912 (time 1782223015.57 total 390.45 last 26.94) - open/close 8133 (time 1782223026.01 total 400.88 last 21.19) - open/close 8314 (time 1782223036.67 total 411.54 last 16.98) - open/close 8511 (time 1782223046.67 total 421.55 last 19.69) - open/close 8693 (time 1782223057.26 total 432.14 last 17.18) - open/close 8902 (time 1782223067.29 total 442.16 last 20.86) - open/close 9099 (time 1782223077.30 total 452.17 last 19.68) - open/close 9294 (time 1782223087.30 total 462.17 last 19.50) - open/close 9540 (time 1782223098.34 total 473.22 last 22.27) - open/close 9774 (time 1782223110.07 total 484.95 last 19.96) - open/close 10000 (time 1782223118.13 total 493.01 last 28.03) - open/close 10102 (time 1782223129.72 total 504.60 last 8.80) - open/close 10280 (time 1782223139.72 total 514.60 last 17.80) - open/close 10407 (time 1782223149.81 total 524.69 last 12.59) - open/close 10672 (time 1782223160.27 total 535.15 last 25.34) - open/close 10834 (time 1782223170.29 total 545.16 last 16.17) - open/close 10947 (time 1782223180.29 total 555.17 last 11.29) - open/close 11056 (time 1782223191.09 total 565.96 last 10.10) - open/close 11311 (time 1782223201.09 total 575.97 last 25.49) - open/close 11546 (time 1782223211.12 total 586.00 last 23.42) - open/close 11768 (time 1782223221.13 total 596.01 last 22.18) - open/close 11994 (time 1782223231.80 total 606.68 last 21.18) - open/close 12238 (time 1782223241.82 total 616.70 last 24.35) - open/close 12511 (time 1782223251.95 total 626.82 last 26.97) - open/close 12709 (time 1782223261.99 total 636.87 last 19.71) - open/close 12867 (time 1782223272.23 total 647.10 last 15.44) - open/close 13008 (time 1782223282.36 total 657.24 last 13.91) - open/close 13137 (time 1782223292.59 total 667.47 last 12.61) - open/close 13272 (time 1782223302.98 total 677.86 last 12.99) - open/close 13458 (time 1782223313.80 total 688.68 last 17.19) - open/close 13527 (time 1782223324.86 total 699.73 last 6.24) - open/close 13742 (time 1782223335.46 total 710.34 last 20.27) - open/close 13950 (time 1782223346.22 total 721.10 last 19.33) - open/close 14161 (time 1782223356.23 total 731.11 last 21.08) - open/close 14359 (time 1782223366.44 total 741.32 last 19.39) - open/close 14599 (time 1782223376.46 total 751.34 last 23.96) - open/close 14878 (time 1782223386.85 total 761.73 last 26.85) - open/close 15070 (time 1782223396.86 total 771.73 last 19.19) - open/close 15277 (time 1782223407.74 total 782.62 last 19.02) - open/close 15373 (time 1782223418.73 total 793.61 last 8.74) - open/close 15527 (time 1782223428.74 total 803.61 last 15.39) - open/close 15812 (time 1782223441.42 total 816.29 last 22.48) - open/close 16037 (time 1782223451.89 total 826.77 last 21.48) - open/close 16246 (time 1782223462.00 total 836.88 last 20.68) - open/close 16438 (time 1782223472.01 total 846.88 last 19.18) - open/close 16654 (time 1782223482.01 total 856.89 last 21.59) - open/close 16908 (time 1782223492.79 total 867.66 last 23.57) - open/close 17103 (time 1782223503.22 total 878.09 last 18.70) - open/close 17266 (time 1782223513.78 total 888.66 last 15.43) - open/close 17534 (time 1782223524.77 total 899.65 last 24.38) - open/close 17798 (time 1782223535.17 total 910.05 last 25.38) - open/close 18020 (time 1782223545.17 total 920.05 last 22.20) - open/close 18277 (time 1782223556.27 total 931.14 last 23.17) - open/close 18448 (time 1782223567.09 total 941.97 last 15.80) - open/close 18646 (time 1782223577.20 total 952.08 last 19.58) - open/close 18766 (time 1782223588.05 total 962.92 last 11.06) - open/close 18924 (time 1782223598.60 total 973.47 last 14.98) - open/close 19190 (time 1782223608.63 total 983.50 last 26.52) - open/close 19448 (time 1782223618.63 total 993.51 last 25.78) - open/close 20000 (time 1782223625.94 total 1000.82 last 75.52) - open/close 20758 (time 1782223635.95 total 1010.83 last 75.74) - open/close 21524 (time 1782223645.96 total 1020.83 last 76.56) - open/close 22180 (time 1782223655.96 total 1030.84 last 65.58) - open/close 22908 (time 1782223665.99 total 1040.86 last 72.62) - open/close 23568 (time 1782223676.00 total 1050.88 last 65.91) - open/close 24354 (time 1782223686.01 total 1060.88 last 78.53) - open/close 25120 (time 1782223696.01 total 1070.89 last 76.56) - open/close 25783 (time 1782223706.03 total 1080.91 last 66.16) - open/close 26429 (time 1782223716.05 total 1090.93 last 64.49) - open/close 27118 (time 1782223726.05 total 1100.93 last 68.89) - open/close 27840 (time 1782223736.06 total 1110.94 last 72.15) - open/close 28612 (time 1782223746.07 total 1120.94 last 77.15) - open/close 29371 (time 1782223756.07 total 1130.95 last 75.84) - open/close 30000 (time 1782223765.34 total 1140.22 last 67.88) - open/close 30710 (time 1782223775.34 total 1150.22 last 70.98) - open/close 31381 (time 1782223785.34 total 1160.22 last 67.09) - open/close 32110 (time 1782223795.36 total 1170.24 last 72.77) - open/close 32805 (time 1782223805.37 total 1180.24 last 69.46) - open/close 33526 (time 1782223815.38 total 1190.26 last 71.99) - open/close 34262 (time 1782223825.39 total 1200.27 last 73.55) - open/close 34989 (time 1782223835.40 total 1210.28 last 72.62) - open/close 35716 (time 1782223845.40 total 1220.28 last 72.67) - open/close 36473 (time 1782223855.41 total 1230.29 last 75.65) - open/close 37215 (time 1782223865.41 total 1240.29 last 74.18) - open/close 37927 (time 1782223875.43 total 1250.30 last 71.11) - open/close 38675 (time 1782223885.43 total 1260.31 last 74.78) - open/close 39411 (time 1782223895.44 total 1270.32 last 73.50) - open/close 40000 (time 1782223904.14 total 1279.02 last 67.73) - open/close 40707 (time 1782223914.15 total 1289.02 last 70.63) - open/close 41419 (time 1782223924.16 total 1299.04 last 71.13) - open/close 42172 (time 1782223934.16 total 1309.04 last 75.26) - open/close 42834 (time 1782223944.17 total 1319.05 last 66.16) - open/close 43461 (time 1782223954.18 total 1329.05 last 62.65) - open/close 44116 (time 1782223964.19 total 1339.07 last 65.42) - open/close 44631 (time 1782223974.21 total 1349.09 last 51.40) - open/close 45135 (time 1782223984.22 total 1359.10 last 50.34) - open/close 45840 (time 1782223994.23 total 1369.10 last 70.46) - open/close 46615 (time 1782224004.23 total 1379.11 last 77.47) - open/close 47390 (time 1782224014.23 total 1389.11 last 77.49) - open/close 48083 (time 1782224024.24 total 1399.12 last 69.23) - open/close 48910 (time 1782224034.24 total 1409.12 last 82.69) - open/close 49554 (time 1782224044.25 total 1419.13 last 64.35) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1420.03 seconds: 34.95 ops/second - unlinked 0 (time 1782224049 ; total 0 ; last 0) - unlinked 10000 (time 1782224190 ; total 141 ; last 141) - unlinked 20000 (time 1782224332 ; total 283 ; last 142) - unlinked 30000 (time 1782224482 ; total 433 ; last 150) - unlinked 40000 (time 1782224626 ; total 577 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 708 seconds: 70.100281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded. pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2