== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:06:02 (1786633562) start mds service on oleg124-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg124-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg124-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1786633680.29 total 10.80 last 12.13) - open/close 348 (time 1786633690.30 total 20.81 last 21.68) - open/close 628 (time 1786633700.30 total 30.81 last 27.99) - open/close 855 (time 1786633711.28 total 41.78 last 20.69) - open/close 1081 (time 1786633723.77 total 54.27 last 18.10) - open/close 1334 (time 1786633734.82 total 65.32 last 22.89) - open/close 1471 (time 1786633744.84 total 75.35 last 13.66) - open/close 1688 (time 1786633754.86 total 85.36 last 21.67) - open/close 1898 (time 1786633767.91 total 98.42 last 16.08) - open/close 2114 (time 1786633777.92 total 108.42 last 21.59) - open/close 2390 (time 1786633787.92 total 118.43 last 27.59) - open/close 2643 (time 1786633800.82 total 131.33 last 19.61) - open/close 2902 (time 1786633810.83 total 141.33 last 25.89) - open/close 3114 (time 1786633820.83 total 151.34 last 21.19) - open/close 3352 (time 1786633831.66 total 162.17 last 21.98) - open/close 3476 (time 1786633841.70 total 172.21 last 12.35) - open/close 3751 (time 1786633852.53 total 183.04 last 25.39) - open/close 3995 (time 1786633863.14 total 193.65 last 23.00) - open/close 4277 (time 1786633873.17 total 203.68 last 28.12) - open/close 4541 (time 1786633883.39 total 213.90 last 25.83) - open/close 4756 (time 1786633893.40 total 223.90 last 21.49) - open/close 5028 (time 1786633903.42 total 233.92 last 27.14) - open/close 5193 (time 1786633913.43 total 243.93 last 16.48) - open/close 5374 (time 1786633923.43 total 253.94 last 18.10) - open/close 5532 (time 1786633933.43 total 263.94 last 15.80) - open/close 5771 (time 1786633943.79 total 274.30 last 23.06) - open/close 5993 (time 1786633954.10 total 284.61 last 21.54) - open/close 6108 (time 1786633964.80 total 295.30 last 10.75) - open/close 6362 (time 1786633975.48 total 305.98 last 23.78) - open/close 6605 (time 1786633986.49 total 317.00 last 22.05) - open/close 6820 (time 1786633996.50 total 327.01 last 21.49) - open/close 6918 (time 1786634006.71 total 337.22 last 9.60) - open/close 7121 (time 1786634016.72 total 347.22 last 20.28) - open/close 7197 (time 1786634026.76 total 357.27 last 7.57) - open/close 7251 (time 1786634037.15 total 367.66 last 5.20) - open/close 7499 (time 1786634047.77 total 378.28 last 23.35) - open/close 7672 (time 1786634058.43 total 388.94 last 16.23) - open/close 7933 (time 1786634068.46 total 398.97 last 26.02) - open/close 8025 (time 1786634078.48 total 408.99 last 9.18) - open/close 8242 (time 1786634089.52 total 420.03 last 19.66) - open/close 8331 (time 1786634099.80 total 430.31 last 8.66) - open/close 8478 (time 1786634109.80 total 440.31 last 14.69) - open/close 8606 (time 1786634119.81 total 450.32 last 12.80) - open/close 8852 (time 1786634129.82 total 460.33 last 24.57) - open/close 9042 (time 1786634139.83 total 470.34 last 18.98) - open/close 9255 (time 1786634149.84 total 480.34 last 21.29) - open/close 9506 (time 1786634159.94 total 490.44 last 24.85) - open/close 9776 (time 1786634169.96 total 500.46 last 26.95) - open/close 9871 (time 1786634179.96 total 510.47 last 9.49) - open/close 10000 (time 1786634184.91 total 515.42 last 26.08) - open/close 10220 (time 1786634195.64 total 526.14 last 20.51) - open/close 10443 (time 1786634205.66 total 536.16 last 22.25) - open/close 10643 (time 1786634215.69 total 546.20 last 19.94) - open/close 10772 (time 1786634226.42 total 556.92 last 12.03) - open/close 10993 (time 1786634236.42 total 566.93 last 22.09) - open/close 11229 (time 1786634246.86 total 577.37 last 22.61) - open/close 11451 (time 1786634257.15 total 587.66 last 21.57) - open/close 11607 (time 1786634267.70 total 598.21 last 14.79) - open/close 11872 (time 1786634278.03 total 608.53 last 25.67) - open/close 12070 (time 1786634288.16 total 618.67 last 19.53) - open/close 12291 (time 1786634298.26 total 628.77 last 21.88) - open/close 12542 (time 1786634309.33 total 639.83 last 22.69) - open/close 12702 (time 1786634322.22 total 652.73 last 12.41) - open/close 12950 (time 1786634332.99 total 663.49 last 23.04) - open/close 13172 (time 1786634343.46 total 673.97 last 21.19) - open/close 13285 (time 1786634353.59 total 684.10 last 11.16) - open/close 13490 (time 1786634363.60 total 694.11 last 20.49) - open/close 13686 (time 1786634373.65 total 704.16 last 19.49) - open/close 13953 (time 1786634383.65 total 714.16 last 26.70) - open/close 14020 (time 1786634395.34 total 725.85 last 5.73) - open/close 14186 (time 1786634405.49 total 736.00 last 16.36) - open/close 14440 (time 1786634415.72 total 746.23 last 24.82) - open/close 14711 (time 1786634426.60 total 757.10 last 24.92) - open/close 14989 (time 1786634436.63 total 767.14 last 27.71) - open/close 15201 (time 1786634446.64 total 777.15 last 21.18) - open/close 15465 (time 1786634456.93 total 787.44 last 25.64) - open/close 15644 (time 1786634467.31 total 797.82 last 17.25) - open/close 15891 (time 1786634478.20 total 808.71 last 22.69) - open/close 16073 (time 1786634489.19 total 819.70 last 16.55) - open/close 16313 (time 1786634500.16 total 830.67 last 21.88) - open/close 16415 (time 1786634510.19 total 840.70 last 10.17) - open/close 16564 (time 1786634520.21 total 850.72 last 14.87) - open/close 16678 (time 1786634531.27 total 861.77 last 10.31) - open/close 16895 (time 1786634541.80 total 872.31 last 20.60) - open/close 17171 (time 1786634551.81 total 882.32 last 27.58) - open/close 17322 (time 1786634561.81 total 892.32 last 15.09) - open/close 17517 (time 1786634572.16 total 902.67 last 18.84) - open/close 17732 (time 1786634582.53 total 913.04 last 20.74) - open/close 17798 (time 1786634593.47 total 923.97 last 6.03) - open/close 18036 (time 1786634604.47 total 934.97 last 21.64) - open/close 18248 (time 1786634614.90 total 945.40 last 20.33) - open/close 18330 (time 1786634625.21 total 955.71 last 7.95) - open/close 18491 (time 1786634637.69 total 968.19 last 12.90) - open/close 18667 (time 1786634647.71 total 978.21 last 17.57) - open/close 18871 (time 1786634657.71 total 988.22 last 20.39) - open/close 19082 (time 1786634668.67 total 999.18 last 19.25) - open/close 19350 (time 1786634679.58 total 1010.08 last 24.58) - open/close 20000 (time 1786634688.44 total 1018.95 last 73.33) - open/close 20823 (time 1786634698.45 total 1028.96 last 82.22) - open/close 21780 (time 1786634708.45 total 1038.96 last 95.67) - open/close 22606 (time 1786634718.46 total 1048.96 last 82.57) - open/close 23426 (time 1786634728.48 total 1058.99 last 81.80) - open/close 24321 (time 1786634738.50 total 1069.01 last 89.31) - open/close 25222 (time 1786634748.51 total 1079.02 last 90.01) - open/close 26257 (time 1786634758.52 total 1089.02 last 103.48) - open/close 27265 (time 1786634768.52 total 1099.02 last 100.79) - open/close 28123 (time 1786634778.52 total 1109.03 last 85.75) - open/close 29095 (time 1786634788.53 total 1119.03 last 97.17) - open/close 29879 (time 1786634798.53 total 1129.04 last 78.37) - open/close 30585 (time 1786634808.54 total 1139.04 last 70.54) - open/close 31230 (time 1786634818.55 total 1149.05 last 64.44) - open/close 31938 (time 1786634828.56 total 1159.06 last 70.73) - open/close 32780 (time 1786634838.56 total 1169.06 last 84.19) - open/close 33623 (time 1786634848.57 total 1179.08 last 84.16) - open/close 34467 (time 1786634858.58 total 1189.09 last 84.35) - open/close 35216 (time 1786634868.59 total 1199.09 last 74.86) - open/close 35918 (time 1786634878.59 total 1209.09 last 70.19) - open/close 36811 (time 1786634888.60 total 1219.10 last 89.21) - open/close 37760 (time 1786634898.60 total 1229.11 last 94.86) - open/close 38614 (time 1786634908.60 total 1239.11 last 85.39) - open/close 39489 (time 1786634918.61 total 1249.12 last 87.41) - open/close 40000 (time 1786634924.57 total 1255.08 last 85.80) - open/close 40789 (time 1786634934.58 total 1265.09 last 78.83) - open/close 41754 (time 1786634944.59 total 1275.10 last 96.35) - open/close 42625 (time 1786634954.60 total 1285.11 last 87.03) - open/close 43480 (time 1786634964.61 total 1295.11 last 85.47) - open/close 44393 (time 1786634974.61 total 1305.12 last 91.27) - open/close 45207 (time 1786634984.61 total 1315.12 last 81.36) - open/close 46213 (time 1786634994.62 total 1325.13 last 100.56) - open/close 47059 (time 1786635004.62 total 1335.13 last 84.56) - open/close 47810 (time 1786635014.63 total 1345.14 last 75.05) - open/close 48608 (time 1786635024.64 total 1355.14 last 79.75) - open/close 49481 (time 1786635034.64 total 1365.15 last 87.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1366.75 seconds: 36.31 ops/second - unlinked 0 (time 1786635040 ; total 0 ; last 0) - unlinked 10000 (time 1786635161 ; total 121 ; last 121) - unlinked 20000 (time 1786635277 ; total 237 ; last 116) - unlinked 30000 (time 1786635342 ; total 302 ; last 65) - unlinked 40000 (time 1786635407 ; total 367 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 425 seconds: 116.778824 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: modules unloaded. pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2