== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:49:34 (1787784574) start mds service on oleg436-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions oleg436-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg436-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg436-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg436-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg436-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1787784702.70 total 11.08 last 22.21) - open/close 311 (time 1787784713.40 total 21.77 last 6.08) - open/close 539 (time 1787784725.93 total 34.31 last 18.19) - open/close 658 (time 1787784736.75 total 45.12 last 11.00) - open/close 853 (time 1787784746.75 total 55.13 last 19.49) - open/close 971 (time 1787784757.88 total 66.25 last 10.61) - open/close 1101 (time 1787784767.91 total 76.28 last 12.96) - open/close 1179 (time 1787784777.91 total 86.28 last 7.80) - open/close 1354 (time 1787784787.91 total 96.28 last 17.50) - open/close 1417 (time 1787784799.59 total 107.97 last 5.39) - open/close 1578 (time 1787784811.26 total 119.63 last 13.80) - open/close 1772 (time 1787784821.26 total 129.64 last 19.39) - open/close 1973 (time 1787784831.27 total 139.64 last 20.09) - open/close 2178 (time 1787784841.39 total 149.76 last 20.25) - open/close 2462 (time 1787784851.41 total 159.78 last 28.35) - open/close 2594 (time 1787784861.80 total 170.18 last 12.70) - open/close 2662 (time 1787784873.58 total 181.95 last 5.77) - open/close 2770 (time 1787784883.60 total 191.98 last 10.78) - open/close 2877 (time 1787784893.64 total 202.01 last 10.67) - open/close 3020 (time 1787784903.65 total 212.02 last 14.28) - open/close 3147 (time 1787784914.74 total 223.11 last 11.45) - open/close 3360 (time 1787784925.16 total 233.53 last 20.45) - open/close 3590 (time 1787784936.05 total 244.42 last 21.11) - open/close 3769 (time 1787784946.94 total 255.31 last 16.44) - open/close 3998 (time 1787784957.95 total 266.32 last 20.79) - open/close 4246 (time 1787784967.96 total 276.34 last 24.77) - open/close 4332 (time 1787784977.97 total 286.34 last 8.60) - open/close 4530 (time 1787784988.03 total 296.40 last 19.69) - open/close 4723 (time 1787784998.15 total 306.52 last 19.07) - open/close 4983 (time 1787785008.15 total 316.53 last 25.98) - open/close 5166 (time 1787785018.86 total 327.24 last 17.09) - open/close 5429 (time 1787785029.73 total 338.10 last 24.21) - open/close 5661 (time 1787785039.78 total 348.16 last 23.08) - open/close 5884 (time 1787785050.48 total 358.85 last 20.85) - open/close 6111 (time 1787785060.91 total 369.28 last 21.76) - open/close 6316 (time 1787785071.59 total 379.96 last 19.20) - open/close 6476 (time 1787785081.61 total 389.98 last 15.97) - open/close 6609 (time 1787785091.95 total 400.32 last 12.86) - open/close 6769 (time 1787785102.90 total 411.27 last 14.60) - open/close 6837 (time 1787785113.22 total 421.59 last 6.59) - open/close 6974 (time 1787785123.70 total 432.07 last 13.07) - open/close 7070 (time 1787785134.05 total 442.42 last 9.27) - open/close 7173 (time 1787785145.90 total 454.27 last 8.69) - open/close 7243 (time 1787785156.85 total 465.22 last 6.40) - open/close 7537 (time 1787785166.85 total 475.22 last 29.39) - open/close 7733 (time 1787785177.03 total 485.40 last 19.26) - open/close 7869 (time 1787785187.18 total 495.55 last 13.40) - open/close 8057 (time 1787785197.88 total 506.25 last 17.57) - open/close 8287 (time 1787785207.89 total 516.26 last 22.98) - open/close 8411 (time 1787785218.91 total 527.29 last 11.24) - open/close 8598 (time 1787785228.97 total 537.34 last 18.59) - open/close 8824 (time 1787785239.73 total 548.10 last 21.01) - open/close 9082 (time 1787785249.78 total 558.15 last 25.67) - open/close 9140 (time 1787785260.77 total 569.14 last 5.28) - open/close 9362 (time 1787785270.80 total 579.17 last 22.14) - open/close 9526 (time 1787785280.82 total 589.19 last 16.36) - open/close 9676 (time 1787785291.51 total 599.89 last 14.03) - open/close 9812 (time 1787785301.95 total 610.32 last 13.03) - open/close 9929 (time 1787785312.44 total 620.81 last 11.15) - open/close 10000 (time 1787785314.62 total 623.00 last 32.51) - open/close 10208 (time 1787785324.66 total 633.03 last 20.73) - open/close 10282 (time 1787785335.22 total 643.60 last 7.00) - open/close 10492 (time 1787785345.39 total 653.76 last 20.66) - open/close 10641 (time 1787785356.99 total 665.36 last 12.85) - open/close 10762 (time 1787785367.29 total 675.66 last 11.75) - open/close 10883 (time 1787785378.00 total 686.37 last 11.29) - open/close 10995 (time 1787785388.09 total 696.46 last 11.10) - open/close 11193 (time 1787785398.10 total 706.48 last 19.77) - open/close 11365 (time 1787785408.19 total 716.56 last 17.06) - open/close 11548 (time 1787785419.88 total 728.25 last 15.65) - open/close 11787 (time 1787785429.89 total 738.26 last 23.88) - open/close 11897 (time 1787785439.92 total 748.29 last 10.97) - open/close 11970 (time 1787785449.94 total 758.31 last 7.28) - open/close 12154 (time 1787785460.40 total 768.77 last 17.59) - open/close 12315 (time 1787785470.42 total 778.79 last 16.07) - open/close 12373 (time 1787785481.01 total 789.38 last 5.48) - open/close 12562 (time 1787785491.02 total 799.40 last 18.87) - open/close 12690 (time 1787785501.04 total 809.41 last 12.78) - open/close 12803 (time 1787785512.10 total 820.47 last 10.22) - open/close 12885 (time 1787785523.37 total 831.74 last 7.28) - open/close 12989 (time 1787785536.07 total 844.44 last 8.19) - open/close 13071 (time 1787785547.63 total 856.00 last 7.09) - open/close 13156 (time 1787785557.80 total 866.17 last 8.35) - open/close 13281 (time 1787785568.83 total 877.20 last 11.34) - open/close 13425 (time 1787785580.85 total 889.22 last 11.98) - open/close 13624 (time 1787785590.86 total 899.23 last 19.88) - open/close 13828 (time 1787785600.87 total 909.24 last 20.39) - open/close 14077 (time 1787785611.52 total 919.90 last 23.36) - open/close 14260 (time 1787785621.53 total 929.90 last 18.29) - open/close 14319 (time 1787785631.97 total 940.35 last 5.65) - open/close 14435 (time 1787785642.68 total 951.06 last 10.83) - open/close 14546 (time 1787785653.11 total 961.48 last 10.64) - open/close 14750 (time 1787785663.33 total 971.70 last 19.97) - open/close 14925 (time 1787785673.65 total 982.02 last 16.96) - open/close 14989 (time 1787785683.71 total 992.08 last 6.36) - open/close 15062 (time 1787785693.93 total 1002.31 last 7.14) - open/close 15156 (time 1787785704.19 total 1012.57 last 9.16) - open/close 15196 (time 1787785714.55 total 1022.92 last 3.86) - open/close 15334 (time 1787785724.98 total 1033.35 last 13.23) - open/close 15475 (time 1787785735.97 total 1044.34 last 12.83) - open/close 15688 (time 1787785748.52 total 1056.89 last 16.97) - open/close 15791 (time 1787785759.20 total 1067.57 last 9.65) - open/close 16079 (time 1787785769.20 total 1077.57 last 28.79) - open/close 16714 (time 1787785779.20 total 1087.57 last 63.50) - open/close 17356 (time 1787785789.20 total 1097.58 last 64.20) - open/close 18081 (time 1787785799.23 total 1107.61 last 72.27) - open/close 18775 (time 1787785809.24 total 1117.62 last 69.33) - open/close 19443 (time 1787785819.25 total 1127.62 last 66.79) - open/close 20000 (time 1787785828.45 total 1136.82 last 60.54) - open/close 20665 (time 1787785838.45 total 1146.82 last 66.49) - open/close 21376 (time 1787785848.45 total 1156.82 last 71.08) - open/close 22125 (time 1787785858.46 total 1166.84 last 74.81) - open/close 22785 (time 1787785868.47 total 1176.84 last 65.95) - open/close 23437 (time 1787785878.48 total 1186.85 last 65.16) - open/close 24047 (time 1787785888.49 total 1196.86 last 60.94) - open/close 24762 (time 1787785898.51 total 1206.88 last 71.36) - open/close 25479 (time 1787785908.51 total 1216.88 last 71.68) - open/close 26189 (time 1787785918.52 total 1226.90 last 70.90) - open/close 26896 (time 1787785928.52 total 1236.90 last 70.70) - open/close 27578 (time 1787785938.53 total 1246.90 last 68.15) - open/close 28265 (time 1787785948.54 total 1256.91 last 68.64) - open/close 28974 (time 1787785958.55 total 1266.93 last 70.79) - open/close 29720 (time 1787785968.56 total 1276.94 last 74.53) - open/close 30000 (time 1787785972.31 total 1280.69 last 74.65) - open/close 30717 (time 1787785982.35 total 1290.72 last 71.44) - open/close 31441 (time 1787785992.35 total 1300.72 last 72.40) - open/close 32117 (time 1787786002.36 total 1310.73 last 67.55) - open/close 32868 (time 1787786012.36 total 1320.74 last 75.07) - open/close 33548 (time 1787786022.36 total 1330.74 last 68.00) - open/close 34248 (time 1787786032.37 total 1340.74 last 69.95) - open/close 34988 (time 1787786042.37 total 1350.75 last 73.99) - open/close 35742 (time 1787786052.38 total 1360.76 last 75.32) - open/close 36553 (time 1787786062.40 total 1370.77 last 81.00) - open/close 37309 (time 1787786072.40 total 1380.77 last 75.58) - open/close 38075 (time 1787786082.40 total 1390.77 last 76.59) - open/close 38821 (time 1787786092.41 total 1400.79 last 74.52) - open/close 39475 (time 1787786102.42 total 1410.79 last 65.36) - open/close 40000 (time 1787786109.61 total 1417.98 last 73.04) - open/close 40718 (time 1787786119.62 total 1427.99 last 71.73) - open/close 41396 (time 1787786129.62 total 1437.99 last 67.76) - open/close 42041 (time 1787786139.63 total 1448.00 last 64.45) - open/close 42742 (time 1787786149.63 total 1458.00 last 70.10) - open/close 43453 (time 1787786159.64 total 1468.01 last 71.06) - open/close 44177 (time 1787786169.64 total 1478.01 last 72.37) - open/close 44845 (time 1787786179.67 total 1488.05 last 66.57) - open/close 45557 (time 1787786189.69 total 1498.06 last 71.10) - open/close 46168 (time 1787786199.71 total 1508.08 last 60.98) - open/close 46861 (time 1787786209.71 total 1518.08 last 69.30) - open/close 47469 (time 1787786219.71 total 1528.08 last 60.80) - open/close 48001 (time 1787786229.71 total 1538.08 last 53.20) - open/close 48460 (time 1787786239.72 total 1548.09 last 45.85) - open/close 49100 (time 1787786249.73 total 1558.10 last 63.96) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1566.40 seconds: 31.68 ops/second - unlinked 0 (time 1787786262 ; total 0 ; last 0) - unlinked 10000 (time 1787786403 ; total 141 ; last 141) - unlinked 20000 (time 1787786544 ; total 282 ; last 141) - unlinked 30000 (time 1787786688 ; total 426 ; last 144) - unlinked 40000 (time 1787786829 ; total 567 ; last 141) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 710 seconds: 69.902817 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg436-server unloading modules via unload_modules_local on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing unload_modules_local oleg436-server: modules unloaded. pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2