== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:25:40 (1775219140) start mds service on oleg651-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg651-server' oleg651-server: oleg651-server.virtnet: executing load_modules_local oleg651-server: Loading modules from /home/green/git/lustre-release/lustre oleg651-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg651-server: detected 4 online CPUs by sysfs oleg651-server: MODOPTS_LIBCFS= oleg651-server: Force libcfs to create 2 CPU partitions oleg651-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg651-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg651-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg651-server: oleg651-server.virtnet: executing set_default_debug -1 all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg651-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug -1 all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 116 (time 1775219225.95 total 10.02 last 11.58) - open/close 225 (time 1775219235.97 total 20.05 last 10.87) - open/close 340 (time 1775219247.49 total 31.56 last 9.99) - open/close 407 (time 1775219258.02 total 42.10 last 6.36) - open/close 527 (time 1775219268.67 total 52.74 last 11.27) - open/close 694 (time 1775219278.96 total 63.03 last 16.23) - open/close 826 (time 1775219289.84 total 73.91 last 12.13) - open/close 1000 (time 1775219300.28 total 84.36 last 16.66) - open/close 1060 (time 1775219310.52 total 94.60 last 5.86) - open/close 1146 (time 1775219321.43 total 105.51 last 7.89) - open/close 1363 (time 1775219331.44 total 115.52 last 21.68) - open/close 1545 (time 1775219341.46 total 125.54 last 18.16) - open/close 1768 (time 1775219352.32 total 136.40 last 20.53) - open/close 2014 (time 1775219362.33 total 146.40 last 24.59) - open/close 2231 (time 1775219372.34 total 156.42 last 21.67) - open/close 2320 (time 1775219382.64 total 166.72 last 8.64) - open/close 2502 (time 1775219393.57 total 177.64 last 16.66) - open/close 2683 (time 1775219403.94 total 188.01 last 17.46) - open/close 2837 (time 1775219414.64 total 198.72 last 14.38) - open/close 3034 (time 1775219424.69 total 208.76 last 19.61) - open/close 3184 (time 1775219435.38 total 219.45 last 14.03) - open/close 3266 (time 1775219446.13 total 230.21 last 7.63) - open/close 3481 (time 1775219456.62 total 240.69 last 20.51) - open/close 3670 (time 1775219467.05 total 251.12 last 18.11) - open/close 3823 (time 1775219479.21 total 263.28 last 12.58) - open/close 4027 (time 1775219489.21 total 273.28 last 20.40) - open/close 4225 (time 1775219499.21 total 283.29 last 19.79) - open/close 4415 (time 1775219510.12 total 294.20 last 17.42) - open/close 4679 (time 1775219521.02 total 305.09 last 24.23) - open/close 4928 (time 1775219531.11 total 315.19 last 24.65) - open/close 5185 (time 1775219541.81 total 325.89 last 24.02) - open/close 5395 (time 1775219551.83 total 335.91 last 20.96) - open/close 5622 (time 1775219561.84 total 345.92 last 22.68) - open/close 5898 (time 1775219572.33 total 356.40 last 26.33) - open/close 6114 (time 1775219583.10 total 367.17 last 20.05) - open/close 6349 (time 1775219594.09 total 378.17 last 21.38) - open/close 6444 (time 1775219606.02 total 390.10 last 7.96) - open/close 6515 (time 1775219616.03 total 400.10 last 7.10) - open/close 6800 (time 1775219627.06 total 411.14 last 25.83) - open/close 7053 (time 1775219637.07 total 421.14 last 25.29) - open/close 7256 (time 1775219647.07 total 431.15 last 20.29) - open/close 7458 (time 1775219657.08 total 441.15 last 20.19) - open/close 7668 (time 1775219667.08 total 451.16 last 20.99) - open/close 7940 (time 1775219678.05 total 462.13 last 24.79) - open/close 8214 (time 1775219688.81 total 472.88 last 25.48) - open/close 8505 (time 1775219698.81 total 482.89 last 29.10) - open/close 8732 (time 1775219708.82 total 492.89 last 22.69) - open/close 8811 (time 1775219718.82 total 502.89 last 7.90) - open/close 9076 (time 1775219728.83 total 512.90 last 26.47) - open/close 9287 (time 1775219739.07 total 523.15 last 20.59) - open/close 9561 (time 1775219750.12 total 534.19 last 24.81) - open/close 9820 (time 1775219761.07 total 545.15 last 23.64) - open/close 10000 (time 1775219769.77 total 553.84 last 20.71) - open/close 10203 (time 1775219779.77 total 563.85 last 20.29) - open/close 10418 (time 1775219789.92 total 574.00 last 21.18) - open/close 10654 (time 1775219800.05 total 584.12 last 23.31) - open/close 10852 (time 1775219811.24 total 595.32 last 17.68) - open/close 10996 (time 1775219822.05 total 606.13 last 13.32) - open/close 11111 (time 1775219833.65 total 617.72 last 9.92) - open/close 11179 (time 1775219843.80 total 627.87 last 6.70) - open/close 11356 (time 1775219853.80 total 637.87 last 17.70) - open/close 11464 (time 1775219866.02 total 650.09 last 8.83) - open/close 11653 (time 1775219876.97 total 661.04 last 17.27) - open/close 11832 (time 1775219887.48 total 671.55 last 17.03) - open/close 11934 (time 1775219897.48 total 681.55 last 10.20) - open/close 12083 (time 1775219907.48 total 691.55 last 14.90) - open/close 12256 (time 1775219917.48 total 701.56 last 17.30) - open/close 12417 (time 1775219928.43 total 712.50 last 14.71) - open/close 12606 (time 1775219938.43 total 722.50 last 18.90) - open/close 12746 (time 1775219948.84 total 732.92 last 13.45) - open/close 12905 (time 1775219959.41 total 743.48 last 15.04) - open/close 13039 (time 1775219969.44 total 753.52 last 13.36) - open/close 13260 (time 1775219979.44 total 763.52 last 22.09) - open/close 13373 (time 1775219989.46 total 773.53 last 11.28) - open/close 13382 (time 1775220000.94 total 785.02 last 0.78) - open/close 13622 (time 1775220011.06 total 795.13 last 23.74) - open/close 13830 (time 1775220023.92 total 807.99 last 16.17) - open/close 14024 (time 1775220034.55 total 818.62 last 18.25) - open/close 14271 (time 1775220044.64 total 828.72 last 24.47) - open/close 14519 (time 1775220055.24 total 839.32 last 23.40) - open/close 14745 (time 1775220065.25 total 849.32 last 22.59) - open/close 14959 (time 1775220075.26 total 859.33 last 21.38) - open/close 15139 (time 1775220085.47 total 869.55 last 17.62) - open/close 15331 (time 1775220096.04 total 880.11 last 18.18) - open/close 15585 (time 1775220107.11 total 891.18 last 22.94) - open/close 15795 (time 1775220117.16 total 901.23 last 20.90) - open/close 16002 (time 1775220127.40 total 911.47 last 20.21) - open/close 16213 (time 1775220138.07 total 922.15 last 19.77) - open/close 16412 (time 1775220148.84 total 932.91 last 18.49) - open/close 16636 (time 1775220159.14 total 943.22 last 21.73) - open/close 16853 (time 1775220170.68 total 954.76 last 18.80) - open/close 17525 (time 1775220180.70 total 964.77 last 67.10) - open/close 18231 (time 1775220190.70 total 974.77 last 70.59) - open/close 18888 (time 1775220200.70 total 984.78 last 65.69) - open/close 19525 (time 1775220210.72 total 994.79 last 63.59) - open/close 20000 (time 1775220219.38 total 1003.45 last 54.86) - open/close 20617 (time 1775220229.40 total 1013.47 last 61.57) - open/close 21233 (time 1775220239.41 total 1023.49 last 61.51) - open/close 21897 (time 1775220249.43 total 1033.51 last 66.26) - open/close 22611 (time 1775220259.43 total 1043.51 last 71.39) - open/close 23319 (time 1775220269.45 total 1053.52 last 70.70) - open/close 24013 (time 1775220279.45 total 1063.53 last 69.36) - open/close 24732 (time 1775220289.47 total 1073.54 last 71.80) - open/close 25449 (time 1775220299.47 total 1083.55 last 71.66) - open/close 26222 (time 1775220309.49 total 1093.57 last 77.15) - open/close 26870 (time 1775220319.51 total 1103.58 last 64.70) - open/close 27486 (time 1775220329.51 total 1113.59 last 61.58) - open/close 28102 (time 1775220339.53 total 1123.61 last 61.47) - open/close 28726 (time 1775220349.54 total 1133.61 last 62.39) - open/close 29457 (time 1775220359.54 total 1143.61 last 73.10) - open/close 30000 (time 1775220367.21 total 1151.28 last 70.81) - open/close 30638 (time 1775220377.22 total 1161.29 last 63.73) - open/close 31263 (time 1775220387.23 total 1171.30 last 62.44) - open/close 31870 (time 1775220397.23 total 1181.30 last 60.70) - open/close 32624 (time 1775220407.25 total 1191.32 last 75.25) - open/close 33386 (time 1775220417.25 total 1201.33 last 76.13) - open/close 34107 (time 1775220427.26 total 1211.34 last 72.03) - open/close 34872 (time 1775220437.28 total 1221.35 last 76.41) - open/close 35526 (time 1775220447.28 total 1231.35 last 65.38) - open/close 36241 (time 1775220457.29 total 1241.36 last 71.44) - open/close 36944 (time 1775220467.29 total 1251.37 last 70.28) - open/close 37702 (time 1775220477.29 total 1261.37 last 75.78) - open/close 38405 (time 1775220487.31 total 1271.38 last 70.21) - open/close 39123 (time 1775220497.31 total 1281.38 last 71.79) - open/close 39942 (time 1775220507.31 total 1291.38 last 81.88) - open/close 40682 (time 1775220517.31 total 1301.39 last 73.97) - open/close 41455 (time 1775220527.32 total 1311.39 last 77.29) - open/close 42211 (time 1775220537.32 total 1321.40 last 75.54) - open/close 42989 (time 1775220547.33 total 1331.41 last 77.73) - open/close 43702 (time 1775220557.34 total 1341.41 last 71.26) - open/close 44473 (time 1775220567.34 total 1351.42 last 77.08) - open/close 45273 (time 1775220577.35 total 1361.42 last 79.94) - open/close 46100 (time 1775220587.36 total 1371.43 last 82.61) - open/close 46801 (time 1775220597.36 total 1381.44 last 70.07) - open/close 47530 (time 1775220607.38 total 1391.45 last 72.81) - open/close 48263 (time 1775220617.39 total 1401.46 last 73.21) - open/close 48919 (time 1775220627.39 total 1411.46 last 65.60) - open/close 49645 (time 1775220637.39 total 1421.47 last 72.56) total: 50000 open/close in 1426.33 seconds: 35.06 ops/second - unlinked 0 (time 1775220645 ; total 0 ; last 0) - unlinked 10000 (time 1775220773 ; total 128 ; last 128) - unlinked 20000 (time 1775220882 ; total 237 ; last 109) - unlinked 30000 (time 1775220975 ; total 330 ; last 93) - unlinked 40000 (time 1775221029 ; total 384 ; last 54) total: 50000 unlinks in 432 seconds: 115.740738 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg651-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg651-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg651-server stop mds service on oleg651-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg651-server unloading modules via unload_modules_local on: 'oleg651-server' oleg651-server: oleg651-server.virtnet: executing unload_modules_local oleg651-server: modules unloaded. pdsh@oleg651-client: oleg651-client: ssh exited with exit code 2 pdsh@oleg651-client: oleg651-server: ssh exited with exit code 2 pdsh@oleg651-client: oleg651-server: ssh exited with exit code 2 pdsh@oleg651-client: oleg651-client: ssh exited with exit code 2