== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:02:40 (1786723360) start mds service on oleg428-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg428-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 119 (time 1786723456.57 total 10.87 last 10.95) - open/close 203 (time 1786723467.11 total 21.41 last 7.97) - open/close 280 (time 1786723477.12 total 31.41 last 7.70) - open/close 384 (time 1786723489.02 total 43.32 last 8.74) - open/close 572 (time 1786723499.02 total 53.32 last 18.79) - open/close 786 (time 1786723510.60 total 64.90 last 18.48) - open/close 956 (time 1786723520.61 total 74.91 last 16.98) - open/close 1053 (time 1786723531.25 total 85.54 last 9.12) - open/close 1166 (time 1786723541.26 total 95.56 last 11.28) - open/close 1343 (time 1786723551.30 total 105.60 last 17.63) - open/close 1428 (time 1786723562.03 total 116.33 last 7.92) - open/close 1514 (time 1786723572.04 total 126.34 last 8.59) - open/close 1670 (time 1786723582.99 total 137.29 last 14.25) - open/close 1863 (time 1786723593.41 total 147.71 last 18.52) - open/close 1961 (time 1786723603.51 total 157.81 last 9.70) - open/close 2071 (time 1786723613.53 total 167.83 last 10.98) - open/close 2206 (time 1786723623.80 total 178.09 last 13.16) - open/close 2419 (time 1786723633.80 total 188.10 last 21.28) - open/close 2523 (time 1786723643.82 total 198.11 last 10.39) - open/close 2623 (time 1786723654.79 total 209.09 last 9.11) - open/close 2721 (time 1786723665.85 total 220.15 last 8.86) - open/close 2900 (time 1786723676.78 total 231.08 last 16.37) - open/close 3084 (time 1786723687.16 total 241.46 last 17.74) - open/close 3198 (time 1786723697.27 total 251.56 last 11.28) - open/close 3309 (time 1786723707.94 total 262.24 last 10.40) - open/close 3473 (time 1786723717.95 total 272.24 last 16.39) - open/close 3620 (time 1786723727.95 total 282.25 last 14.69) - open/close 3811 (time 1786723737.97 total 292.27 last 19.07) - open/close 3953 (time 1786723748.73 total 303.03 last 13.20) - open/close 4042 (time 1786723760.81 total 315.10 last 7.37) - open/close 4284 (time 1786723770.85 total 325.15 last 24.09) - open/close 4322 (time 1786723781.35 total 335.65 last 3.62) - open/close 4454 (time 1786723792.58 total 346.88 last 11.76) - open/close 4591 (time 1786723803.12 total 357.42 last 12.99) - open/close 4723 (time 1786723813.27 total 367.57 last 13.00) - open/close 4924 (time 1786723823.28 total 377.57 last 20.10) - open/close 5028 (time 1786723834.12 total 388.41 last 9.59) - open/close 5183 (time 1786723844.14 total 398.43 last 15.47) - open/close 5301 (time 1786723855.17 total 409.46 last 10.70) - open/close 5415 (time 1786723867.77 total 422.07 last 9.05) - open/close 5617 (time 1786723877.80 total 432.10 last 20.14) - open/close 5838 (time 1786723888.50 total 442.79 last 20.66) - open/close 6060 (time 1786723899.08 total 453.38 last 20.97) - open/close 6201 (time 1786723909.49 total 463.78 last 13.56) - open/close 6418 (time 1786723919.53 total 473.83 last 21.61) - open/close 6571 (time 1786723930.42 total 484.72 last 14.05) - open/close 6668 (time 1786723940.45 total 494.74 last 9.67) - open/close 6834 (time 1786723950.70 total 505.00 last 16.18) - open/close 6952 (time 1786723961.71 total 516.01 last 10.72) - open/close 7026 (time 1786723972.28 total 526.58 last 7.00) - open/close 7133 (time 1786723982.71 total 537.01 last 10.25) - open/close 7227 (time 1786723992.77 total 547.06 last 9.35) - open/close 7327 (time 1786724003.30 total 557.60 last 9.49) - open/close 7572 (time 1786724013.91 total 568.21 last 23.09) - open/close 7698 (time 1786724023.91 total 578.21 last 12.59) - open/close 7856 (time 1786724033.94 total 588.23 last 15.76) - open/close 7954 (time 1786724045.00 total 599.30 last 8.85) - open/close 8021 (time 1786724055.38 total 609.68 last 6.46) - open/close 8125 (time 1786724065.41 total 619.71 last 10.37) - open/close 8215 (time 1786724075.42 total 629.72 last 9.00) - open/close 8350 (time 1786724085.42 total 639.72 last 13.49) - open/close 8481 (time 1786724095.75 total 650.05 last 12.69) - open/close 8557 (time 1786724105.77 total 660.06 last 7.59) - open/close 8736 (time 1786724116.43 total 670.73 last 16.79) - open/close 8923 (time 1786724127.49 total 681.79 last 16.91) - open/close 9137 (time 1786724137.50 total 691.80 last 21.37) - open/close 9368 (time 1786724147.51 total 701.81 last 23.07) - open/close 9520 (time 1786724157.87 total 712.17 last 14.67) - open/close 9747 (time 1786724167.87 total 722.17 last 22.70) - open/close 9903 (time 1786724177.91 total 732.21 last 15.54) - open/close 10000 (time 1786724186.33 total 740.63 last 11.51) - open/close 10138 (time 1786724196.34 total 750.64 last 13.79) - open/close 10322 (time 1786724206.36 total 760.66 last 18.36) - open/close 10631 (time 1786724216.57 total 770.86 last 30.28) - open/close 10814 (time 1786724227.00 total 781.30 last 17.54) - open/close 11021 (time 1786724237.03 total 791.33 last 20.63) - open/close 11185 (time 1786724247.05 total 801.35 last 16.37) - open/close 11297 (time 1786724258.04 total 812.34 last 10.19) - open/close 11435 (time 1786724268.21 total 822.51 last 13.57) - open/close 11629 (time 1786724278.23 total 832.52 last 19.38) - open/close 11891 (time 1786724289.27 total 843.57 last 23.73) - open/close 12102 (time 1786724299.69 total 853.99 last 20.24) - open/close 12349 (time 1786724309.70 total 864.00 last 24.69) - open/close 12598 (time 1786724319.96 total 874.25 last 24.28) - open/close 12844 (time 1786724329.96 total 884.26 last 24.59) - open/close 13042 (time 1786724340.48 total 894.78 last 18.81) - open/close 13159 (time 1786724351.49 total 905.79 last 10.63) - open/close 13301 (time 1786724363.04 total 917.34 last 12.30) - open/close 13571 (time 1786724373.07 total 927.37 last 26.91) - open/close 13735 (time 1786724383.61 total 937.91 last 15.57) - open/close 13991 (time 1786724393.66 total 947.95 last 25.48) - open/close 14208 (time 1786724404.53 total 958.83 last 19.95) - open/close 14410 (time 1786724414.85 total 969.15 last 19.58) - open/close 14547 (time 1786724424.95 total 979.25 last 13.56) - open/close 14660 (time 1786724434.96 total 989.26 last 11.29) - open/close 14911 (time 1786724444.97 total 999.27 last 25.06) - open/close 15141 (time 1786724455.80 total 1010.09 last 21.25) - open/close 15357 (time 1786724465.82 total 1020.12 last 21.54) - open/close 15592 (time 1786724476.85 total 1031.15 last 21.32) - open/close 15730 (time 1786724487.58 total 1041.87 last 12.86) - open/close 15844 (time 1786724497.93 total 1052.22 last 11.02) - open/close 15949 (time 1786724507.93 total 1062.23 last 10.50) - open/close 16062 (time 1786724517.93 total 1072.23 last 11.29) - open/close 16146 (time 1786724527.94 total 1082.24 last 8.39) - open/close 16491 (time 1786724537.94 total 1092.24 last 34.49) - open/close 17088 (time 1786724547.95 total 1102.25 last 59.63) - open/close 17703 (time 1786724557.96 total 1112.26 last 61.46) - open/close 18382 (time 1786724567.97 total 1122.27 last 67.84) - open/close 18962 (time 1786724577.98 total 1132.27 last 57.97) - open/close 19568 (time 1786724587.98 total 1142.28 last 60.55) - open/close 20000 (time 1786724595.94 total 1150.24 last 54.27) - open/close 20463 (time 1786724605.96 total 1160.25 last 46.24) - open/close 21006 (time 1786724615.97 total 1170.26 last 54.24) - open/close 21575 (time 1786724625.97 total 1180.27 last 56.87) - open/close 22210 (time 1786724635.98 total 1190.28 last 63.47) - open/close 22754 (time 1786724645.99 total 1200.28 last 54.35) - open/close 23190 (time 1786724656.00 total 1210.29 last 43.56) - open/close 23568 (time 1786724666.02 total 1220.32 last 37.72) - open/close 24026 (time 1786724676.06 total 1230.35 last 45.63) - open/close 24595 (time 1786724686.06 total 1240.35 last 56.90) - open/close 25129 (time 1786724696.18 total 1250.48 last 52.75) - open/close 25495 (time 1786724706.23 total 1260.53 last 36.42) - open/close 25993 (time 1786724716.24 total 1270.53 last 49.77) - open/close 26576 (time 1786724726.25 total 1280.54 last 58.25) - open/close 27137 (time 1786724736.28 total 1290.58 last 55.91) - open/close 27729 (time 1786724746.29 total 1300.58 last 59.16) - open/close 28263 (time 1786724756.30 total 1310.60 last 53.31) - open/close 28805 (time 1786724766.31 total 1320.61 last 54.16) - open/close 29323 (time 1786724776.32 total 1330.62 last 51.73) - open/close 29894 (time 1786724786.33 total 1340.63 last 57.07) - open/close 30510 (time 1786724796.33 total 1350.63 last 61.59) - open/close 31107 (time 1786724806.34 total 1360.63 last 59.67) - open/close 31703 (time 1786724816.34 total 1370.64 last 59.58) - open/close 32252 (time 1786724826.35 total 1380.64 last 54.86) - open/close 32851 (time 1786724836.36 total 1390.66 last 59.83) - open/close 33459 (time 1786724846.36 total 1400.66 last 60.77) - open/close 33933 (time 1786724856.37 total 1410.67 last 47.36) - open/close 34449 (time 1786724866.37 total 1420.67 last 51.58) - open/close 35060 (time 1786724876.39 total 1430.69 last 61.01) - open/close 35654 (time 1786724886.39 total 1440.69 last 59.37) - open/close 36268 (time 1786724896.40 total 1450.69 last 61.40) - open/close 36935 (time 1786724906.41 total 1460.71 last 66.59) - open/close 37548 (time 1786724916.43 total 1470.73 last 61.20) - open/close 38129 (time 1786724926.43 total 1480.73 last 58.06) - open/close 38715 (time 1786724936.45 total 1490.75 last 58.52) - open/close 39355 (time 1786724946.45 total 1500.75 last 63.98) - open/close 40000 (time 1786724955.81 total 1510.11 last 68.93) - open/close 40526 (time 1786724965.81 total 1520.11 last 52.57) - open/close 40976 (time 1786724975.82 total 1530.12 last 44.96) - open/close 41502 (time 1786724985.83 total 1540.13 last 52.57) - open/close 42056 (time 1786724995.84 total 1550.14 last 55.35) - open/close 42563 (time 1786725005.85 total 1560.14 last 50.66) - open/close 43035 (time 1786725015.87 total 1570.17 last 47.09) - open/close 43497 (time 1786725025.88 total 1580.17 last 46.18) - open/close 43954 (time 1786725035.90 total 1590.20 last 45.57) - open/close 44516 (time 1786725045.91 total 1600.21 last 56.16) - open/close 45035 (time 1786725055.93 total 1610.23 last 51.79) - open/close 45509 (time 1786725065.93 total 1620.23 last 47.40) - open/close 46091 (time 1786725075.94 total 1630.24 last 58.16) - open/close 46630 (time 1786725085.94 total 1640.24 last 53.89) - open/close 47141 (time 1786725095.94 total 1650.24 last 51.09) - open/close 47665 (time 1786725105.96 total 1660.25 last 52.32) - open/close 48241 (time 1786725115.98 total 1670.27 last 57.48) - open/close 48789 (time 1786725125.98 total 1680.28 last 54.75) - open/close 49343 (time 1786725136.00 total 1690.30 last 55.30) - open/close 49827 (time 1786725146.01 total 1700.31 last 48.38) total: 50000 open/close in 1703.67 seconds: 29.35 ops/second - unlinked 0 (time 1786725155 ; total 0 ; last 0) - unlinked 10000 (time 1786725336 ; total 181 ; last 181) - unlinked 20000 (time 1786725519 ; total 364 ; last 183) - unlinked 30000 (time 1786725703 ; total 548 ; last 184) - unlinked 40000 (time 1786725879 ; total 724 ; last 176) total: 50000 unlinks in 910 seconds: 54.945053 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: modules unloaded. pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2