== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:17:46 (1779977866) start mds service on oleg450-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg450-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 175 (time 1779977953.84 total 10.98 last 15.94) - open/close 354 (time 1779977963.95 total 21.09 last 17.71) - open/close 581 (time 1779977973.96 total 31.10 last 22.68) - open/close 780 (time 1779977984.51 total 41.64 last 18.87) - open/close 845 (time 1779977994.57 total 51.70 last 6.46) - open/close 1130 (time 1779978004.59 total 61.73 last 28.44) - open/close 1222 (time 1779978015.10 total 72.24 last 8.75) - open/close 1497 (time 1779978025.10 total 82.24 last 27.49) - open/close 1613 (time 1779978035.10 total 92.24 last 11.60) - open/close 1795 (time 1779978046.08 total 103.21 last 16.59) - open/close 1977 (time 1779978056.18 total 113.32 last 18.01) - open/close 2208 (time 1779978066.18 total 123.32 last 23.10) - open/close 2354 (time 1779978076.36 total 133.49 last 14.35) - open/close 2414 (time 1779978087.29 total 144.42 last 5.49) - open/close 2521 (time 1779978097.78 total 154.91 last 10.20) - open/close 2727 (time 1779978108.68 total 165.82 last 18.90) - open/close 2796 (time 1779978118.78 total 175.91 last 6.83) - open/close 2962 (time 1779978129.85 total 186.99 last 14.99) - open/close 3134 (time 1779978139.87 total 197.01 last 17.16) - open/close 3288 (time 1779978149.88 total 207.02 last 15.39) - open/close 3390 (time 1779978159.90 total 217.03 last 10.18) - open/close 3484 (time 1779978170.86 total 227.99 last 8.58) - open/close 3599 (time 1779978181.82 total 238.96 last 10.49) - open/close 3764 (time 1779978192.74 total 249.88 last 15.11) - open/close 3828 (time 1779978202.99 total 260.13 last 6.24) - open/close 3923 (time 1779978213.95 total 271.09 last 8.67) - open/close 4070 (time 1779978225.01 total 282.15 last 13.29) - open/close 4207 (time 1779978237.30 total 294.44 last 11.15) - open/close 4373 (time 1779978247.31 total 304.44 last 16.59) - open/close 4575 (time 1779978257.34 total 314.48 last 20.13) - open/close 4672 (time 1779978268.17 total 325.31 last 8.96) - open/close 4795 (time 1779978278.39 total 335.52 last 12.04) - open/close 4953 (time 1779978288.40 total 345.54 last 15.78) - open/close 5080 (time 1779978298.41 total 355.55 last 12.69) - open/close 5242 (time 1779978309.38 total 366.52 last 14.76) - open/close 5406 (time 1779978319.44 total 376.57 last 16.31) - open/close 5586 (time 1779978330.40 total 387.54 last 16.41) - open/close 5765 (time 1779978342.52 total 399.66 last 14.78) - open/close 5931 (time 1779978353.58 total 410.72 last 15.00) - open/close 6003 (time 1779978363.61 total 420.74 last 7.18) - open/close 6119 (time 1779978376.40 total 433.54 last 9.07) - open/close 6226 (time 1779978386.61 total 443.75 last 10.48) - open/close 6343 (time 1779978396.94 total 454.08 last 11.33) - open/close 6453 (time 1779978407.71 total 464.85 last 10.21) - open/close 6572 (time 1779978417.72 total 474.85 last 11.90) - open/close 6723 (time 1779978428.36 total 485.50 last 14.19) - open/close 6960 (time 1779978438.65 total 495.78 last 23.04) - open/close 7186 (time 1779978449.26 total 506.39 last 21.30) - open/close 7384 (time 1779978459.31 total 516.45 last 19.69) - open/close 7589 (time 1779978469.31 total 526.45 last 20.50) - open/close 7823 (time 1779978480.11 total 537.25 last 21.67) - open/close 7858 (time 1779978490.37 total 547.51 last 3.41) - open/close 8025 (time 1779978500.39 total 557.52 last 16.68) - open/close 8186 (time 1779978510.43 total 567.57 last 16.03) - open/close 8288 (time 1779978520.43 total 577.57 last 10.20) - open/close 8384 (time 1779978530.44 total 587.57 last 9.59) - open/close 8554 (time 1779978541.04 total 598.17 last 16.04) - open/close 8627 (time 1779978551.47 total 608.61 last 6.99) - open/close 8758 (time 1779978561.48 total 618.62 last 13.09) - open/close 8837 (time 1779978572.21 total 629.35 last 7.36) - open/close 8971 (time 1779978582.25 total 639.39 last 13.35) - open/close 9066 (time 1779978592.32 total 649.45 last 9.44) - open/close 9198 (time 1779978602.95 total 660.08 last 12.42) - open/close 9464 (time 1779978613.87 total 671.01 last 24.35) - open/close 9620 (time 1779978624.65 total 681.78 last 14.48) - open/close 9724 (time 1779978635.70 total 692.83 last 9.41) - open/close 9845 (time 1779978646.14 total 703.27 last 11.59) - open/close 10000 (time 1779978652.83 total 709.96 last 23.17) - open/close 10130 (time 1779978664.01 total 721.15 last 11.62) - open/close 10269 (time 1779978675.26 total 732.40 last 12.36) - open/close 10378 (time 1779978687.24 total 744.37 last 9.10) - open/close 10497 (time 1779978697.24 total 754.38 last 11.89) - open/close 10708 (time 1779978707.51 total 764.65 last 20.54) - open/close 10887 (time 1779978717.53 total 774.67 last 17.87) - open/close 11029 (time 1779978727.92 total 785.06 last 13.66) - open/close 11222 (time 1779978738.88 total 796.01 last 17.62) - open/close 11408 (time 1779978749.56 total 806.69 last 17.42) - open/close 11563 (time 1779978759.56 total 816.70 last 15.49) - open/close 11673 (time 1779978769.85 total 826.98 last 10.70) - open/close 11762 (time 1779978779.85 total 836.99 last 8.90) - open/close 11952 (time 1779978790.39 total 847.53 last 18.02) - open/close 12187 (time 1779978801.00 total 858.14 last 22.15) - open/close 12425 (time 1779978811.82 total 868.95 last 22.01) - open/close 12601 (time 1779978821.83 total 878.97 last 17.57) - open/close 12771 (time 1779978831.83 total 888.97 last 17.00) - open/close 12908 (time 1779978843.88 total 901.02 last 11.37) - open/close 13033 (time 1779978853.90 total 911.03 last 12.48) - open/close 13141 (time 1779978863.93 total 921.07 last 10.76) - open/close 13184 (time 1779978874.51 total 931.65 last 4.07) - open/close 13296 (time 1779978884.80 total 941.94 last 10.88) - open/close 13373 (time 1779978895.29 total 952.43 last 7.34) - open/close 13506 (time 1779978905.93 total 963.07 last 12.50) - open/close 13650 (time 1779978916.54 total 973.68 last 13.58) - open/close 13846 (time 1779978927.22 total 984.36 last 18.35) - open/close 14001 (time 1779978937.23 total 994.37 last 15.48) - open/close 14172 (time 1779978948.23 total 1005.37 last 15.54) - open/close 14341 (time 1779978959.10 total 1016.23 last 15.56) - open/close 14412 (time 1779978969.26 total 1026.40 last 6.98) - open/close 14612 (time 1779978982.01 total 1039.14 last 15.69) - open/close 14770 (time 1779978993.03 total 1050.17 last 14.33) - open/close 14916 (time 1779979003.47 total 1060.61 last 13.98) - open/close 15152 (time 1779979016.38 total 1073.52 last 18.28) - open/close 15362 (time 1779979027.26 total 1084.40 last 19.30) - open/close 15632 (time 1779979037.29 total 1094.42 last 26.94) - open/close 16230 (time 1779979047.29 total 1104.42 last 59.79) - open/close 16797 (time 1779979057.30 total 1114.44 last 56.60) - open/close 17381 (time 1779979067.33 total 1124.46 last 58.28) - open/close 18023 (time 1779979077.33 total 1134.47 last 64.18) - open/close 18641 (time 1779979087.33 total 1144.47 last 61.78) - open/close 19271 (time 1779979097.34 total 1154.48 last 62.92) - open/close 19880 (time 1779979107.37 total 1164.50 last 60.77) - open/close 20495 (time 1779979117.37 total 1174.51 last 61.48) - open/close 21077 (time 1779979127.38 total 1184.51 last 58.16) - open/close 21679 (time 1779979137.38 total 1194.52 last 60.16) - open/close 22321 (time 1779979147.39 total 1204.53 last 64.15) - open/close 22944 (time 1779979157.40 total 1214.53 last 62.25) - open/close 23587 (time 1779979167.41 total 1224.54 last 64.24) - open/close 24199 (time 1779979177.41 total 1234.55 last 61.19) - open/close 24816 (time 1779979187.42 total 1244.56 last 61.63) - open/close 25415 (time 1779979197.43 total 1254.56 last 59.86) - open/close 26020 (time 1779979207.44 total 1264.57 last 60.44) - open/close 26619 (time 1779979217.46 total 1274.59 last 59.78) - open/close 27225 (time 1779979227.47 total 1284.61 last 60.49) - open/close 27789 (time 1779979237.48 total 1294.62 last 56.34) - open/close 28375 (time 1779979247.49 total 1304.62 last 58.58) - open/close 28879 (time 1779979257.50 total 1314.64 last 50.34) - open/close 29434 (time 1779979267.50 total 1324.64 last 55.49) - open/close 29977 (time 1779979277.51 total 1334.65 last 54.26) - open/close 30473 (time 1779979287.54 total 1344.68 last 49.44) - open/close 31090 (time 1779979297.55 total 1354.69 last 61.64) - open/close 31638 (time 1779979307.56 total 1364.70 last 54.73) - open/close 32191 (time 1779979317.58 total 1374.72 last 55.22) - open/close 32665 (time 1779979327.59 total 1384.72 last 47.36) - open/close 33260 (time 1779979337.60 total 1394.74 last 59.40) - open/close 33816 (time 1779979347.61 total 1404.75 last 55.56) - open/close 34333 (time 1779979357.62 total 1414.75 last 51.67) - open/close 35011 (time 1779979367.62 total 1424.76 last 67.77) - open/close 35561 (time 1779979377.63 total 1434.77 last 54.96) - open/close 36114 (time 1779979387.63 total 1444.77 last 55.29) - open/close 36716 (time 1779979397.63 total 1454.77 last 60.19) - open/close 37215 (time 1779979407.64 total 1464.78 last 49.86) - open/close 37774 (time 1779979417.65 total 1474.78 last 55.87) - open/close 38279 (time 1779979427.65 total 1484.79 last 50.49) - open/close 38839 (time 1779979437.65 total 1494.79 last 56.00) - open/close 39393 (time 1779979447.65 total 1504.79 last 55.37) - open/close 40000 (time 1779979457.54 total 1514.68 last 61.40) - open/close 40651 (time 1779979467.55 total 1524.69 last 65.03) - open/close 41239 (time 1779979477.58 total 1534.71 last 58.64) - open/close 41837 (time 1779979487.59 total 1544.72 last 59.74) - open/close 42390 (time 1779979497.60 total 1554.74 last 55.23) - open/close 43053 (time 1779979507.60 total 1564.74 last 66.29) - open/close 43678 (time 1779979517.63 total 1574.77 last 62.33) - open/close 44345 (time 1779979527.64 total 1584.78 last 66.62) - open/close 44892 (time 1779979537.65 total 1594.78 last 54.67) - open/close 45462 (time 1779979547.65 total 1604.78 last 56.99) - open/close 46074 (time 1779979557.65 total 1614.78 last 61.20) - open/close 46639 (time 1779979567.67 total 1624.80 last 56.39) - open/close 47250 (time 1779979577.68 total 1634.81 last 61.05) - open/close 47804 (time 1779979587.68 total 1644.82 last 55.36) - open/close 48284 (time 1779979597.69 total 1654.83 last 47.97) - open/close 48714 (time 1779979607.71 total 1664.85 last 42.92) - open/close 49100 (time 1779979617.73 total 1674.86 last 38.54) - open/close 49680 (time 1779979627.73 total 1684.87 last 57.99) total: 50000 open/close in 1689.50 seconds: 29.59 ops/second - unlinked 0 (time 1779979639 ; total 1 ; last 1) - unlinked 10000 (time 1779979839 ; total 201 ; last 200) - unlinked 20000 (time 1779980038 ; total 400 ; last 199) - unlinked 30000 (time 1779980248 ; total 610 ; last 210) - unlinked 40000 (time 1779980435 ; total 797 ; last 187) total: 50000 unlinks in 978 seconds: 51.124744 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server unloading modules via unload_modules_local on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local oleg450-server: modules unloaded. pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2