== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:33:52 (1786952032) start mds service on oleg403-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg403-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 134 (time 1786952133.43 total 10.02 last 13.37) - open/close 225 (time 1786952143.45 total 20.03 last 9.09) - open/close 398 (time 1786952154.23 total 30.82 last 16.04) - open/close 544 (time 1786952165.22 total 41.81 last 13.28) - open/close 784 (time 1786952175.22 total 51.81 last 23.99) - open/close 1043 (time 1786952186.02 total 62.61 last 23.99) - open/close 1295 (time 1786952196.89 total 73.48 last 23.19) - open/close 1506 (time 1786952206.89 total 83.48 last 21.09) - open/close 1759 (time 1786952216.90 total 93.49 last 25.27) - open/close 1916 (time 1786952227.76 total 104.35 last 14.46) - open/close 2116 (time 1786952238.46 total 115.05 last 18.69) - open/close 2243 (time 1786952248.47 total 125.06 last 12.69) - open/close 2448 (time 1786952258.49 total 135.07 last 20.47) - open/close 2579 (time 1786952268.49 total 145.08 last 13.09) - open/close 2742 (time 1786952278.97 total 155.56 last 15.56) - open/close 2886 (time 1786952290.27 total 166.86 last 12.74) - open/close 2983 (time 1786952300.82 total 177.40 last 9.20) - open/close 3159 (time 1786952311.01 total 187.60 last 17.26) - open/close 3324 (time 1786952321.71 total 198.29 last 15.43) - open/close 3525 (time 1786952331.90 total 208.49 last 19.72) - open/close 3797 (time 1786952341.91 total 218.50 last 27.17) - open/close 3927 (time 1786952352.37 total 228.96 last 12.42) - open/close 4185 (time 1786952362.38 total 238.96 last 25.79) - open/close 4426 (time 1786952372.89 total 249.48 last 22.92) - open/close 4681 (time 1786952383.96 total 260.55 last 23.03) - open/close 4933 (time 1786952393.98 total 270.57 last 25.16) - open/close 5112 (time 1786952404.00 total 280.59 last 17.87) - open/close 5241 (time 1786952414.14 total 290.73 last 12.72) - open/close 5373 (time 1786952425.13 total 301.72 last 12.00) - open/close 5550 (time 1786952435.16 total 311.75 last 17.65) - open/close 5708 (time 1786952445.17 total 321.76 last 15.78) - open/close 5830 (time 1786952455.18 total 331.76 last 12.20) - open/close 6076 (time 1786952465.21 total 341.80 last 24.52) - open/close 6307 (time 1786952476.26 total 352.84 last 20.91) - open/close 6489 (time 1786952487.19 total 363.78 last 16.64) - open/close 6761 (time 1786952497.20 total 373.79 last 27.18) - open/close 6975 (time 1786952507.25 total 383.84 last 21.28) - open/close 7143 (time 1786952517.29 total 393.88 last 16.74) - open/close 7397 (time 1786952527.29 total 403.88 last 25.40) - open/close 7628 (time 1786952537.83 total 414.42 last 21.92) - open/close 7776 (time 1786952547.84 total 424.43 last 14.78) - open/close 7951 (time 1786952558.62 total 435.21 last 16.24) - open/close 8068 (time 1786952569.19 total 445.78 last 11.06) - open/close 8266 (time 1786952579.21 total 455.80 last 19.77) - open/close 8457 (time 1786952590.16 total 466.75 last 17.45) - open/close 8658 (time 1786952600.17 total 476.76 last 20.07) - open/close 8907 (time 1786952610.80 total 487.39 last 23.43) - open/close 9116 (time 1786952621.60 total 498.19 last 19.35) - open/close 9374 (time 1786952631.61 total 508.20 last 25.78) - open/close 9622 (time 1786952642.28 total 518.87 last 23.24) - open/close 9838 (time 1786952652.58 total 529.17 last 20.97) - open/close 10000 (time 1786952661.37 total 537.95 last 18.45) - open/close 10177 (time 1786952673.65 total 550.23 last 14.41) - open/close 10328 (time 1786952683.75 total 560.34 last 14.95) - open/close 10521 (time 1786952694.05 total 570.64 last 18.73) - open/close 10715 (time 1786952704.35 total 580.94 last 18.84) - open/close 10880 (time 1786952715.78 total 592.37 last 14.44) - open/close 10956 (time 1786952725.81 total 602.39 last 7.58) - open/close 11031 (time 1786952737.00 total 613.59 last 6.70) - open/close 11220 (time 1786952747.75 total 624.34 last 17.58) - open/close 11480 (time 1786952757.76 total 634.35 last 25.98) - open/close 11763 (time 1786952768.48 total 645.07 last 26.40) - open/close 11996 (time 1786952779.17 total 655.76 last 21.79) - open/close 12236 (time 1786952789.18 total 665.77 last 23.98) - open/close 12436 (time 1786952799.18 total 675.77 last 20.00) - open/close 12665 (time 1786952809.19 total 685.78 last 22.88) - open/close 12886 (time 1786952819.20 total 695.79 last 22.08) - open/close 13115 (time 1786952829.80 total 706.39 last 21.61) - open/close 13366 (time 1786952840.88 total 717.46 last 22.66) - open/close 13558 (time 1786952850.88 total 727.47 last 19.19) - open/close 13799 (time 1786952861.73 total 738.32 last 22.22) - open/close 14032 (time 1786952872.93 total 749.52 last 20.80) - open/close 14238 (time 1786952885.38 total 761.96 last 16.56) - open/close 14444 (time 1786952895.39 total 771.98 last 20.57) - open/close 14697 (time 1786952906.21 total 782.80 last 23.39) - open/close 14945 (time 1786952916.21 total 792.80 last 24.79) - open/close 15178 (time 1786952927.17 total 803.75 last 21.28) - open/close 15429 (time 1786952937.84 total 814.43 last 23.52) - open/close 15683 (time 1786952948.79 total 825.38 last 23.20) - open/close 15906 (time 1786952958.80 total 835.38 last 22.28) - open/close 16172 (time 1786952969.46 total 846.04 last 24.95) - open/close 16401 (time 1786952979.50 total 856.08 last 22.81) - open/close 16631 (time 1786952990.56 total 867.15 last 20.78) - open/close 16764 (time 1786953000.57 total 877.16 last 13.29) - open/close 16909 (time 1786953010.81 total 887.40 last 14.16) - open/close 16995 (time 1786953020.83 total 897.41 last 8.59) - open/close 17130 (time 1786953031.99 total 908.58 last 12.09) - open/close 17328 (time 1786953042.29 total 918.88 last 19.21) - open/close 17473 (time 1786953053.10 total 929.69 last 13.41) - open/close 17726 (time 1786953065.19 total 941.78 last 20.93) - open/close 17893 (time 1786953076.91 total 953.50 last 14.25) - open/close 18068 (time 1786953086.96 total 963.55 last 17.41) - open/close 18253 (time 1786953099.50 total 976.09 last 14.75) - open/close 18426 (time 1786953109.52 total 986.10 last 17.28) - open/close 18764 (time 1786953119.53 total 996.12 last 33.74) - open/close 19413 (time 1786953129.54 total 1006.13 last 64.86) - open/close 20000 (time 1786953137.86 total 1014.45 last 70.56) - open/close 20681 (time 1786953147.86 total 1024.45 last 68.07) - open/close 21293 (time 1786953157.88 total 1034.46 last 61.11) - open/close 21916 (time 1786953167.90 total 1044.49 last 62.14) - open/close 22514 (time 1786953177.90 total 1054.49 last 59.80) - open/close 23139 (time 1786953187.92 total 1064.50 last 62.42) - open/close 23729 (time 1786953197.92 total 1074.51 last 58.97) - open/close 24333 (time 1786953207.92 total 1084.51 last 60.37) - open/close 25020 (time 1786953217.93 total 1094.52 last 68.64) - open/close 25677 (time 1786953227.94 total 1104.53 last 65.65) - open/close 26311 (time 1786953237.94 total 1114.53 last 63.38) - open/close 26935 (time 1786953247.94 total 1124.53 last 62.39) - open/close 27588 (time 1786953257.95 total 1134.54 last 65.26) - open/close 28191 (time 1786953267.97 total 1144.55 last 60.21) - open/close 28789 (time 1786953277.97 total 1154.55 last 59.79) - open/close 29353 (time 1786953287.97 total 1164.56 last 56.36) - open/close 30000 (time 1786953297.64 total 1174.23 last 66.94) - open/close 30538 (time 1786953307.65 total 1184.24 last 53.72) - open/close 31109 (time 1786953317.67 total 1194.26 last 56.99) - open/close 31732 (time 1786953327.68 total 1204.26 last 62.28) - open/close 32374 (time 1786953337.69 total 1214.28 last 64.12) - open/close 32982 (time 1786953347.70 total 1224.29 last 60.73) - open/close 33650 (time 1786953357.70 total 1234.29 last 66.78) - open/close 34309 (time 1786953367.71 total 1244.30 last 65.84) - open/close 34951 (time 1786953377.71 total 1254.30 last 64.19) - open/close 35554 (time 1786953387.74 total 1264.33 last 60.13) - open/close 36256 (time 1786953397.76 total 1274.34 last 70.10) - open/close 36854 (time 1786953407.78 total 1284.36 last 59.68) - open/close 37292 (time 1786953417.78 total 1294.37 last 43.78) - open/close 37867 (time 1786953427.79 total 1304.38 last 57.46) - open/close 38366 (time 1786953437.80 total 1314.39 last 49.83) - open/close 38942 (time 1786953447.82 total 1324.40 last 57.52) - open/close 39529 (time 1786953457.82 total 1334.41 last 58.69) - open/close 40000 (time 1786953464.91 total 1341.50 last 66.37) - open/close 40664 (time 1786953474.92 total 1351.51 last 66.34) - open/close 41238 (time 1786953484.93 total 1361.52 last 57.34) - open/close 41743 (time 1786953494.95 total 1371.54 last 50.39) - open/close 42303 (time 1786953504.96 total 1381.55 last 55.98) - open/close 42844 (time 1786953514.96 total 1391.55 last 54.08) - open/close 43341 (time 1786953524.98 total 1401.57 last 49.60) - open/close 43827 (time 1786953534.98 total 1411.57 last 48.58) - open/close 44428 (time 1786953545.00 total 1421.58 last 60.03) - open/close 45046 (time 1786953555.00 total 1431.58 last 61.79) - open/close 45719 (time 1786953565.00 total 1441.59 last 67.26) - open/close 46205 (time 1786953575.07 total 1451.66 last 48.27) - open/close 46768 (time 1786953585.07 total 1461.66 last 56.28) - open/close 47337 (time 1786953595.10 total 1471.69 last 56.76) - open/close 47888 (time 1786953605.12 total 1481.71 last 54.96) - open/close 48538 (time 1786953615.13 total 1491.72 last 64.93) - open/close 49100 (time 1786953625.14 total 1501.73 last 56.16) - open/close 49659 (time 1786953635.15 total 1511.74 last 55.85) total: 50000 open/close in 1517.35 seconds: 32.95 ops/second - unlinked 0 (time 1786953646 ; total 0 ; last 0) - unlinked 10000 (time 1786953830 ; total 184 ; last 184) - unlinked 20000 (time 1786954019 ; total 373 ; last 189) - unlinked 30000 (time 1786954203 ; total 557 ; last 184) - unlinked 40000 (time 1786954371 ; total 725 ; last 168) total: 50000 unlinks in 893 seconds: 55.991043 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2