== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:41:56 (1783446116) start mds service on oleg255-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg255-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg255-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1783446187.84 total 10.05 last 18.11) - open/close 394 (time 1783446198.20 total 20.41 last 20.45) - open/close 607 (time 1783446208.63 total 30.83 last 20.44) - open/close 786 (time 1783446218.63 total 40.84 last 17.89) - open/close 998 (time 1783446229.10 total 51.31 last 20.25) - open/close 1188 (time 1783446239.20 total 61.41 last 18.81) - open/close 1316 (time 1783446249.26 total 71.47 last 12.72) - open/close 1503 (time 1783446259.27 total 81.48 last 18.69) - open/close 1747 (time 1783446269.28 total 91.49 last 24.37) - open/close 1920 (time 1783446279.55 total 101.76 last 16.84) - open/close 2166 (time 1783446289.56 total 111.77 last 24.58) - open/close 2431 (time 1783446300.21 total 122.42 last 24.87) - open/close 2643 (time 1783446310.64 total 132.85 last 20.33) - open/close 2836 (time 1783446321.14 total 143.34 last 18.39) - open/close 3012 (time 1783446331.95 total 154.16 last 16.27) - open/close 3220 (time 1783446342.76 total 164.97 last 19.24) - open/close 3440 (time 1783446353.58 total 175.79 last 20.34) - open/close 3609 (time 1783446364.35 total 186.55 last 15.70) - open/close 3790 (time 1783446375.20 total 197.41 last 16.67) - open/close 3950 (time 1783446385.66 total 207.87 last 15.30) - open/close 4078 (time 1783446395.67 total 217.88 last 12.79) - open/close 4323 (time 1783446405.68 total 227.89 last 24.47) - open/close 4514 (time 1783446415.70 total 237.91 last 19.06) - open/close 4703 (time 1783446426.73 total 248.94 last 17.14) - open/close 4835 (time 1783446436.91 total 259.12 last 12.96) - open/close 5014 (time 1783446447.52 total 269.73 last 16.87) - open/close 5223 (time 1783446457.54 total 279.74 last 20.87) - open/close 5411 (time 1783446468.21 total 290.42 last 17.61) - open/close 5675 (time 1783446479.27 total 301.48 last 23.86) - open/close 5844 (time 1783446489.96 total 312.17 last 15.81) - open/close 6055 (time 1783446500.33 total 322.54 last 20.36) - open/close 6248 (time 1783446510.33 total 332.54 last 19.30) - open/close 6452 (time 1783446521.07 total 343.28 last 18.99) - open/close 6675 (time 1783446531.08 total 353.29 last 22.28) - open/close 6867 (time 1783446541.81 total 364.02 last 17.90) - open/close 7121 (time 1783446552.17 total 374.38 last 24.52) - open/close 7382 (time 1783446563.20 total 385.40 last 23.67) - open/close 7591 (time 1783446574.20 total 396.41 last 18.99) - open/close 7785 (time 1783446584.55 total 406.76 last 18.75) - open/close 7977 (time 1783446594.56 total 416.77 last 19.18) - open/close 8053 (time 1783446605.04 total 427.24 last 7.26) - open/close 8252 (time 1783446615.15 total 437.36 last 19.67) - open/close 8501 (time 1783446625.16 total 447.37 last 24.89) - open/close 8721 (time 1783446635.56 total 457.77 last 21.15) - open/close 8903 (time 1783446646.57 total 468.78 last 16.53) - open/close 9113 (time 1783446657.00 total 479.21 last 20.13) - open/close 9264 (time 1783446667.50 total 489.71 last 14.38) - open/close 9465 (time 1783446677.95 total 500.16 last 19.24) - open/close 9672 (time 1783446688.81 total 511.02 last 19.06) - open/close 9901 (time 1783446699.69 total 521.90 last 21.04) - open/close 10000 (time 1783446706.18 total 528.39 last 15.27) - open/close 10206 (time 1783446716.39 total 538.60 last 20.17) - open/close 10396 (time 1783446726.45 total 548.65 last 18.90) - open/close 10515 (time 1783446737.44 total 559.65 last 10.82) - open/close 10635 (time 1783446748.41 total 570.62 last 10.94) - open/close 10795 (time 1783446758.91 total 581.12 last 15.23) - open/close 10922 (time 1783446768.91 total 591.12 last 12.70) - open/close 11019 (time 1783446779.44 total 601.65 last 9.21) - open/close 11153 (time 1783446789.55 total 611.76 last 13.25) - open/close 11288 (time 1783446799.72 total 621.93 last 13.28) - open/close 11416 (time 1783446809.72 total 631.93 last 12.80) - open/close 11640 (time 1783446819.89 total 642.10 last 22.02) - open/close 11830 (time 1783446829.89 total 652.10 last 19.00) - open/close 12016 (time 1783446839.90 total 662.11 last 18.59) - open/close 12132 (time 1783446850.93 total 673.13 last 10.52) - open/close 12322 (time 1783446860.94 total 683.15 last 18.97) - open/close 12789 (time 1783446870.95 total 693.15 last 46.69) - open/close 13685 (time 1783446880.95 total 703.16 last 89.53) - open/close 14730 (time 1783446890.96 total 713.17 last 104.43) - open/close 15786 (time 1783446900.97 total 723.18 last 105.50) - open/close 16866 (time 1783446910.97 total 733.18 last 107.97) - open/close 18029 (time 1783446920.99 total 743.20 last 116.12) - open/close 19124 (time 1783446930.99 total 753.20 last 109.44) - open/close 20000 (time 1783446938.23 total 760.44 last 120.98) - open/close 21070 (time 1783446948.25 total 770.45 last 106.87) - open/close 22067 (time 1783446958.25 total 780.46 last 99.66) - open/close 23092 (time 1783446968.26 total 790.47 last 102.44) - open/close 24129 (time 1783446978.26 total 800.47 last 103.68) - open/close 25169 (time 1783446988.26 total 810.47 last 103.96) - open/close 26149 (time 1783446998.26 total 820.47 last 98.00) - open/close 27166 (time 1783447008.27 total 830.47 last 101.68) - open/close 28038 (time 1783447018.27 total 840.48 last 87.18) - open/close 29050 (time 1783447028.27 total 850.48 last 101.18) - open/close 30000 (time 1783447037.35 total 859.56 last 104.58) - open/close 30902 (time 1783447047.36 total 869.57 last 90.14) - open/close 31921 (time 1783447057.37 total 879.58 last 101.83) - open/close 32979 (time 1783447067.37 total 889.58 last 105.74) - open/close 33978 (time 1783447077.38 total 899.59 last 99.82) - open/close 34868 (time 1783447087.39 total 909.60 last 88.92) - open/close 35894 (time 1783447097.39 total 919.60 last 102.55) - open/close 36962 (time 1783447107.39 total 929.60 last 106.80) - open/close 38042 (time 1783447117.40 total 939.61 last 107.91) - open/close 39095 (time 1783447127.40 total 949.61 last 105.29) - open/close 40000 (time 1783447136.94 total 959.15 last 94.91) - open/close 41097 (time 1783447146.94 total 969.15 last 109.66) - open/close 42174 (time 1783447156.95 total 979.16 last 107.62) - open/close 43214 (time 1783447166.96 total 989.17 last 103.86) - open/close 44239 (time 1783447176.97 total 999.18 last 102.46) - open/close 45211 (time 1783447186.97 total 1009.18 last 97.19) - open/close 46139 (time 1783447196.97 total 1019.18 last 92.76) - open/close 47274 (time 1783447206.99 total 1029.20 last 113.31) - open/close 48166 (time 1783447217.00 total 1039.20 last 89.13) - open/close 49243 (time 1783447227.00 total 1049.21 last 107.68) total: 50000 open/close in 1056.63 seconds: 47.32 ops/second - unlinked 0 (time 1783447236 ; total 0 ; last 0) - unlinked 10000 (time 1783447340 ; total 104 ; last 104) - unlinked 20000 (time 1783447405 ; total 169 ; last 65) - unlinked 30000 (time 1783447461 ; total 225 ; last 56) - unlinked 40000 (time 1783447508 ; total 272 ; last 47) total: 50000 unlinks in 322 seconds: 155.279510 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2