== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:33:29 (1787830409) start mds service on oleg417-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg417-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1787830497.65 total 10.03 last 18.15) - open/close 254 (time 1787830508.55 total 20.92 last 6.61) - open/close 378 (time 1787830518.55 total 30.93 last 12.39) - open/close 607 (time 1787830529.54 total 41.91 last 20.84) - open/close 733 (time 1787830539.70 total 52.08 last 12.40) - open/close 904 (time 1787830550.22 total 62.59 last 16.26) - open/close 1018 (time 1787830561.06 total 73.43 last 10.51) - open/close 1183 (time 1787830571.35 total 83.72 last 16.04) - open/close 1409 (time 1787830584.00 total 96.37 last 17.86) - open/close 1570 (time 1787830594.00 total 106.38 last 16.09) - open/close 1706 (time 1787830604.80 total 117.18 last 12.60) - open/close 1821 (time 1787830614.83 total 127.21 last 11.47) - open/close 2002 (time 1787830625.60 total 137.97 last 16.81) - open/close 2153 (time 1787830636.29 total 148.66 last 14.12) - open/close 2284 (time 1787830646.79 total 159.17 last 12.47) - open/close 2410 (time 1787830657.61 total 169.99 last 11.64) - open/close 2587 (time 1787830667.62 total 179.99 last 17.70) - open/close 2847 (time 1787830678.68 total 191.05 last 23.51) - open/close 3118 (time 1787830688.70 total 201.07 last 27.04) - open/close 3328 (time 1787830700.60 total 212.98 last 17.64) - open/close 3535 (time 1787830710.86 total 223.24 last 20.18) - open/close 3786 (time 1787830721.67 total 234.04 last 23.23) - open/close 3921 (time 1787830731.68 total 244.06 last 13.48) - open/close 4173 (time 1787830742.74 total 255.11 last 22.80) - open/close 4264 (time 1787830752.97 total 265.35 last 8.89) - open/close 4500 (time 1787830762.98 total 275.36 last 23.58) - open/close 4744 (time 1787830773.96 total 286.34 last 22.22) - open/close 4949 (time 1787830783.98 total 296.35 last 20.47) - open/close 5099 (time 1787830796.41 total 308.78 last 12.07) - open/close 5321 (time 1787830807.09 total 319.47 last 20.78) - open/close 5500 (time 1787830817.67 total 330.05 last 16.92) - open/close 5713 (time 1787830828.16 total 340.53 last 20.31) - open/close 5851 (time 1787830838.16 total 350.54 last 13.80) - open/close 6001 (time 1787830848.46 total 360.83 last 14.57) - open/close 6394 (time 1787830858.47 total 370.85 last 39.25) - open/close 6625 (time 1787830868.49 total 380.87 last 23.06) - open/close 6817 (time 1787830879.23 total 391.61 last 17.87) - open/close 7016 (time 1787830890.67 total 403.05 last 17.40) - open/close 7296 (time 1787830905.37 total 417.75 last 19.04) - open/close 7517 (time 1787830916.17 total 428.54 last 20.48) - open/close 7724 (time 1787830926.61 total 438.98 last 19.82) - open/close 7927 (time 1787830937.01 total 449.39 last 19.51) - open/close 8069 (time 1787830947.53 total 459.90 last 13.51) - open/close 8322 (time 1787830957.81 total 470.19 last 24.59) - open/close 8506 (time 1787830967.82 total 480.19 last 18.40) - open/close 8748 (time 1787830977.91 total 490.29 last 23.98) - open/close 8982 (time 1787830988.41 total 500.78 last 22.29) - open/close 9234 (time 1787830999.19 total 511.57 last 23.36) - open/close 9339 (time 1787831009.89 total 522.27 last 9.82) - open/close 9420 (time 1787831019.90 total 532.28 last 8.09) - open/close 9652 (time 1787831029.92 total 542.30 last 23.15) - open/close 9886 (time 1787831040.52 total 552.89 last 22.09) - open/close 10000 (time 1787831048.12 total 560.50 last 14.99) - open/close 10222 (time 1787831058.12 total 570.50 last 22.20) - open/close 10379 (time 1787831068.48 total 580.86 last 15.15) - open/close 10631 (time 1787831078.49 total 590.86 last 25.18) - open/close 10860 (time 1787831089.16 total 601.53 last 21.47) - open/close 11097 (time 1787831099.25 total 611.63 last 23.48) - open/close 11312 (time 1787831109.26 total 621.63 last 21.49) - open/close 11596 (time 1787831120.15 total 632.52 last 26.07) - open/close 11797 (time 1787831130.16 total 642.54 last 20.07) - open/close 12033 (time 1787831140.35 total 652.73 last 23.16) - open/close 12306 (time 1787831150.36 total 662.74 last 27.27) - open/close 12501 (time 1787831160.36 total 672.74 last 19.50) - open/close 12689 (time 1787831172.80 total 685.17 last 15.12) - open/close 12883 (time 1787831182.81 total 695.18 last 19.38) - open/close 12969 (time 1787831193.86 total 706.24 last 7.78) - open/close 13113 (time 1787831203.88 total 716.26 last 14.37) - open/close 13236 (time 1787831213.88 total 726.26 last 12.30) - open/close 13400 (time 1787831224.32 total 736.69 last 15.72) - open/close 13655 (time 1787831234.32 total 746.70 last 25.50) - open/close 13922 (time 1787831245.14 total 757.52 last 24.67) - open/close 14121 (time 1787831257.09 total 769.47 last 16.65) - open/close 14426 (time 1787831268.03 total 780.40 last 27.89) - open/close 14619 (time 1787831278.62 total 790.99 last 18.22) - open/close 14710 (time 1787831289.09 total 801.46 last 8.69) - open/close 14883 (time 1787831299.85 total 812.23 last 16.07) - open/close 15012 (time 1787831309.88 total 822.26 last 12.86) - open/close 15299 (time 1787831319.89 total 832.27 last 28.67) - open/close 15569 (time 1787831330.69 total 843.06 last 25.01) - open/close 15843 (time 1787831340.70 total 853.07 last 27.38) - open/close 16043 (time 1787831352.13 total 864.50 last 17.50) - open/close 16183 (time 1787831364.73 total 877.10 last 11.11) - open/close 16413 (time 1787831374.74 total 887.11 last 22.97) - open/close 16530 (time 1787831385.34 total 897.72 last 11.04) - open/close 16629 (time 1787831397.89 total 910.27 last 7.89) - open/close 16757 (time 1787831408.66 total 921.04 last 11.88) - open/close 17004 (time 1787831418.69 total 931.07 last 24.62) - open/close 17127 (time 1787831428.76 total 941.14 last 12.21) - open/close 17389 (time 1787831439.38 total 951.75 last 24.68) - open/close 17676 (time 1787831449.38 total 961.76 last 28.69) - open/close 17867 (time 1787831459.39 total 971.77 last 19.08) - open/close 18127 (time 1787831470.34 total 982.72 last 23.74) - open/close 18444 (time 1787831480.36 total 992.74 last 31.64) - open/close 18795 (time 1787831490.36 total 1002.74 last 35.10) - open/close 19575 (time 1787831500.37 total 1012.75 last 77.94) - open/close 20000 (time 1787831505.74 total 1018.12 last 79.15) - open/close 20740 (time 1787831515.75 total 1028.13 last 73.94) - open/close 21442 (time 1787831525.76 total 1038.14 last 70.13) - open/close 22173 (time 1787831535.77 total 1048.14 last 73.06) - open/close 22881 (time 1787831545.77 total 1058.15 last 70.77) - open/close 23630 (time 1787831555.77 total 1068.15 last 74.87) - open/close 24308 (time 1787831565.78 total 1078.16 last 67.75) - open/close 24943 (time 1787831575.78 total 1088.16 last 63.49) - open/close 25615 (time 1787831585.83 total 1098.20 last 66.90) - open/close 26276 (time 1787831595.83 total 1108.20 last 66.10) - open/close 26943 (time 1787831605.84 total 1118.21 last 66.64) - open/close 27614 (time 1787831615.84 total 1128.22 last 67.08) - open/close 28224 (time 1787831625.86 total 1138.23 last 60.89) - open/close 28950 (time 1787831635.86 total 1148.24 last 72.56) - open/close 29695 (time 1787831645.87 total 1158.24 last 74.48) - open/close 30000 (time 1787831650.17 total 1162.55 last 70.80) - open/close 30602 (time 1787831660.18 total 1172.56 last 60.17) - open/close 31231 (time 1787831670.18 total 1182.56 last 62.90) - open/close 31888 (time 1787831680.19 total 1192.57 last 65.64) - open/close 32542 (time 1787831690.19 total 1202.57 last 65.38) - open/close 33236 (time 1787831700.22 total 1212.60 last 69.19) - open/close 33967 (time 1787831710.23 total 1222.60 last 73.08) - open/close 34618 (time 1787831720.23 total 1232.60 last 65.09) - open/close 35347 (time 1787831730.24 total 1242.61 last 72.81) - open/close 36082 (time 1787831740.24 total 1252.62 last 73.49) - open/close 36720 (time 1787831750.26 total 1262.63 last 63.70) - open/close 37487 (time 1787831760.26 total 1272.64 last 76.64) - open/close 38144 (time 1787831770.31 total 1282.68 last 65.41) - open/close 38736 (time 1787831780.34 total 1292.72 last 59.01) - open/close 39386 (time 1787831790.35 total 1302.72 last 64.94) - open/close 40000 (time 1787831799.46 total 1311.84 last 67.35) - open/close 40667 (time 1787831809.47 total 1321.85 last 66.64) - open/close 41357 (time 1787831819.48 total 1331.85 last 68.96) - open/close 42057 (time 1787831829.49 total 1341.86 last 69.93) - open/close 42765 (time 1787831839.50 total 1351.87 last 70.75) - open/close 43520 (time 1787831849.50 total 1361.87 last 75.50) - open/close 44267 (time 1787831859.50 total 1371.88 last 74.68) - open/close 44977 (time 1787831869.50 total 1381.88 last 70.98) - open/close 45701 (time 1787831879.51 total 1391.89 last 72.32) - open/close 46415 (time 1787831889.55 total 1401.93 last 71.13) - open/close 47114 (time 1787831899.56 total 1411.93 last 69.85) - open/close 47755 (time 1787831909.56 total 1421.94 last 64.08) - open/close 48511 (time 1787831919.58 total 1431.95 last 75.49) - open/close 49134 (time 1787831929.59 total 1441.96 last 62.24) - open/close 49849 (time 1787831939.59 total 1451.97 last 71.45) total: 50000 open/close in 1453.82 seconds: 34.39 ops/second - unlinked 0 (time 1787831946 ; total 0 ; last 0) - unlinked 10000 (time 1787832092 ; total 146 ; last 146) - unlinked 20000 (time 1787832243 ; total 297 ; last 151) - unlinked 30000 (time 1787832387 ; total 441 ; last 144) - unlinked 40000 (time 1787832531 ; total 585 ; last 144) total: 50000 unlinks in 740 seconds: 67.567566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local oleg417-server: modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2