== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:23:29 (1787585009) start mds service on oleg148-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions oleg148-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg148-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg148-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg148-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1787585103.24 total 10.46 last 12.43) - open/close 294 (time 1787585113.27 total 20.48 last 16.36) - open/close 485 (time 1787585124.06 total 31.27 last 17.70) - open/close 565 (time 1787585134.67 total 41.88 last 7.54) - open/close 756 (time 1787585145.11 total 52.32 last 18.31) - open/close 826 (time 1787585155.67 total 62.89 last 6.62) - open/close 978 (time 1787585165.70 total 72.91 last 15.16) - open/close 1138 (time 1787585176.58 total 83.79 last 14.70) - open/close 1356 (time 1787585187.66 total 94.87 last 19.69) - open/close 1455 (time 1787585197.74 total 104.95 last 9.82) - open/close 1562 (time 1787585208.12 total 115.33 last 10.31) - open/close 1633 (time 1787585218.76 total 125.97 last 6.67) - open/close 1682 (time 1787585229.57 total 136.78 last 4.53) - open/close 1766 (time 1787585240.08 total 147.29 last 7.99) - open/close 1896 (time 1787585250.48 total 157.69 last 12.51) - open/close 2060 (time 1787585261.13 total 168.34 last 15.39) - open/close 2226 (time 1787585271.15 total 178.36 last 16.58) - open/close 2331 (time 1787585281.16 total 188.37 last 10.49) - open/close 2421 (time 1787585291.91 total 199.12 last 8.37) - open/close 2549 (time 1787585301.94 total 209.15 last 12.77) - open/close 2731 (time 1787585311.94 total 219.15 last 18.19) - open/close 2873 (time 1787585321.98 total 229.19 last 14.15) - open/close 2992 (time 1787585332.55 total 239.77 last 11.25) - open/close 3062 (time 1787585342.58 total 249.79 last 6.98) - open/close 3175 (time 1787585352.81 total 260.02 last 11.04) - open/close 3400 (time 1787585362.82 total 270.03 last 22.48) - open/close 3595 (time 1787585372.90 total 280.11 last 19.36) - open/close 3837 (time 1787585382.90 total 290.11 last 24.18) - open/close 3981 (time 1787585392.91 total 300.12 last 14.39) - open/close 4097 (time 1787585402.93 total 310.14 last 11.58) - open/close 4222 (time 1787585413.02 total 320.23 last 12.39) - open/close 4377 (time 1787585423.89 total 331.11 last 14.25) - open/close 4616 (time 1787585434.39 total 341.61 last 22.76) - open/close 4851 (time 1787585444.41 total 351.62 last 23.47) - open/close 4938 (time 1787585454.85 total 362.06 last 8.33) - open/close 5179 (time 1787585464.96 total 372.18 last 23.83) - open/close 5303 (time 1787585475.34 total 382.55 last 11.96) - open/close 5392 (time 1787585486.23 total 393.44 last 8.17) - open/close 5543 (time 1787585496.46 total 403.67 last 14.76) - open/close 5681 (time 1787585506.47 total 413.68 last 13.78) - open/close 5803 (time 1787585517.38 total 424.59 last 11.18) - open/close 5992 (time 1787585527.69 total 434.90 last 18.33) - open/close 6087 (time 1787585537.69 total 444.90 last 9.50) - open/close 6215 (time 1787585547.72 total 454.93 last 12.76) - open/close 6400 (time 1787585558.27 total 465.48 last 17.54) - open/close 6475 (time 1787585568.62 total 475.83 last 7.25) - open/close 6553 (time 1787585578.92 total 486.13 last 7.58) - open/close 6733 (time 1787585591.95 total 499.16 last 13.81) - open/close 6842 (time 1787585602.89 total 510.10 last 9.97) - open/close 7002 (time 1787585613.84 total 521.05 last 14.61) - open/close 7278 (time 1787585623.85 total 531.07 last 27.56) - open/close 7427 (time 1787585633.87 total 541.08 last 14.87) - open/close 7541 (time 1787585643.91 total 551.12 last 11.36) - open/close 7774 (time 1787585654.54 total 561.75 last 21.91) - open/close 8002 (time 1787585664.54 total 571.75 last 22.79) - open/close 8103 (time 1787585674.56 total 581.77 last 10.08) - open/close 8255 (time 1787585684.64 total 591.85 last 15.08) - open/close 8412 (time 1787585696.78 total 603.99 last 12.93) - open/close 8539 (time 1787585707.79 total 615.01 last 11.53) - open/close 8651 (time 1787585718.16 total 625.37 last 10.81) - open/close 8752 (time 1787585728.88 total 636.09 last 9.42) - open/close 8939 (time 1787585739.91 total 647.12 last 16.95) - open/close 9107 (time 1787585749.92 total 657.13 last 16.78) - open/close 9194 (time 1787585759.94 total 667.15 last 8.68) - open/close 9329 (time 1787585770.22 total 677.43 last 13.13) - open/close 9441 (time 1787585780.89 total 688.10 last 10.49) - open/close 9536 (time 1787585791.30 total 698.51 last 9.12) - open/close 9642 (time 1787585801.68 total 708.89 last 10.21) - open/close 9821 (time 1787585813.47 total 720.68 last 15.18) - open/close 10000 (time 1787585820.06 total 727.27 last 27.17) - open/close 10181 (time 1787585831.05 total 738.26 last 16.48) - open/close 10487 (time 1787585842.14 total 749.35 last 27.58) - open/close 10650 (time 1787585852.74 total 759.95 last 15.39) - open/close 10751 (time 1787585863.42 total 770.63 last 9.45) - open/close 10846 (time 1787585873.80 total 781.01 last 9.15) - open/close 10920 (time 1787585884.06 total 791.27 last 7.22) - open/close 11016 (time 1787585894.27 total 801.48 last 9.40) - open/close 11161 (time 1787585904.60 total 811.82 last 14.03) - open/close 11221 (time 1787585914.61 total 821.82 last 6.00) - open/close 11409 (time 1787585924.62 total 831.83 last 18.79) - open/close 11566 (time 1787585935.07 total 842.28 last 15.02) - open/close 11734 (time 1787585945.60 total 852.81 last 15.95) - open/close 11883 (time 1787585958.68 total 865.89 last 11.40) - open/close 12063 (time 1787585968.68 total 875.89 last 17.99) - open/close 12297 (time 1787585979.56 total 886.77 last 21.52) - open/close 12449 (time 1787585989.57 total 896.78 last 15.17) - open/close 12659 (time 1787586002.10 total 909.31 last 16.77) - open/close 12897 (time 1787586013.06 total 920.27 last 21.71) - open/close 13060 (time 1787586024.64 total 931.85 last 14.08) - open/close 13254 (time 1787586034.64 total 941.85 last 19.40) - open/close 13439 (time 1787586044.64 total 951.85 last 18.50) - open/close 13617 (time 1787586054.65 total 961.86 last 17.79) - open/close 13828 (time 1787586065.15 total 972.36 last 20.09) - open/close 14007 (time 1787586075.41 total 982.62 last 17.45) - open/close 14272 (time 1787586085.43 total 992.64 last 26.45) - open/close 14489 (time 1787586096.27 total 1003.48 last 20.02) - open/close 14734 (time 1787586107.07 total 1014.28 last 22.68) - open/close 14932 (time 1787586117.54 total 1024.75 last 18.92) - open/close 15159 (time 1787586127.55 total 1034.76 last 22.67) - open/close 15334 (time 1787586138.59 total 1045.80 last 15.85) - open/close 15423 (time 1787586148.61 total 1055.82 last 8.88) - open/close 15642 (time 1787586158.62 total 1065.83 last 21.88) - open/close 15871 (time 1787586169.22 total 1076.43 last 21.60) - open/close 16313 (time 1787586179.24 total 1086.45 last 44.12) - open/close 16901 (time 1787586189.24 total 1096.45 last 58.77) - open/close 17598 (time 1787586199.26 total 1106.47 last 69.54) - open/close 18269 (time 1787586209.26 total 1116.47 last 67.10) - open/close 19035 (time 1787586219.27 total 1126.48 last 76.55) - open/close 19728 (time 1787586229.28 total 1136.49 last 69.26) - open/close 20000 (time 1787586233.59 total 1140.80 last 63.09) - open/close 20663 (time 1787586243.59 total 1150.80 last 66.28) - open/close 21365 (time 1787586253.60 total 1160.82 last 70.10) - open/close 22090 (time 1787586263.61 total 1170.82 last 72.47) - open/close 22828 (time 1787586273.62 total 1180.83 last 73.76) - open/close 23553 (time 1787586283.62 total 1190.83 last 72.46) - open/close 24157 (time 1787586293.64 total 1200.85 last 60.30) - open/close 24748 (time 1787586303.66 total 1210.87 last 58.96) - open/close 25356 (time 1787586313.66 total 1220.87 last 60.80) - open/close 25929 (time 1787586323.67 total 1230.88 last 57.28) - open/close 26591 (time 1787586333.67 total 1240.88 last 66.20) - open/close 27364 (time 1787586343.67 total 1250.88 last 77.24) - open/close 28079 (time 1787586353.69 total 1260.90 last 71.42) - open/close 28749 (time 1787586363.69 total 1270.90 last 66.96) - open/close 29492 (time 1787586373.69 total 1280.90 last 74.28) - open/close 30000 (time 1787586380.30 total 1287.51 last 76.95) - open/close 30604 (time 1787586390.30 total 1297.51 last 60.39) - open/close 31245 (time 1787586400.30 total 1307.52 last 64.06) - open/close 31736 (time 1787586410.33 total 1317.54 last 48.99) - open/close 32242 (time 1787586420.34 total 1327.55 last 50.55) - open/close 32880 (time 1787586430.35 total 1337.56 last 63.74) - open/close 33467 (time 1787586440.36 total 1347.57 last 58.64) - open/close 34017 (time 1787586450.38 total 1357.59 last 54.88) - open/close 34575 (time 1787586460.38 total 1367.59 last 55.79) - open/close 35069 (time 1787586470.44 total 1377.65 last 49.11) - open/close 35736 (time 1787586480.45 total 1387.67 last 66.60) - open/close 36402 (time 1787586490.46 total 1397.67 last 66.55) - open/close 37066 (time 1787586500.47 total 1407.68 last 66.33) - open/close 37635 (time 1787586510.48 total 1417.69 last 56.85) - open/close 38436 (time 1787586520.49 total 1427.70 last 80.04) - open/close 39038 (time 1787586530.51 total 1437.73 last 60.05) - open/close 39635 (time 1787586540.53 total 1447.74 last 59.63) - open/close 40000 (time 1787586546.33 total 1453.55 last 62.85) - open/close 40651 (time 1787586556.33 total 1463.55 last 65.10) - open/close 41340 (time 1787586566.34 total 1473.55 last 68.89) - open/close 42011 (time 1787586576.34 total 1483.55 last 67.09) - open/close 42668 (time 1787586586.35 total 1493.56 last 65.61) - open/close 43267 (time 1787586596.36 total 1503.57 last 59.84) - open/close 43875 (time 1787586606.36 total 1513.58 last 60.78) - open/close 44588 (time 1787586616.37 total 1523.58 last 71.25) - open/close 45311 (time 1787586626.38 total 1533.59 last 72.26) - open/close 46029 (time 1787586636.39 total 1543.60 last 71.68) - open/close 46714 (time 1787586646.40 total 1553.61 last 68.48) - open/close 47208 (time 1787586656.41 total 1563.62 last 49.34) - open/close 47760 (time 1787586666.41 total 1573.62 last 55.18) - open/close 48434 (time 1787586676.41 total 1583.62 last 67.39) - open/close 49033 (time 1787586686.42 total 1593.63 last 59.87) - open/close 49667 (time 1787586696.43 total 1603.64 last 63.35) total: 50000 open/close in 1608.35 seconds: 31.09 ops/second - unlinked 0 (time 1787586706 ; total 0 ; last 0) - unlinked 10000 (time 1787586867 ; total 161 ; last 161) - unlinked 20000 (time 1787587027 ; total 321 ; last 160) - unlinked 30000 (time 1787587190 ; total 484 ; last 163) - unlinked 40000 (time 1787587347 ; total 641 ; last 157) total: 50000 unlinks in 805 seconds: 62.111801 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server unloading modules via unload_modules_local on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing unload_modules_local oleg148-server: modules unloaded. pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2