== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:06:20 (1781586380) start mds service on oleg641-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing load_modules_local oleg641-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg641-server: Loading modules from /home/green/git/lustre-release/lustre oleg641-server: detected 4 online CPUs by sysfs oleg641-server: MODOPTS_LIBCFS= oleg641-server: Force libcfs to create 2 CPU partitions oleg641-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg641-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg641-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg641-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg641-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg641-server: oleg641-server.virtnet: executing set_default_debug -1 all pdsh@oleg641-client: oleg641-server: ssh exited with exit code 1 Started lustre-OST0000 oleg641-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg641-client.virtnet: -o user_xattr,flock 192.168.206.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 186 (time 1781586454.88 total 10.00 last 18.60) - open/close 372 (time 1781586465.08 total 20.21 last 18.23) - open/close 555 (time 1781586475.13 total 30.25 last 18.22) - open/close 753 (time 1781586485.55 total 40.67 last 19.00) - open/close 977 (time 1781586495.56 total 50.68 last 22.38) - open/close 1185 (time 1781586505.67 total 60.79 last 20.57) - open/close 1409 (time 1781586516.18 total 71.30 last 21.32) - open/close 1640 (time 1781586526.58 total 81.70 last 22.20) - open/close 1838 (time 1781586536.89 total 92.01 last 19.21) - open/close 2028 (time 1781586547.90 total 103.03 last 17.25) - open/close 2279 (time 1781586558.71 total 113.84 last 23.22) - open/close 2507 (time 1781586569.32 total 124.44 last 21.50) - open/close 2726 (time 1781586579.96 total 135.08 last 20.58) - open/close 2875 (time 1781586590.65 total 145.77 last 13.94) - open/close 2946 (time 1781586601.16 total 156.28 last 6.76) - open/close 3074 (time 1781586611.17 total 166.29 last 12.79) - open/close 3295 (time 1781586621.63 total 176.75 last 21.13) - open/close 3486 (time 1781586631.81 total 186.93 last 18.75) - open/close 3713 (time 1781586641.91 total 197.04 last 22.47) - open/close 3890 (time 1781586652.17 total 207.29 last 17.26) - open/close 4078 (time 1781586663.18 total 218.30 last 17.07) - open/close 4328 (time 1781586673.35 total 228.47 last 24.58) - open/close 4561 (time 1781586683.37 total 238.50 last 23.25) - open/close 4764 (time 1781586694.28 total 249.41 last 18.61) - open/close 4947 (time 1781586706.11 total 261.23 last 15.47) - open/close 5118 (time 1781586716.13 total 271.25 last 17.08) - open/close 5273 (time 1781586726.51 total 281.63 last 14.93) - open/close 5492 (time 1781586736.54 total 291.67 last 21.82) - open/close 5646 (time 1781586746.94 total 302.06 last 14.81) - open/close 5770 (time 1781586757.81 total 312.93 last 11.41) - open/close 6020 (time 1781586768.47 total 323.59 last 23.46) - open/close 6270 (time 1781586779.44 total 334.56 last 22.79) - open/close 6505 (time 1781586789.45 total 344.57 last 23.47) - open/close 6716 (time 1781586799.92 total 355.04 last 20.15) - open/close 6935 (time 1781586809.93 total 365.06 last 21.88) - open/close 7065 (time 1781586821.24 total 376.36 last 11.50) - open/close 7156 (time 1781586831.26 total 386.38 last 9.08) - open/close 7287 (time 1781586842.55 total 397.67 last 11.60) - open/close 7462 (time 1781586852.92 total 408.04 last 16.88) - open/close 7626 (time 1781586862.93 total 418.05 last 16.39) - open/close 7891 (time 1781586872.93 total 428.05 last 26.48) - open/close 8061 (time 1781586882.94 total 438.06 last 16.99) - open/close 8279 (time 1781586892.95 total 448.07 last 21.78) - open/close 8419 (time 1781586904.20 total 459.32 last 12.45) - open/close 8612 (time 1781586914.21 total 469.33 last 19.28) - open/close 8810 (time 1781586924.56 total 479.69 last 19.12) - open/close 9048 (time 1781586935.24 total 490.36 last 22.29) - open/close 9284 (time 1781586945.26 total 500.38 last 23.57) - open/close 9502 (time 1781586955.26 total 510.38 last 21.80) - open/close 9702 (time 1781586965.28 total 520.40 last 19.96) - open/close 9899 (time 1781586976.06 total 531.18 last 18.26) - open/close 10000 (time 1781586980.95 total 536.07 last 20.66) - open/close 10177 (time 1781586991.22 total 546.34 last 17.24) - open/close 10358 (time 1781587004.01 total 559.13 last 14.15) - open/close 10557 (time 1781587014.02 total 569.14 last 19.89) - open/close 10735 (time 1781587024.02 total 579.14 last 17.79) - open/close 10969 (time 1781587034.05 total 589.17 last 23.34) - open/close 11197 (time 1781587044.17 total 599.29 last 22.53) - open/close 11314 (time 1781587054.89 total 610.01 last 10.91) - open/close 11562 (time 1781587064.90 total 620.03 last 24.77) - open/close 11774 (time 1781587075.14 total 630.26 last 20.71) - open/close 12040 (time 1781587085.16 total 640.28 last 26.56) - open/close 12284 (time 1781587095.16 total 650.28 last 24.40) - open/close 12503 (time 1781587105.17 total 660.30 last 21.86) - open/close 12725 (time 1781587115.18 total 670.31 last 22.18) - open/close 12960 (time 1781587125.25 total 680.37 last 23.35) - open/close 13189 (time 1781587135.27 total 690.39 last 22.85) - open/close 13413 (time 1781587145.29 total 700.41 last 22.35) - open/close 13597 (time 1781587155.84 total 710.96 last 17.44) - open/close 13791 (time 1781587166.19 total 721.31 last 18.75) - open/close 13998 (time 1781587176.20 total 731.32 last 20.68) - open/close 14199 (time 1781587186.23 total 741.35 last 20.04) - open/close 14300 (time 1781587197.04 total 752.16 last 9.34) - open/close 14444 (time 1781587208.80 total 763.92 last 12.24) - open/close 14710 (time 1781587218.81 total 773.93 last 26.59) - open/close 14929 (time 1781587229.52 total 784.64 last 20.44) - open/close 15102 (time 1781587239.53 total 794.65 last 17.29) - open/close 15302 (time 1781587250.30 total 805.42 last 18.57) - open/close 15465 (time 1781587261.56 total 816.68 last 14.48) - open/close 15714 (time 1781587271.70 total 826.82 last 24.55) - open/close 15957 (time 1781587281.70 total 836.82 last 24.29) - open/close 16129 (time 1781587292.15 total 847.27 last 16.46) - open/close 16367 (time 1781587302.66 total 857.79 last 22.64) - open/close 16531 (time 1781587312.73 total 867.85 last 16.29) - open/close 16744 (time 1781587323.13 total 878.25 last 20.48) - open/close 16963 (time 1781587334.08 total 889.20 last 20.00) - open/close 17149 (time 1781587344.08 total 899.20 last 18.59) - open/close 17335 (time 1781587354.63 total 909.75 last 17.64) - open/close 17555 (time 1781587365.56 total 920.68 last 20.13) - open/close 18036 (time 1781587375.56 total 930.68 last 48.08) - open/close 18594 (time 1781587385.57 total 940.69 last 55.73) - open/close 19200 (time 1781587395.57 total 950.69 last 60.59) - open/close 19795 (time 1781587405.58 total 960.70 last 59.44) - open/close 20000 (time 1781587408.93 total 964.05 last 61.25) - open/close 20565 (time 1781587418.95 total 974.07 last 56.41) - open/close 21126 (time 1781587428.95 total 984.07 last 56.07) - open/close 21759 (time 1781587438.96 total 994.08 last 63.24) - open/close 22365 (time 1781587448.97 total 1004.09 last 60.53) - open/close 22954 (time 1781587458.98 total 1014.10 last 58.86) - open/close 23330 (time 1781587469.00 total 1024.13 last 37.50) - open/close 23811 (time 1781587479.00 total 1034.13 last 48.10) - open/close 24340 (time 1781587489.01 total 1044.13 last 52.90) - open/close 24924 (time 1781587499.02 total 1054.15 last 58.28) - open/close 25498 (time 1781587509.03 total 1064.15 last 57.37) - open/close 26113 (time 1781587519.06 total 1074.18 last 61.30) - open/close 26654 (time 1781587529.08 total 1084.20 last 54.01) - open/close 27246 (time 1781587539.09 total 1094.21 last 59.15) - open/close 27786 (time 1781587549.09 total 1104.21 last 54.00) - open/close 28407 (time 1781587559.09 total 1114.21 last 62.09) - open/close 29011 (time 1781587569.09 total 1124.21 last 60.37) - open/close 29661 (time 1781587579.10 total 1134.22 last 64.97) - open/close 30000 (time 1781587584.68 total 1139.80 last 60.72) - open/close 30647 (time 1781587594.69 total 1149.81 last 64.65) - open/close 31234 (time 1781587604.69 total 1159.81 last 58.68) - open/close 31894 (time 1781587614.70 total 1169.82 last 65.97) - open/close 32542 (time 1781587624.71 total 1179.83 last 64.74) - open/close 33231 (time 1781587634.74 total 1189.86 last 68.70) - open/close 33891 (time 1781587644.74 total 1199.86 last 65.96) - open/close 34599 (time 1781587654.74 total 1209.86 last 70.80) - open/close 35217 (time 1781587664.76 total 1219.88 last 61.69) - open/close 35883 (time 1781587674.77 total 1229.89 last 66.57) - open/close 36459 (time 1781587684.78 total 1239.90 last 57.52) - open/close 37077 (time 1781587694.79 total 1249.91 last 61.77) - open/close 37673 (time 1781587704.79 total 1259.91 last 59.56) - open/close 38309 (time 1781587714.81 total 1269.93 last 63.52) - open/close 39004 (time 1781587724.82 total 1279.94 last 69.43) - open/close 39681 (time 1781587734.82 total 1289.94 last 67.67) - open/close 40000 (time 1781587739.60 total 1294.72 last 66.80) - open/close 40641 (time 1781587749.60 total 1304.73 last 64.05) - open/close 41274 (time 1781587759.61 total 1314.74 last 63.24) - open/close 41900 (time 1781587769.63 total 1324.75 last 62.49) - open/close 42526 (time 1781587779.63 total 1334.75 last 62.59) - open/close 43174 (time 1781587789.64 total 1344.77 last 64.72) - open/close 43839 (time 1781587799.66 total 1354.78 last 66.42) - open/close 44472 (time 1781587809.67 total 1364.79 last 63.24) - open/close 45081 (time 1781587819.67 total 1374.79 last 60.87) - open/close 45691 (time 1781587829.69 total 1384.81 last 60.90) - open/close 46242 (time 1781587839.70 total 1394.82 last 55.02) - open/close 46858 (time 1781587849.71 total 1404.83 last 61.56) - open/close 47502 (time 1781587859.71 total 1414.83 last 64.37) - open/close 48099 (time 1781587869.73 total 1424.85 last 59.61) - open/close 48699 (time 1781587879.73 total 1434.85 last 59.98) - open/close 49320 (time 1781587889.74 total 1444.86 last 62.05) - open/close 49910 (time 1781587899.74 total 1454.87 last 58.98) total: 50000 open/close in 1456.47 seconds: 34.33 ops/second - unlinked 0 (time 1781587906 ; total 0 ; last 0) - unlinked 10000 (time 1781588074 ; total 168 ; last 168) - unlinked 20000 (time 1781588229 ; total 323 ; last 155) - unlinked 30000 (time 1781588372 ; total 466 ; last 143) - unlinked 40000 (time 1781588511 ; total 605 ; last 139) total: 50000 unlinks in 739 seconds: 67.658997 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg641-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg641-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg641-server stop mds service on oleg641-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg641-server unloading modules via unload_modules_local on: 'oleg641-server' oleg641-server: oleg641-server.virtnet: executing unload_modules_local oleg641-server: modules unloaded. pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-server: ssh exited with exit code 2 pdsh@oleg641-client: oleg641-client: ssh exited with exit code 2