== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:13:33 (1789121613) start mds service on oleg422-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg422-server: detected 4 online CPUs by sysfs oleg422-server: MODOPTS_LIBCFS= oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg422-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg422-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 198 (time 1789121703.24 total 10.51 last 18.85) - open/close 271 (time 1789121713.25 total 20.52 last 7.29) - open/close 489 (time 1789121723.50 total 30.77 last 21.27) - open/close 702 (time 1789121734.03 total 41.30 last 20.23) - open/close 851 (time 1789121744.06 total 51.33 last 14.86) - open/close 1109 (time 1789121754.09 total 61.36 last 25.72) - open/close 1362 (time 1789121765.00 total 72.27 last 23.18) - open/close 1647 (time 1789121775.62 total 82.89 last 26.84) - open/close 1707 (time 1789121786.55 total 93.82 last 5.49) - open/close 1848 (time 1789121796.93 total 104.20 last 13.59) - open/close 2061 (time 1789121808.21 total 115.48 last 18.88) - open/close 2249 (time 1789121818.72 total 125.99 last 17.89) - open/close 2384 (time 1789121829.57 total 136.84 last 12.44) - open/close 2474 (time 1789121840.06 total 147.33 last 8.58) - open/close 2653 (time 1789121850.06 total 157.33 last 17.90) - open/close 2773 (time 1789121860.68 total 167.95 last 11.30) - open/close 3020 (time 1789121871.57 total 178.84 last 22.67) - open/close 3153 (time 1789121882.52 total 189.79 last 12.15) - open/close 3362 (time 1789121892.53 total 199.80 last 20.88) - open/close 3572 (time 1789121902.73 total 210.00 last 20.59) - open/close 3795 (time 1789121912.79 total 220.06 last 22.17) - open/close 3966 (time 1789121923.27 total 230.54 last 16.32) - open/close 4216 (time 1789121934.33 total 241.60 last 22.59) - open/close 4516 (time 1789121944.88 total 252.15 last 28.44) - open/close 4673 (time 1789121954.95 total 262.22 last 15.60) - open/close 4929 (time 1789121964.95 total 272.22 last 25.59) - open/close 5186 (time 1789121974.96 total 282.23 last 25.66) - open/close 5400 (time 1789121985.46 total 292.73 last 20.40) - open/close 5668 (time 1789121996.30 total 303.57 last 24.72) - open/close 5938 (time 1789122006.85 total 314.12 last 25.60) - open/close 6204 (time 1789122016.85 total 324.12 last 26.60) - open/close 6384 (time 1789122027.65 total 334.92 last 16.65) - open/close 6522 (time 1789122037.66 total 344.93 last 13.79) - open/close 6698 (time 1789122048.34 total 355.61 last 16.49) - open/close 6793 (time 1789122058.46 total 365.73 last 9.39) - open/close 6980 (time 1789122068.46 total 375.73 last 18.70) - open/close 7178 (time 1789122081.54 total 388.81 last 15.13) - open/close 7300 (time 1789122091.55 total 398.82 last 12.19) - open/close 7581 (time 1789122101.56 total 408.83 last 28.07) - open/close 7839 (time 1789122112.38 total 419.65 last 23.85) - open/close 8085 (time 1789122125.05 total 432.32 last 19.41) - open/close 8245 (time 1789122135.92 total 443.19 last 14.72) - open/close 8386 (time 1789122145.99 total 453.26 last 14.02) - open/close 8598 (time 1789122157.13 total 464.40 last 19.02) - open/close 8762 (time 1789122167.24 total 474.51 last 16.22) - open/close 8904 (time 1789122177.48 total 484.75 last 13.86) - open/close 9091 (time 1789122188.30 total 495.57 last 17.29) - open/close 9270 (time 1789122199.24 total 506.51 last 16.36) - open/close 9344 (time 1789122209.39 total 516.66 last 7.29) - open/close 9525 (time 1789122220.36 total 527.63 last 16.49) - open/close 9738 (time 1789122230.45 total 537.72 last 21.12) - open/close 9931 (time 1789122241.23 total 548.50 last 17.90) - open/close 10000 (time 1789122245.16 total 552.43 last 17.55) - open/close 10161 (time 1789122255.17 total 562.44 last 16.08) - open/close 10262 (time 1789122265.34 total 572.61 last 9.93) - open/close 10413 (time 1789122276.02 total 583.29 last 14.14) - open/close 10539 (time 1789122286.21 total 593.48 last 12.37) - open/close 10619 (time 1789122296.21 total 603.48 last 7.99) - open/close 10668 (time 1789122306.76 total 614.03 last 4.65) - open/close 10876 (time 1789122317.58 total 624.84 last 19.24) - open/close 11058 (time 1789122327.87 total 635.14 last 17.68) - open/close 11299 (time 1789122338.94 total 646.21 last 21.76) - open/close 11528 (time 1789122349.62 total 656.89 last 21.45) - open/close 11743 (time 1789122359.87 total 667.14 last 20.98) - open/close 11936 (time 1789122369.87 total 677.14 last 19.29) - open/close 12096 (time 1789122379.88 total 687.15 last 15.99) - open/close 12388 (time 1789122389.89 total 697.16 last 29.17) - open/close 12619 (time 1789122400.26 total 707.53 last 22.28) - open/close 12744 (time 1789122411.26 total 718.53 last 11.36) - open/close 12839 (time 1789122422.14 total 729.41 last 8.73) - open/close 12941 (time 1789122432.51 total 739.78 last 9.83) - open/close 13050 (time 1789122443.00 total 750.27 last 10.40) - open/close 13182 (time 1789122453.30 total 760.57 last 12.81) - open/close 13298 (time 1789122463.99 total 771.26 last 10.85) - open/close 13399 (time 1789122475.58 total 782.84 last 8.72) - open/close 13525 (time 1789122485.59 total 792.86 last 12.58) - open/close 13727 (time 1789122496.17 total 803.44 last 19.10) - open/close 13929 (time 1789122506.37 total 813.64 last 19.80) - open/close 14147 (time 1789122516.49 total 823.76 last 21.55) - open/close 14361 (time 1789122526.86 total 834.13 last 20.64) - open/close 14624 (time 1789122536.91 total 844.18 last 26.16) - open/close 14825 (time 1789122546.94 total 854.21 last 20.05) - open/close 14962 (time 1789122558.01 total 865.28 last 12.37) - open/close 15118 (time 1789122568.84 total 876.11 last 14.41) - open/close 15254 (time 1789122578.85 total 886.12 last 13.58) - open/close 15480 (time 1789122589.12 total 896.39 last 22.01) - open/close 15684 (time 1789122599.16 total 906.43 last 20.31) - open/close 15814 (time 1789122609.18 total 916.45 last 12.98) - open/close 15935 (time 1789122619.25 total 926.52 last 12.02) - open/close 16205 (time 1789122630.27 total 937.54 last 24.50) - open/close 16332 (time 1789122640.27 total 947.54 last 12.69) - open/close 16466 (time 1789122650.68 total 957.95 last 12.88) - open/close 16685 (time 1789122660.69 total 967.96 last 21.89) - open/close 16886 (time 1789122670.69 total 977.96 last 20.10) - open/close 17132 (time 1789122680.76 total 988.03 last 24.42) - open/close 17243 (time 1789122690.89 total 998.16 last 10.97) - open/close 17373 (time 1789122701.17 total 1008.44 last 12.64) - open/close 17659 (time 1789122711.18 total 1018.45 last 28.56) - open/close 18033 (time 1789122721.19 total 1028.46 last 37.39) - open/close 18812 (time 1789122731.20 total 1038.47 last 77.83) - open/close 19586 (time 1789122741.21 total 1048.48 last 77.32) - open/close 20000 (time 1789122746.34 total 1053.61 last 80.57) - open/close 20710 (time 1789122756.35 total 1063.62 last 70.98) - open/close 21304 (time 1789122766.39 total 1073.66 last 59.15) - open/close 21915 (time 1789122776.40 total 1083.67 last 61.05) - open/close 22589 (time 1789122786.40 total 1093.67 last 67.35) - open/close 23261 (time 1789122796.43 total 1103.70 last 67.05) - open/close 23940 (time 1789122806.43 total 1113.70 last 67.86) - open/close 24614 (time 1789122816.46 total 1123.73 last 67.19) - open/close 25247 (time 1789122826.46 total 1133.73 last 63.30) - open/close 25925 (time 1789122836.46 total 1143.73 last 67.80) - open/close 26543 (time 1789122846.47 total 1153.74 last 61.76) - open/close 27098 (time 1789122856.48 total 1163.75 last 55.43) - open/close 27647 (time 1789122866.49 total 1173.76 last 54.87) - open/close 28298 (time 1789122876.50 total 1183.77 last 65.04) - open/close 28965 (time 1789122886.52 total 1193.79 last 66.56) - open/close 29635 (time 1789122896.52 total 1203.79 last 66.96) - open/close 30000 (time 1789122901.02 total 1208.29 last 81.11) - open/close 30687 (time 1789122911.02 total 1218.29 last 68.70) - open/close 31400 (time 1789122921.03 total 1228.30 last 71.27) - open/close 32104 (time 1789122931.03 total 1238.30 last 70.38) - open/close 32778 (time 1789122941.04 total 1248.31 last 67.33) - open/close 33410 (time 1789122951.04 total 1258.31 last 63.18) - open/close 34035 (time 1789122961.05 total 1268.32 last 62.44) - open/close 34768 (time 1789122971.06 total 1278.33 last 73.22) - open/close 35425 (time 1789122981.07 total 1288.33 last 65.69) - open/close 36021 (time 1789122991.09 total 1298.36 last 59.48) - open/close 36642 (time 1789123001.09 total 1308.36 last 62.09) - open/close 37279 (time 1789123011.09 total 1318.36 last 63.70) - open/close 37959 (time 1789123021.09 total 1328.36 last 67.96) - open/close 38677 (time 1789123031.09 total 1338.36 last 71.79) - open/close 39303 (time 1789123041.09 total 1348.36 last 62.60) - open/close 39957 (time 1789123051.12 total 1358.38 last 65.26) - open/close 40661 (time 1789123061.12 total 1368.39 last 70.39) - open/close 41408 (time 1789123071.13 total 1378.40 last 74.62) - open/close 42088 (time 1789123081.13 total 1388.40 last 68.00) - open/close 42824 (time 1789123091.14 total 1398.41 last 73.51) - open/close 43536 (time 1789123101.15 total 1408.42 last 71.11) - open/close 44212 (time 1789123111.16 total 1418.42 last 67.58) - open/close 44871 (time 1789123121.16 total 1428.42 last 65.90) - open/close 45577 (time 1789123131.17 total 1438.44 last 70.49) - open/close 46333 (time 1789123141.17 total 1448.44 last 75.59) - open/close 47017 (time 1789123151.18 total 1458.45 last 68.35) - open/close 47661 (time 1789123161.19 total 1468.46 last 64.37) - open/close 48311 (time 1789123171.20 total 1478.47 last 64.92) - open/close 49082 (time 1789123181.20 total 1488.47 last 77.09) - open/close 49847 (time 1789123191.20 total 1498.47 last 76.49) total: 50000 open/close in 1500.53 seconds: 33.32 ops/second - unlinked 0 (time 1789123198 ; total 0 ; last 0) - unlinked 10000 (time 1789123350 ; total 152 ; last 152) - unlinked 20000 (time 1789123500 ; total 302 ; last 150) - unlinked 30000 (time 1789123645 ; total 447 ; last 145) - unlinked 40000 (time 1789123791 ; total 593 ; last 146) total: 50000 unlinks in 735 seconds: 68.027214 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server unloading modules via unload_modules_local on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing unload_modules_local oleg422-server: modules unloaded. pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2