== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:39:55 (1784047195) start mds service on oleg139-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg139-server: detected 4 online CPUs by sysfs oleg139-server: MODOPTS_LIBCFS= oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg139-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 83 (time 1784047293.17 total 11.79 last 7.04) - open/close 146 (time 1784047304.24 total 22.86 last 5.69) - open/close 219 (time 1784047314.29 total 32.91 last 7.26) - open/close 319 (time 1784047324.30 total 42.92 last 9.99) - open/close 369 (time 1784047335.07 total 53.69 last 4.64) - open/close 506 (time 1784047346.21 total 64.84 last 12.29) - open/close 731 (time 1784047356.24 total 74.86 last 22.45) - open/close 895 (time 1784047367.17 total 85.79 last 15.00) - open/close 1012 (time 1784047377.45 total 96.07 last 11.38) - open/close 1193 (time 1784047389.48 total 108.10 last 15.05) - open/close 1323 (time 1784047400.43 total 119.05 last 11.88) - open/close 1472 (time 1784047410.60 total 129.22 last 14.65) - open/close 1655 (time 1784047420.61 total 139.23 last 18.28) - open/close 1873 (time 1784047430.82 total 149.44 last 21.35) - open/close 1960 (time 1784047442.35 total 160.97 last 7.54) - open/close 2117 (time 1784047452.77 total 171.39 last 15.07) - open/close 2329 (time 1784047462.78 total 181.40 last 21.20) - open/close 2418 (time 1784047472.80 total 191.42 last 8.88) - open/close 2521 (time 1784047483.19 total 201.81 last 9.92) - open/close 2644 (time 1784047493.23 total 211.85 last 12.25) - open/close 2793 (time 1784047504.07 total 222.69 last 13.75) - open/close 2897 (time 1784047514.98 total 233.60 last 9.54) - open/close 3013 (time 1784047525.73 total 244.35 last 10.78) - open/close 3225 (time 1784047535.78 total 254.40 last 21.10) - open/close 3397 (time 1784047545.80 total 264.42 last 17.16) - open/close 3556 (time 1784047555.96 total 274.58 last 15.66) - open/close 3632 (time 1784047566.57 total 285.19 last 7.16) - open/close 3742 (time 1784047576.81 total 295.43 last 10.74) - open/close 3933 (time 1784047587.32 total 305.94 last 18.17) - open/close 4104 (time 1784047597.35 total 315.97 last 17.06) - open/close 4139 (time 1784047607.68 total 326.30 last 3.39) - open/close 4343 (time 1784047617.69 total 336.31 last 20.37) - open/close 4497 (time 1784047628.75 total 347.37 last 13.92) - open/close 4652 (time 1784047638.76 total 357.38 last 15.48) - open/close 4815 (time 1784047649.72 total 368.34 last 14.87) - open/close 5044 (time 1784047659.88 total 378.50 last 22.54) - open/close 5116 (time 1784047670.20 total 388.82 last 6.98) - open/close 5213 (time 1784047680.26 total 398.88 last 9.65) - open/close 5325 (time 1784047691.13 total 409.75 last 10.30) - open/close 5446 (time 1784047701.14 total 419.76 last 12.09) - open/close 5594 (time 1784047711.72 total 430.34 last 13.99) - open/close 5698 (time 1784047721.91 total 440.53 last 10.21) - open/close 5830 (time 1784047731.93 total 450.55 last 13.17) - open/close 6002 (time 1784047741.98 total 460.60 last 17.12) - open/close 6078 (time 1784047752.65 total 471.27 last 7.12) - open/close 6237 (time 1784047763.33 total 481.95 last 14.89) - open/close 6382 (time 1784047773.70 total 492.32 last 13.98) - open/close 6550 (time 1784047784.24 total 502.86 last 15.95) - open/close 6602 (time 1784047795.12 total 513.74 last 4.78) - open/close 6729 (time 1784047805.80 total 524.42 last 11.89) - open/close 6834 (time 1784047816.83 total 535.45 last 9.52) - open/close 6925 (time 1784047826.93 total 545.55 last 9.01) - open/close 7011 (time 1784047837.52 total 556.14 last 8.12) - open/close 7040 (time 1784047849.86 total 568.48 last 2.35) - open/close 7170 (time 1784047860.27 total 578.89 last 12.49) - open/close 7229 (time 1784047870.38 total 589.00 last 5.83) - open/close 7393 (time 1784047880.49 total 599.11 last 16.23) - open/close 7495 (time 1784047890.52 total 609.14 last 10.17) - open/close 7613 (time 1784047900.54 total 619.16 last 11.78) - open/close 7766 (time 1784047910.54 total 629.16 last 15.29) - open/close 7836 (time 1784047920.76 total 639.38 last 6.85) - open/close 7912 (time 1784047930.90 total 649.52 last 7.50) - open/close 8106 (time 1784047941.07 total 659.69 last 19.08) - open/close 8240 (time 1784047955.58 total 674.20 last 9.23) - open/close 8361 (time 1784047966.52 total 685.14 last 11.06) - open/close 8468 (time 1784047976.54 total 695.16 last 10.68) - open/close 8524 (time 1784047986.57 total 705.19 last 5.58) - open/close 8657 (time 1784047997.04 total 715.66 last 12.71) - open/close 8732 (time 1784048007.49 total 726.11 last 7.17) - open/close 8805 (time 1784048018.05 total 736.67 last 6.91) - open/close 8885 (time 1784048028.05 total 746.67 last 8.00) - open/close 9001 (time 1784048038.06 total 756.68 last 11.59) - open/close 9213 (time 1784048048.08 total 766.70 last 21.16) - open/close 9312 (time 1784048058.22 total 776.84 last 9.77) - open/close 9487 (time 1784048068.25 total 786.87 last 17.44) - open/close 9644 (time 1784048079.23 total 797.85 last 14.29) - open/close 9774 (time 1784048090.15 total 808.78 last 11.90) - open/close 10000 (time 1784048100.04 total 818.66 last 22.87) - open/close 10212 (time 1784048110.06 total 828.68 last 21.15) - open/close 10399 (time 1784048120.82 total 839.44 last 17.38) - open/close 10508 (time 1784048132.60 total 851.22 last 9.25) - open/close 10623 (time 1784048143.47 total 862.09 last 10.57) - open/close 10711 (time 1784048154.40 total 873.03 last 8.05) - open/close 10838 (time 1784048164.46 total 883.08 last 12.63) - open/close 11021 (time 1784048175.05 total 893.67 last 17.29) - open/close 11137 (time 1784048186.02 total 904.64 last 10.57) - open/close 11258 (time 1784048196.47 total 915.09 last 11.58) - open/close 11412 (time 1784048206.57 total 925.19 last 15.25) - open/close 11497 (time 1784048217.22 total 935.84 last 7.98) - open/close 11690 (time 1784048227.23 total 945.85 last 19.28) - open/close 11796 (time 1784048237.24 total 955.86 last 10.59) - open/close 11893 (time 1784048248.22 total 966.84 last 8.83) - open/close 12091 (time 1784048258.89 total 977.51 last 18.57) - open/close 12229 (time 1784048268.97 total 987.59 last 13.68) - open/close 12327 (time 1784048280.00 total 998.62 last 8.89) - open/close 12497 (time 1784048290.48 total 1009.10 last 16.23) - open/close 12621 (time 1784048301.05 total 1019.67 last 11.73) - open/close 12746 (time 1784048311.53 total 1030.15 last 11.93) - open/close 12837 (time 1784048322.49 total 1041.12 last 8.30) - open/close 12878 (time 1784048332.96 total 1051.58 last 3.92) - open/close 12975 (time 1784048343.92 total 1062.54 last 8.85) - open/close 13080 (time 1784048355.04 total 1073.66 last 9.45) - open/close 13233 (time 1784048365.94 total 1084.56 last 14.04) - open/close 13429 (time 1784048376.96 total 1095.58 last 17.78) - open/close 13611 (time 1784048386.99 total 1105.61 last 18.15) - open/close 13746 (time 1784048397.64 total 1116.26 last 12.67) - open/close 13888 (time 1784048408.67 total 1127.29 last 12.87) - open/close 13992 (time 1784048418.71 total 1137.33 last 10.37) - open/close 14096 (time 1784048428.78 total 1147.40 last 10.32) - open/close 14225 (time 1784048439.34 total 1157.96 last 12.21) - open/close 14261 (time 1784048449.78 total 1168.40 last 3.45) - open/close 14361 (time 1784048460.23 total 1178.85 last 9.57) - open/close 14729 (time 1784048470.23 total 1188.85 last 36.78) - open/close 15266 (time 1784048480.24 total 1198.86 last 53.65) - open/close 15728 (time 1784048490.25 total 1208.87 last 46.14) - open/close 16277 (time 1784048500.27 total 1218.89 last 54.83) - open/close 16940 (time 1784048510.27 total 1228.89 last 66.29) - open/close 17447 (time 1784048520.28 total 1238.90 last 50.64) - open/close 18011 (time 1784048530.30 total 1248.92 last 56.30) - open/close 18529 (time 1784048540.30 total 1258.92 last 51.78) - open/close 19053 (time 1784048550.31 total 1268.93 last 52.36) - open/close 19604 (time 1784048560.31 total 1278.93 last 55.06) - open/close 20000 (time 1784048568.39 total 1287.01 last 49.06) - open/close 20511 (time 1784048578.39 total 1297.02 last 51.06) - open/close 21114 (time 1784048588.41 total 1307.03 last 60.21) - open/close 21802 (time 1784048598.41 total 1317.03 last 68.78) - open/close 22344 (time 1784048608.42 total 1327.04 last 54.18) - open/close 22899 (time 1784048618.42 total 1337.04 last 55.48) - open/close 23564 (time 1784048628.43 total 1347.05 last 66.44) - open/close 24064 (time 1784048638.43 total 1357.05 last 50.00) - open/close 24646 (time 1784048648.44 total 1367.06 last 58.12) - open/close 25273 (time 1784048658.45 total 1377.07 last 62.65) - open/close 25858 (time 1784048668.47 total 1387.09 last 58.41) - open/close 26302 (time 1784048678.48 total 1397.10 last 44.32) - open/close 27094 (time 1784048688.49 total 1407.11 last 79.17) - open/close 27768 (time 1784048698.49 total 1417.11 last 67.40) - open/close 28354 (time 1784048708.50 total 1427.12 last 58.55) - open/close 28955 (time 1784048718.50 total 1437.12 last 60.07) - open/close 29486 (time 1784048728.51 total 1447.13 last 53.05) - open/close 30000 (time 1784048737.44 total 1456.06 last 57.59) - open/close 30643 (time 1784048747.54 total 1466.16 last 63.66) - open/close 31172 (time 1784048757.56 total 1476.18 last 52.78) - open/close 31705 (time 1784048767.56 total 1486.18 last 53.29) - open/close 32224 (time 1784048777.57 total 1496.19 last 51.85) - open/close 32835 (time 1784048787.58 total 1506.21 last 61.03) - open/close 33339 (time 1784048797.59 total 1516.21 last 50.36) - open/close 33917 (time 1784048807.60 total 1526.22 last 57.73) - open/close 34429 (time 1784048817.62 total 1536.24 last 51.14) - open/close 35112 (time 1784048827.62 total 1546.24 last 68.27) - open/close 35722 (time 1784048837.64 total 1556.26 last 60.87) - open/close 36342 (time 1784048847.66 total 1566.28 last 61.87) - open/close 36816 (time 1784048857.68 total 1576.30 last 47.33) - open/close 37442 (time 1784048867.68 total 1586.30 last 62.57) - open/close 38164 (time 1784048877.69 total 1596.31 last 72.17) - open/close 38875 (time 1784048887.70 total 1606.32 last 71.03) - open/close 39560 (time 1784048897.70 total 1616.32 last 68.47) - open/close 40000 (time 1784048903.94 total 1622.56 last 70.54) - open/close 40739 (time 1784048913.95 total 1632.57 last 73.83) - open/close 41400 (time 1784048923.95 total 1642.57 last 66.07) - open/close 42127 (time 1784048933.96 total 1652.58 last 72.67) - open/close 42833 (time 1784048943.97 total 1662.59 last 70.53) - open/close 43573 (time 1784048953.97 total 1672.59 last 73.98) - open/close 44075 (time 1784048963.99 total 1682.61 last 50.10) - open/close 44588 (time 1784048973.99 total 1692.61 last 51.27) - open/close 45169 (time 1784048984.01 total 1702.63 last 58.02) - open/close 45829 (time 1784048994.02 total 1712.64 last 65.95) - open/close 46460 (time 1784049004.02 total 1722.64 last 63.05) - open/close 47078 (time 1784049014.03 total 1732.65 last 61.76) - open/close 47684 (time 1784049024.07 total 1742.70 last 60.33) - open/close 48206 (time 1784049034.08 total 1752.70 last 52.19) - open/close 48853 (time 1784049044.08 total 1762.70 last 64.69) - open/close 49512 (time 1784049054.08 total 1772.70 last 65.86) total: 50000 open/close in 1779.53 seconds: 28.10 ops/second - unlinked 0 (time 1784049065 ; total 0 ; last 0) - unlinked 10000 (time 1784049232 ; total 167 ; last 167) - unlinked 20000 (time 1784049394 ; total 329 ; last 162) - unlinked 30000 (time 1784049561 ; total 496 ; last 167) - unlinked 40000 (time 1784049732 ; total 667 ; last 171) total: 50000 unlinks in 843 seconds: 59.311981 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server unloading modules via unload_modules_local on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2