== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:49:50 (1787863790) start mds service on oleg142-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg142-server: detected 4 online CPUs by sysfs oleg142-server: MODOPTS_LIBCFS= oleg142-server: Force libcfs to create 2 CPU partitions oleg142-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg142-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg142-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg142-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 209 (time 1787863888.57 total 10.00 last 20.90) - open/close 365 (time 1787863899.96 total 21.40 last 13.68) - open/close 539 (time 1787863909.97 total 31.41 last 17.39) - open/close 730 (time 1787863920.27 total 41.71 last 18.54) - open/close 899 (time 1787863931.21 total 52.65 last 15.45) - open/close 1064 (time 1787863941.22 total 62.66 last 16.48) - open/close 1274 (time 1787863951.25 total 72.68 last 20.96) - open/close 1345 (time 1787863963.23 total 84.67 last 5.92) - open/close 1553 (time 1787863975.11 total 96.55 last 17.51) - open/close 1700 (time 1787863985.87 total 107.30 last 13.67) - open/close 1873 (time 1787863996.86 total 118.30 last 15.74) - open/close 1971 (time 1787864007.68 total 129.12 last 9.05) - open/close 2039 (time 1787864017.89 total 139.33 last 6.66) - open/close 2103 (time 1787864028.17 total 149.60 last 6.23) - open/close 2304 (time 1787864038.79 total 160.23 last 18.92) - open/close 2533 (time 1787864048.79 total 170.23 last 22.90) - open/close 2757 (time 1787864060.48 total 181.92 last 19.16) - open/close 2961 (time 1787864071.14 total 192.58 last 19.14) - open/close 3151 (time 1787864081.15 total 202.59 last 18.98) - open/close 3346 (time 1787864091.16 total 212.59 last 19.49) - open/close 3576 (time 1787864101.16 total 222.60 last 23.00) - open/close 3747 (time 1787864113.93 total 235.37 last 13.39) - open/close 3913 (time 1787864123.94 total 245.38 last 16.59) - open/close 4003 (time 1787864135.19 total 256.63 last 8.00) - open/close 4177 (time 1787864145.87 total 267.30 last 16.30) - open/close 4331 (time 1787864156.50 total 277.94 last 14.48) - open/close 4393 (time 1787864167.52 total 288.96 last 5.63) - open/close 4526 (time 1787864178.07 total 299.50 last 12.61) - open/close 4619 (time 1787864188.79 total 310.23 last 8.67) - open/close 4742 (time 1787864198.80 total 320.24 last 12.29) - open/close 4851 (time 1787864209.73 total 331.16 last 9.98) - open/close 4979 (time 1787864220.75 total 342.19 last 11.61) - open/close 5056 (time 1787864230.77 total 352.21 last 7.68) - open/close 5142 (time 1787864241.40 total 362.84 last 8.09) - open/close 5252 (time 1787864251.42 total 372.85 last 10.98) - open/close 5416 (time 1787864261.43 total 382.86 last 16.39) - open/close 5513 (time 1787864272.26 total 393.70 last 8.95) - open/close 5651 (time 1787864282.72 total 404.16 last 13.19) - open/close 5762 (time 1787864293.59 total 415.02 last 10.22) - open/close 5930 (time 1787864304.71 total 426.15 last 15.10) - open/close 6063 (time 1787864317.33 total 438.76 last 10.54) - open/close 6178 (time 1787864327.87 total 449.30 last 10.91) - open/close 6286 (time 1787864338.36 total 459.80 last 10.29) - open/close 6460 (time 1787864348.39 total 469.82 last 17.35) - open/close 6530 (time 1787864358.60 total 480.04 last 6.85) - open/close 6694 (time 1787864368.61 total 490.05 last 16.38) - open/close 6829 (time 1787864378.63 total 500.07 last 13.47) - open/close 7050 (time 1787864388.65 total 510.09 last 22.05) - open/close 7249 (time 1787864399.10 total 520.54 last 19.05) - open/close 7457 (time 1787864410.12 total 531.56 last 18.88) - open/close 7754 (time 1787864420.12 total 541.56 last 29.69) - open/close 7963 (time 1787864430.47 total 551.90 last 20.21) - open/close 8158 (time 1787864440.47 total 561.90 last 19.50) - open/close 8328 (time 1787864450.47 total 571.91 last 16.99) - open/close 8522 (time 1787864461.37 total 582.81 last 17.80) - open/close 8674 (time 1787864472.33 total 593.77 last 13.87) - open/close 8775 (time 1787864482.35 total 603.79 last 10.08) - open/close 8873 (time 1787864493.09 total 614.52 last 9.13) - open/close 9148 (time 1787864503.88 total 625.31 last 25.49) - open/close 9390 (time 1787864514.18 total 635.62 last 23.49) - open/close 9524 (time 1787864524.49 total 645.93 last 12.99) - open/close 9597 (time 1787864534.66 total 656.10 last 7.18) - open/close 9720 (time 1787864544.68 total 666.11 last 12.29) - open/close 9878 (time 1787864554.69 total 676.13 last 15.77) - open/close 10000 (time 1787864559.08 total 680.52 last 27.79) - open/close 10183 (time 1787864569.85 total 691.29 last 16.99) - open/close 10286 (time 1787864580.74 total 702.17 last 9.46) - open/close 10396 (time 1787864591.07 total 712.51 last 10.64) - open/close 10478 (time 1787864601.16 total 722.59 last 8.13) - open/close 10697 (time 1787864616.93 total 738.37 last 13.88) - open/close 10850 (time 1787864628.12 total 749.55 last 13.68) - open/close 11027 (time 1787864638.15 total 759.58 last 17.65) - open/close 11269 (time 1787864648.15 total 769.59 last 24.18) - open/close 11487 (time 1787864658.18 total 779.61 last 21.75) - open/close 11558 (time 1787864668.19 total 789.63 last 7.09) - open/close 11750 (time 1787864678.82 total 800.26 last 18.07) - open/close 12011 (time 1787864688.83 total 810.27 last 26.07) - open/close 12128 (time 1787864700.69 total 822.12 last 9.87) - open/close 12342 (time 1787864711.68 total 833.12 last 19.47) - open/close 12528 (time 1787864722.70 total 844.13 last 16.88) - open/close 12761 (time 1787864732.72 total 854.15 last 23.25) - open/close 12944 (time 1787864742.79 total 864.22 last 18.18) - open/close 12980 (time 1787864752.79 total 874.23 last 3.60) - open/close 13155 (time 1787864763.47 total 884.90 last 16.39) - open/close 13345 (time 1787864774.41 total 895.85 last 17.35) - open/close 13488 (time 1787864784.49 total 905.93 last 14.19) - open/close 13579 (time 1787864795.52 total 916.96 last 8.25) - open/close 13633 (time 1787864805.95 total 927.38 last 5.18) - open/close 13733 (time 1787864816.59 total 938.02 last 9.40) - open/close 13918 (time 1787864828.80 total 950.24 last 15.15) - open/close 14088 (time 1787864838.92 total 960.36 last 16.80) - open/close 14270 (time 1787864848.93 total 970.36 last 18.19) - open/close 14460 (time 1787864859.59 total 981.03 last 17.81) - open/close 14699 (time 1787864869.95 total 991.38 last 23.09) - open/close 14818 (time 1787864879.97 total 1001.41 last 11.87) - open/close 14975 (time 1787864891.11 total 1012.55 last 14.10) - open/close 15102 (time 1787864901.12 total 1022.56 last 12.69) - open/close 15301 (time 1787864911.78 total 1033.22 last 18.66) - open/close 15397 (time 1787864922.32 total 1043.76 last 9.11) - open/close 15568 (time 1787864932.67 total 1054.11 last 16.52) - open/close 15694 (time 1787864943.38 total 1064.81 last 11.77) - open/close 15857 (time 1787864953.47 total 1074.91 last 16.15) - open/close 16333 (time 1787864963.48 total 1084.92 last 47.55) - open/close 17046 (time 1787864973.48 total 1094.92 last 71.29) - open/close 17667 (time 1787864983.49 total 1104.92 last 62.07) - open/close 18306 (time 1787864993.49 total 1114.93 last 63.86) - open/close 18984 (time 1787865003.51 total 1124.95 last 67.70) - open/close 19589 (time 1787865013.52 total 1134.95 last 60.45) - open/close 20000 (time 1787865019.65 total 1141.09 last 66.99) - open/close 20651 (time 1787865029.67 total 1151.11 last 64.98) - open/close 21300 (time 1787865039.68 total 1161.12 last 64.82) - open/close 21979 (time 1787865049.68 total 1171.12 last 67.90) - open/close 22561 (time 1787865059.69 total 1181.13 last 58.17) - open/close 23159 (time 1787865069.71 total 1191.14 last 59.69) - open/close 23800 (time 1787865079.71 total 1201.15 last 64.10) - open/close 24385 (time 1787865089.72 total 1211.16 last 58.43) - open/close 25012 (time 1787865099.73 total 1221.16 last 62.67) - open/close 25598 (time 1787865109.73 total 1231.17 last 58.57) - open/close 26216 (time 1787865119.74 total 1241.17 last 61.76) - open/close 26721 (time 1787865129.76 total 1251.20 last 50.37) - open/close 27224 (time 1787865139.77 total 1261.20 last 50.29) - open/close 27873 (time 1787865149.77 total 1271.21 last 64.87) - open/close 28397 (time 1787865159.81 total 1281.24 last 52.20) - open/close 28954 (time 1787865169.81 total 1291.25 last 55.67) - open/close 29539 (time 1787865179.82 total 1301.25 last 58.49) - open/close 30000 (time 1787865186.98 total 1308.42 last 64.32) - open/close 30712 (time 1787865196.99 total 1318.43 last 71.15) - open/close 31314 (time 1787865207.01 total 1328.44 last 60.10) - open/close 32012 (time 1787865217.03 total 1338.46 last 69.65) - open/close 32594 (time 1787865227.03 total 1348.47 last 58.16) - open/close 33399 (time 1787865237.04 total 1358.47 last 80.48) - open/close 33988 (time 1787865247.04 total 1368.48 last 58.86) - open/close 34594 (time 1787865257.04 total 1378.48 last 60.59) - open/close 35217 (time 1787865267.05 total 1388.49 last 62.25) - open/close 35888 (time 1787865277.06 total 1398.50 last 67.04) - open/close 36573 (time 1787865287.07 total 1408.51 last 68.43) - open/close 37203 (time 1787865297.08 total 1418.51 last 62.96) - open/close 37793 (time 1787865307.08 total 1428.52 last 58.97) - open/close 38447 (time 1787865317.11 total 1438.54 last 65.24) - open/close 38947 (time 1787865327.11 total 1448.55 last 49.99) - open/close 39531 (time 1787865337.11 total 1458.55 last 58.39) - open/close 40000 (time 1787865345.91 total 1467.34 last 53.32) - open/close 40659 (time 1787865355.93 total 1477.37 last 65.72) - open/close 41283 (time 1787865365.94 total 1487.37 last 62.40) - open/close 41888 (time 1787865375.94 total 1497.38 last 60.45) - open/close 42511 (time 1787865385.95 total 1507.38 last 62.26) - open/close 43092 (time 1787865395.95 total 1517.39 last 58.08) - open/close 43791 (time 1787865405.95 total 1527.39 last 69.89) - open/close 44384 (time 1787865415.99 total 1537.42 last 59.11) - open/close 45050 (time 1787865425.99 total 1547.43 last 66.55) - open/close 45712 (time 1787865436.00 total 1557.44 last 66.16) - open/close 46253 (time 1787865446.01 total 1567.45 last 54.05) - open/close 46867 (time 1787865456.02 total 1577.46 last 61.32) - open/close 47455 (time 1787865466.03 total 1587.47 last 58.76) - open/close 48029 (time 1787865476.03 total 1597.47 last 57.38) - open/close 48649 (time 1787865486.03 total 1607.47 last 61.99) - open/close 49268 (time 1787865496.05 total 1617.49 last 61.78) - open/close 49963 (time 1787865506.06 total 1627.50 last 69.44) total: 50000 open/close in 1627.99 seconds: 30.71 ops/second - unlinked 0 (time 1787865511 ; total 0 ; last 0) - unlinked 10000 (time 1787865678 ; total 167 ; last 167) - unlinked 20000 (time 1787865838 ; total 327 ; last 160) - unlinked 30000 (time 1787865996 ; total 485 ; last 158) - unlinked 40000 (time 1787866154 ; total 643 ; last 158) total: 50000 unlinks in 798 seconds: 62.656643 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg142-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server unloading modules via unload_modules_local on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing unload_modules_local oleg142-server: modules unloaded. pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2