== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:22:56 (1787368976) start mds service on oleg443-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg443-server: detected 4 online CPUs by sysfs oleg443-server: MODOPTS_LIBCFS= oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg443-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg443-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 236 (time 1787369055.38 total 10.72 last 22.01) - open/close 442 (time 1787369066.32 total 21.66 last 18.84) - open/close 645 (time 1787369076.32 total 31.67 last 20.29) - open/close 739 (time 1787369086.35 total 41.70 last 9.37) - open/close 1063 (time 1787369096.38 total 51.72 last 32.33) - open/close 1284 (time 1787369108.06 total 63.41 last 18.91) - open/close 1560 (time 1787369118.07 total 73.41 last 27.58) - open/close 1838 (time 1787369129.00 total 84.35 last 25.43) - open/close 2011 (time 1787369140.00 total 95.35 last 15.72) - open/close 2237 (time 1787369150.81 total 106.16 last 20.91) - open/close 2471 (time 1787369160.93 total 116.28 last 23.12) - open/close 2630 (time 1787369170.94 total 126.29 last 15.88) - open/close 2859 (time 1787369181.60 total 136.94 last 21.49) - open/close 3001 (time 1787369191.60 total 146.95 last 14.19) - open/close 3088 (time 1787369203.68 total 159.02 last 7.21) - open/close 3231 (time 1787369213.68 total 169.03 last 14.29) - open/close 3425 (time 1787369223.70 total 179.04 last 19.38) - open/close 3650 (time 1787369234.13 total 189.48 last 21.56) - open/close 3902 (time 1787369244.14 total 199.48 last 25.18) - open/close 4086 (time 1787369254.94 total 210.28 last 17.04) - open/close 4349 (time 1787369264.95 total 220.30 last 26.27) - open/close 4588 (time 1787369274.95 total 230.30 last 23.90) - open/close 4765 (time 1787369284.97 total 240.32 last 17.67) - open/close 4951 (time 1787369294.98 total 250.33 last 18.58) - open/close 5066 (time 1787369306.00 total 261.35 last 10.44) - open/close 5335 (time 1787369316.04 total 271.38 last 26.80) - open/close 5546 (time 1787369327.09 total 282.43 last 19.10) - open/close 5783 (time 1787369338.00 total 293.35 last 21.72) - open/close 5946 (time 1787369349.24 total 304.58 last 14.51) - open/close 6073 (time 1787369360.24 total 315.58 last 11.55) - open/close 6278 (time 1787369370.60 total 325.95 last 19.78) - open/close 6522 (time 1787369380.62 total 335.96 last 24.37) - open/close 6671 (time 1787369391.33 total 346.68 last 13.91) - open/close 6831 (time 1787369401.91 total 357.26 last 15.12) - open/close 7054 (time 1787369412.35 total 367.70 last 21.36) - open/close 7242 (time 1787369422.42 total 377.76 last 18.67) - open/close 7430 (time 1787369432.42 total 387.77 last 18.79) - open/close 7543 (time 1787369443.48 total 398.82 last 10.22) - open/close 7681 (time 1787369453.51 total 408.86 last 13.75) - open/close 7786 (time 1787369463.52 total 418.86 last 10.49) - open/close 7981 (time 1787369476.19 total 431.53 last 15.39) - open/close 8213 (time 1787369486.49 total 441.83 last 22.53) - open/close 8351 (time 1787369499.62 total 454.96 last 10.51) - open/close 8606 (time 1787369509.62 total 464.97 last 25.50) - open/close 8846 (time 1787369519.65 total 475.00 last 23.92) - open/close 9068 (time 1787369530.64 total 485.98 last 20.21) - open/close 9284 (time 1787369540.64 total 495.99 last 21.59) - open/close 9528 (time 1787369550.65 total 505.99 last 24.39) - open/close 9586 (time 1787369560.65 total 516.00 last 5.80) - open/close 9659 (time 1787369570.90 total 526.25 last 7.12) - open/close 9846 (time 1787369581.78 total 537.13 last 17.18) - open/close 10000 (time 1787369587.59 total 542.94 last 26.51) - open/close 10110 (time 1787369597.96 total 553.31 last 10.61) - open/close 10287 (time 1787369607.97 total 563.31 last 17.69) - open/close 10437 (time 1787369617.97 total 573.31 last 14.99) - open/close 10652 (time 1787369629.08 total 584.43 last 19.35) - open/close 10890 (time 1787369639.09 total 594.44 last 23.77) - open/close 11098 (time 1787369649.12 total 604.46 last 20.75) - open/close 11301 (time 1787369659.48 total 614.82 last 19.60) - open/close 11507 (time 1787369669.49 total 624.83 last 20.58) - open/close 11737 (time 1787369680.40 total 635.75 last 21.07) - open/close 11956 (time 1787369690.45 total 645.79 last 21.80) - open/close 12099 (time 1787369701.39 total 656.74 last 13.07) - open/close 12297 (time 1787369711.90 total 667.25 last 18.84) - open/close 12504 (time 1787369721.90 total 677.25 last 20.70) - open/close 12647 (time 1787369732.39 total 687.73 last 13.64) - open/close 12779 (time 1787369742.40 total 697.74 last 13.19) - open/close 12953 (time 1787369752.41 total 707.75 last 17.39) - open/close 13239 (time 1787369762.44 total 717.79 last 28.49) - open/close 13425 (time 1787369773.08 total 728.42 last 17.49) - open/close 13559 (time 1787369783.10 total 738.44 last 13.38) - open/close 13615 (time 1787369793.10 total 748.44 last 5.60) - open/close 13786 (time 1787369803.99 total 759.34 last 15.69) - open/close 14068 (time 1787369814.71 total 770.06 last 26.31) - open/close 14204 (time 1787369824.99 total 780.33 last 13.23) - open/close 14334 (time 1787369835.94 total 791.29 last 11.87) - open/close 14411 (time 1787369845.96 total 801.31 last 7.69) - open/close 14589 (time 1787369856.60 total 811.94 last 16.73) - open/close 14788 (time 1787369866.84 total 822.19 last 19.43) - open/close 14954 (time 1787369876.85 total 832.19 last 16.60) - open/close 15112 (time 1787369887.57 total 842.91 last 14.74) - open/close 15320 (time 1787369898.38 total 853.72 last 19.24) - open/close 15464 (time 1787369908.40 total 863.74 last 14.37) - open/close 15591 (time 1787369919.25 total 874.59 last 11.71) - open/close 15624 (time 1787369929.44 total 884.78 last 3.24) - open/close 15780 (time 1787369939.47 total 894.82 last 15.54) - open/close 15960 (time 1787369952.04 total 907.39 last 14.32) - open/close 16161 (time 1787369962.58 total 917.93 last 19.07) - open/close 16340 (time 1787369972.59 total 927.94 last 17.88) - open/close 16457 (time 1787369982.94 total 938.28 last 11.31) - open/close 16570 (time 1787369993.25 total 948.60 last 10.95) - open/close 16685 (time 1787370003.27 total 958.61 last 11.48) - open/close 16785 (time 1787370013.98 total 969.32 last 9.34) - open/close 16903 (time 1787370024.59 total 979.94 last 11.11) - open/close 17057 (time 1787370034.59 total 989.94 last 15.40) - open/close 17191 (time 1787370044.62 total 999.96 last 13.36) - open/close 17315 (time 1787370054.63 total 1009.97 last 12.39) - open/close 17410 (time 1787370065.37 total 1020.72 last 8.84) - open/close 17517 (time 1787370075.67 total 1031.02 last 10.38) - open/close 17596 (time 1787370086.03 total 1041.38 last 7.63) - open/close 17691 (time 1787370096.59 total 1051.93 last 9.00) - open/close 18195 (time 1787370106.59 total 1061.93 last 50.40) - open/close 18885 (time 1787370116.59 total 1071.94 last 68.97) - open/close 19677 (time 1787370126.60 total 1081.94 last 79.18) - open/close 20000 (time 1787370131.14 total 1086.49 last 71.04) - open/close 20756 (time 1787370141.14 total 1096.49 last 75.60) - open/close 21598 (time 1787370151.14 total 1106.49 last 84.18) - open/close 22315 (time 1787370161.15 total 1116.49 last 71.67) - open/close 23087 (time 1787370171.15 total 1126.50 last 77.17) - open/close 23847 (time 1787370181.16 total 1136.51 last 75.91) - open/close 24543 (time 1787370191.17 total 1146.51 last 69.57) - open/close 25236 (time 1787370201.17 total 1156.52 last 69.26) - open/close 25924 (time 1787370211.19 total 1166.53 last 68.70) - open/close 26611 (time 1787370221.20 total 1176.55 last 68.62) - open/close 27286 (time 1787370231.20 total 1186.55 last 67.49) - open/close 28058 (time 1787370241.21 total 1196.55 last 77.15) - open/close 28750 (time 1787370251.21 total 1206.55 last 69.19) - open/close 29439 (time 1787370261.22 total 1216.56 last 68.85) - open/close 30000 (time 1787370270.31 total 1225.65 last 61.71) - open/close 30561 (time 1787370280.31 total 1235.66 last 56.06) - open/close 31149 (time 1787370290.32 total 1245.66 last 58.78) - open/close 31787 (time 1787370300.35 total 1255.69 last 63.61) - open/close 32410 (time 1787370310.36 total 1265.70 last 62.24) - open/close 33053 (time 1787370320.37 total 1275.71 last 64.21) - open/close 33695 (time 1787370330.38 total 1285.72 last 64.15) - open/close 34376 (time 1787370340.38 total 1295.72 last 68.10) - open/close 35068 (time 1787370350.38 total 1305.73 last 69.17) - open/close 35683 (time 1787370360.39 total 1315.73 last 61.46) - open/close 36276 (time 1787370370.39 total 1325.73 last 59.29) - open/close 36855 (time 1787370380.41 total 1335.75 last 57.79) - open/close 37272 (time 1787370390.41 total 1345.76 last 41.67) - open/close 37777 (time 1787370400.44 total 1355.78 last 50.40) - open/close 38344 (time 1787370410.45 total 1365.79 last 56.63) - open/close 38835 (time 1787370420.45 total 1375.79 last 49.10) - open/close 39527 (time 1787370430.46 total 1385.80 last 69.12) - open/close 40000 (time 1787370437.42 total 1392.76 last 67.98) - open/close 40609 (time 1787370447.43 total 1402.77 last 60.84) - open/close 41272 (time 1787370457.44 total 1412.78 last 66.23) - open/close 42068 (time 1787370467.44 total 1422.79 last 79.55) - open/close 42644 (time 1787370477.45 total 1432.79 last 57.57) - open/close 43308 (time 1787370487.46 total 1442.81 last 66.32) - open/close 44079 (time 1787370497.47 total 1452.81 last 77.04) - open/close 44724 (time 1787370507.47 total 1462.82 last 64.48) - open/close 45488 (time 1787370517.49 total 1472.84 last 76.26) - open/close 46199 (time 1787370527.50 total 1482.84 last 71.06) - open/close 46996 (time 1787370537.50 total 1492.85 last 79.63) - open/close 47704 (time 1787370547.51 total 1502.86 last 70.73) - open/close 48506 (time 1787370557.53 total 1512.88 last 80.07) - open/close 49226 (time 1787370567.54 total 1522.88 last 71.94) - open/close 49919 (time 1787370577.54 total 1532.89 last 69.27) total: 50000 open/close in 1534.05 seconds: 32.59 ops/second - unlinked 0 (time 1787370583 ; total 0 ; last 0) - unlinked 10000 (time 1787370724 ; total 141 ; last 141) - unlinked 20000 (time 1787370875 ; total 292 ; last 151) - unlinked 30000 (time 1787371023 ; total 440 ; last 148) - unlinked 40000 (time 1787371173 ; total 590 ; last 150) total: 50000 unlinks in 725 seconds: 68.965515 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server unloading modules via unload_modules_local on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local oleg443-server: modules unloaded. pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2