== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:39:33 (1787737173) start mds service on oleg446-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing load_modules_local oleg446-server: Loading modules from /home/green/git/lustre-release/lustre oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg446-server: detected 4 online CPUs by sysfs oleg446-server: MODOPTS_LIBCFS= oleg446-server: Force libcfs to create 2 CPU partitions oleg446-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg446-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg446-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg446-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg446-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Started lustre-OST0000 oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 247 (time 1787737303.86 total 10.72 last 23.04) - open/close 387 (time 1787737314.58 total 21.44 last 13.06) - open/close 463 (time 1787737329.04 total 35.90 last 5.26) - open/close 646 (time 1787737340.10 total 46.96 last 16.54) - open/close 886 (time 1787737350.11 total 56.97 last 23.98) - open/close 1067 (time 1787737360.51 total 67.37 last 17.40) - open/close 1227 (time 1787737370.52 total 77.38 last 15.99) - open/close 1314 (time 1787737380.57 total 87.43 last 8.65) - open/close 1520 (time 1787737390.59 total 97.45 last 20.56) - open/close 1739 (time 1787737400.61 total 107.47 last 21.86) - open/close 2015 (time 1787737410.62 total 117.48 last 27.56) - open/close 2267 (time 1787737420.89 total 127.75 last 24.54) - open/close 2502 (time 1787737430.91 total 137.77 last 23.46) - open/close 2684 (time 1787737440.92 total 147.78 last 18.17) - open/close 2890 (time 1787737450.93 total 157.79 last 20.58) - open/close 3013 (time 1787737461.72 total 168.58 last 11.40) - open/close 3242 (time 1787737471.74 total 178.60 last 22.87) - open/close 3448 (time 1787737484.67 total 191.53 last 15.92) - open/close 3601 (time 1787737495.05 total 201.91 last 14.74) - open/close 3728 (time 1787737505.06 total 211.92 last 12.69) - open/close 3982 (time 1787737515.70 total 222.56 last 23.87) - open/close 4210 (time 1787737526.08 total 232.94 last 21.98) - open/close 4411 (time 1787737536.60 total 243.46 last 19.11) - open/close 4529 (time 1787737548.76 total 255.62 last 9.70) - open/close 4589 (time 1787737558.86 total 265.72 last 5.94) - open/close 4693 (time 1787737569.80 total 276.66 last 9.50) - open/close 4818 (time 1787737580.80 total 287.66 last 11.37) - open/close 4968 (time 1787737591.11 total 297.97 last 14.55) - open/close 5133 (time 1787737601.73 total 308.59 last 15.54) - open/close 5331 (time 1787737611.80 total 318.66 last 19.67) - open/close 5483 (time 1787737622.85 total 329.71 last 13.76) - open/close 5718 (time 1787737633.78 total 340.64 last 21.50) - open/close 5920 (time 1787737643.79 total 350.65 last 20.18) - open/close 6098 (time 1787737654.46 total 361.32 last 16.69) - open/close 6330 (time 1787737664.48 total 371.34 last 23.14) - open/close 6549 (time 1787737675.79 total 382.65 last 19.37) - open/close 6803 (time 1787737685.80 total 392.66 last 25.37) - open/close 7006 (time 1787737696.06 total 402.92 last 19.79) - open/close 7261 (time 1787737706.89 total 413.74 last 23.56) - open/close 7506 (time 1787737717.85 total 424.71 last 22.35) - open/close 7654 (time 1787737727.87 total 434.73 last 14.76) - open/close 7778 (time 1787737738.64 total 445.50 last 11.51) - open/close 7947 (time 1787737749.12 total 455.98 last 16.13) - open/close 8029 (time 1787737759.76 total 466.62 last 7.71) - open/close 8331 (time 1787737769.78 total 476.64 last 30.15) - open/close 8447 (time 1787737779.79 total 486.65 last 11.58) - open/close 8672 (time 1787737790.53 total 497.39 last 20.96) - open/close 8919 (time 1787737800.53 total 507.39 last 24.69) - open/close 9171 (time 1787737810.93 total 517.79 last 24.22) - open/close 9399 (time 1787737820.95 total 527.81 last 22.77) - open/close 9601 (time 1787737831.74 total 538.60 last 18.72) - open/close 9812 (time 1787737841.86 total 548.72 last 20.84) - open/close 10000 (time 1787737851.29 total 558.15 last 19.94) - open/close 10229 (time 1787737861.30 total 568.16 last 22.88) - open/close 10471 (time 1787737871.31 total 578.17 last 24.17) - open/close 10569 (time 1787737884.15 total 591.01 last 7.63) - open/close 10697 (time 1787737894.28 total 601.14 last 12.63) - open/close 10819 (time 1787737905.10 total 611.96 last 11.28) - open/close 10951 (time 1787737916.04 total 622.90 last 12.07) - open/close 11191 (time 1787737926.81 total 633.67 last 22.27) - open/close 11427 (time 1787737937.28 total 644.14 last 22.54) - open/close 11630 (time 1787737947.30 total 654.16 last 20.26) - open/close 11878 (time 1787737957.65 total 664.51 last 23.97) - open/close 12143 (time 1787737967.66 total 674.52 last 26.47) - open/close 12365 (time 1787737977.73 total 684.59 last 22.04) - open/close 12616 (time 1787737987.77 total 694.63 last 25.00) - open/close 12908 (time 1787737998.59 total 705.45 last 27.00) - open/close 13161 (time 1787738009.09 total 715.95 last 24.09) - open/close 13420 (time 1787738019.57 total 726.43 last 24.71) - open/close 13682 (time 1787738030.15 total 737.01 last 24.77) - open/close 13945 (time 1787738040.16 total 747.01 last 26.29) - open/close 14124 (time 1787738050.16 total 757.02 last 17.89) - open/close 14326 (time 1787738061.11 total 767.97 last 18.44) - open/close 14605 (time 1787738071.15 total 778.01 last 27.81) - open/close 14896 (time 1787738083.60 total 790.46 last 23.36) - open/close 15096 (time 1787738094.21 total 801.07 last 18.86) - open/close 15254 (time 1787738105.27 total 812.13 last 14.28) - open/close 15513 (time 1787738116.09 total 822.95 last 23.94) - open/close 15710 (time 1787738126.10 total 832.96 last 19.68) - open/close 15999 (time 1787738137.02 total 843.87 last 26.48) - open/close 16249 (time 1787738147.02 total 853.88 last 25.00) - open/close 16359 (time 1787738157.45 total 864.31 last 10.54) - open/close 16501 (time 1787738167.96 total 874.82 last 13.52) - open/close 16703 (time 1787738177.97 total 884.83 last 20.17) - open/close 16932 (time 1787738187.99 total 894.85 last 22.86) - open/close 17103 (time 1787738198.15 total 905.01 last 16.83) - open/close 17357 (time 1787738209.72 total 916.58 last 21.95) - open/close 17436 (time 1787738219.73 total 926.59 last 7.90) - open/close 17688 (time 1787738229.76 total 936.62 last 25.13) - open/close 17900 (time 1787738239.79 total 946.65 last 21.14) - open/close 18187 (time 1787738249.80 total 956.66 last 28.65) - open/close 18419 (time 1787738260.25 total 967.11 last 22.20) - open/close 18673 (time 1787738270.67 total 977.53 last 24.38) - open/close 18984 (time 1787738280.69 total 987.55 last 31.04) - open/close 19523 (time 1787738290.71 total 997.57 last 53.79) - open/close 20000 (time 1787738296.63 total 1003.49 last 80.59) - open/close 20794 (time 1787738306.63 total 1013.49 last 79.40) - open/close 21465 (time 1787738316.64 total 1023.50 last 67.04) - open/close 22139 (time 1787738326.64 total 1033.50 last 67.36) - open/close 22982 (time 1787738336.65 total 1043.50 last 84.29) - open/close 23914 (time 1787738346.67 total 1053.53 last 92.96) - open/close 24683 (time 1787738356.67 total 1063.53 last 76.88) - open/close 25428 (time 1787738366.68 total 1073.54 last 74.47) - open/close 26125 (time 1787738376.68 total 1083.54 last 69.67) - open/close 26928 (time 1787738386.68 total 1093.54 last 80.29) - open/close 27736 (time 1787738396.69 total 1103.55 last 80.74) - open/close 28510 (time 1787738406.69 total 1113.55 last 77.40) - open/close 29284 (time 1787738416.70 total 1123.56 last 77.32) - open/close 30000 (time 1787738426.22 total 1133.08 last 75.19) - open/close 30848 (time 1787738436.24 total 1143.10 last 84.69) - open/close 31595 (time 1787738446.24 total 1153.10 last 74.70) - open/close 32422 (time 1787738456.24 total 1163.10 last 82.66) - open/close 33289 (time 1787738466.25 total 1173.11 last 86.65) - open/close 34131 (time 1787738476.25 total 1183.11 last 84.15) - open/close 34949 (time 1787738486.26 total 1193.12 last 81.74) - open/close 35803 (time 1787738496.27 total 1203.13 last 85.33) - open/close 36654 (time 1787738506.27 total 1213.13 last 85.08) - open/close 37389 (time 1787738516.28 total 1223.14 last 73.44) - open/close 37931 (time 1787738526.30 total 1233.16 last 54.10) - open/close 38533 (time 1787738536.30 total 1243.16 last 60.19) - open/close 39295 (time 1787738546.33 total 1253.19 last 75.99) - open/close 40000 (time 1787738555.82 total 1262.68 last 74.26) - open/close 40718 (time 1787738565.82 total 1272.68 last 71.79) - open/close 41325 (time 1787738575.83 total 1282.69 last 60.65) - open/close 41889 (time 1787738585.85 total 1292.71 last 56.32) - open/close 42557 (time 1787738595.85 total 1302.71 last 66.75) - open/close 43341 (time 1787738605.86 total 1312.72 last 78.32) - open/close 44119 (time 1787738615.88 total 1322.74 last 77.71) - open/close 44821 (time 1787738625.88 total 1332.74 last 70.19) - open/close 45643 (time 1787738635.88 total 1342.74 last 82.16) - open/close 46425 (time 1787738645.89 total 1352.75 last 78.16) - open/close 47059 (time 1787738655.89 total 1362.75 last 63.40) - open/close 47878 (time 1787738665.89 total 1372.75 last 81.88) - open/close 48618 (time 1787738675.89 total 1382.75 last 73.99) - open/close 49312 (time 1787738685.90 total 1392.76 last 69.35) total: 50000 open/close in 1401.75 seconds: 35.67 ops/second - unlinked 0 (time 1787738699 ; total 0 ; last 0) - unlinked 10000 (time 1787738835 ; total 136 ; last 136) - unlinked 20000 (time 1787738971 ; total 272 ; last 136) - unlinked 30000 (time 1787739113 ; total 414 ; last 142) - unlinked 40000 (time 1787739254 ; total 555 ; last 141) total: 50000 unlinks in 690 seconds: 72.463768 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg446-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server stop mds service on oleg446-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server stop mds service on oleg446-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg446-server unloading modules via unload_modules_local on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing unload_modules_local oleg446-server: modules unloaded. pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2