== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:40:26 (1781624426) start mds service on oleg139-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing load_modules_local oleg139-server: Loading modules from /home/green/git/lustre-release/lustre oleg139-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg139-server: detected 4 online CPUs by sysfs oleg139-server: MODOPTS_LIBCFS= oleg139-server: Force libcfs to create 2 CPU partitions oleg139-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg139-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg139-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg139-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg139-server: oleg139-server.virtnet: executing set_default_debug -1 all pdsh@oleg139-client: oleg139-server: ssh exited with exit code 1 Started lustre-OST0000 oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg139-client.virtnet: -o user_xattr,flock 192.168.201.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 119 (time 1781624488.49 total 10.50 last 11.33) - open/close 357 (time 1781624499.24 total 21.26 last 22.12) - open/close 514 (time 1781624509.25 total 31.27 last 15.69) - open/close 677 (time 1781624520.17 total 42.18 last 14.93) - open/close 853 (time 1781624530.18 total 52.19 last 17.59) - open/close 1060 (time 1781624540.59 total 62.60 last 19.88) - open/close 1182 (time 1781624551.34 total 73.35 last 11.35) - open/close 1517 (time 1781624561.51 total 83.53 last 32.93) - open/close 1841 (time 1781624571.51 total 93.53 last 32.39) - open/close 2067 (time 1781624581.81 total 103.82 last 21.95) - open/close 2325 (time 1781624592.67 total 114.69 last 23.75) - open/close 2608 (time 1781624603.36 total 125.38 last 26.48) - open/close 2924 (time 1781624613.73 total 135.75 last 30.46) - open/close 3236 (time 1781624624.11 total 146.12 last 30.08) - open/close 3503 (time 1781624634.11 total 156.12 last 26.69) - open/close 3761 (time 1781624644.51 total 166.53 last 24.79) - open/close 3972 (time 1781624654.62 total 176.64 last 20.88) - open/close 4220 (time 1781624665.25 total 187.26 last 23.34) - open/close 4363 (time 1781624675.26 total 197.27 last 14.29) - open/close 4626 (time 1781624685.86 total 207.88 last 24.80) - open/close 4953 (time 1781624696.48 total 218.50 last 30.79) - open/close 5194 (time 1781624706.98 total 229.00 last 22.95) - open/close 5466 (time 1781624716.98 total 239.00 last 27.20) - open/close 5647 (time 1781624727.08 total 249.09 last 17.92) - open/close 5946 (time 1781624738.98 total 261.00 last 25.13) - open/close 6225 (time 1781624748.98 total 271.00 last 27.89) - open/close 6548 (time 1781624758.99 total 281.00 last 32.29) - open/close 6858 (time 1781624769.96 total 291.97 last 28.26) - open/close 7090 (time 1781624780.00 total 302.02 last 23.09) - open/close 7404 (time 1781624790.00 total 312.02 last 31.39) - open/close 7579 (time 1781624800.99 total 323.01 last 15.93) - open/close 7836 (time 1781624811.75 total 333.76 last 23.89) - open/close 8108 (time 1781624821.79 total 343.81 last 27.09) - open/close 8422 (time 1781624831.80 total 353.81 last 31.37) - open/close 8575 (time 1781624841.80 total 363.82 last 15.30) - open/close 8798 (time 1781624852.46 total 374.47 last 20.92) - open/close 8915 (time 1781624863.42 total 385.43 last 10.68) - open/close 9053 (time 1781624873.47 total 395.49 last 13.72) - open/close 9251 (time 1781624884.51 total 406.53 last 17.93) - open/close 9386 (time 1781624894.62 total 416.64 last 13.35) - open/close 9560 (time 1781624905.21 total 427.22 last 16.44) - open/close 9695 (time 1781624915.27 total 437.29 last 13.41) - open/close 9832 (time 1781624925.82 total 447.84 last 12.99) - open/close 9957 (time 1781624936.77 total 458.78 last 11.42) - open/close 10000 (time 1781624941.85 total 463.86 last 8.47) - open/close 10171 (time 1781624951.86 total 473.87 last 17.09) - open/close 10309 (time 1781624961.89 total 483.90 last 13.76) - open/close 10549 (time 1781624971.90 total 493.91 last 23.98) - open/close 10785 (time 1781624982.77 total 504.79 last 21.70) - open/close 10889 (time 1781624994.27 total 516.29 last 9.04) - open/close 11147 (time 1781625004.72 total 526.73 last 24.71) - open/close 11340 (time 1781625014.72 total 536.74 last 19.29) - open/close 11494 (time 1781625025.13 total 547.15 last 14.79) - open/close 11619 (time 1781625036.05 total 558.07 last 11.45) - open/close 11828 (time 1781625046.38 total 568.39 last 20.25) - open/close 12016 (time 1781625056.39 total 578.40 last 18.78) - open/close 12181 (time 1781625066.40 total 588.42 last 16.48) - open/close 12409 (time 1781625076.42 total 598.43 last 22.77) - open/close 12549 (time 1781625089.62 total 611.64 last 10.60) - open/close 12745 (time 1781625100.41 total 622.42 last 18.18) - open/close 12982 (time 1781625110.66 total 632.68 last 23.12) - open/close 13477 (time 1781625120.67 total 642.68 last 49.46) - open/close 14192 (time 1781625130.67 total 652.69 last 71.47) - open/close 14911 (time 1781625140.69 total 662.71 last 71.77) - open/close 15615 (time 1781625150.69 total 672.71 last 70.37) - open/close 16215 (time 1781625160.70 total 682.71 last 59.98) - open/close 16885 (time 1781625170.71 total 692.72 last 66.93) - open/close 17575 (time 1781625180.72 total 702.73 last 68.95) - open/close 18287 (time 1781625190.72 total 712.73 last 71.19) - open/close 18899 (time 1781625200.72 total 722.73 last 61.19) - open/close 19581 (time 1781625210.73 total 732.74 last 68.15) - open/close 20000 (time 1781625216.55 total 738.56 last 71.98) - open/close 20730 (time 1781625226.55 total 748.57 last 72.94) - open/close 21463 (time 1781625236.56 total 758.58 last 73.24) - open/close 22196 (time 1781625246.56 total 768.58 last 73.28) - open/close 22862 (time 1781625256.58 total 778.59 last 66.53) - open/close 23629 (time 1781625266.59 total 788.61 last 76.56) - open/close 24389 (time 1781625276.60 total 798.61 last 75.96) - open/close 25139 (time 1781625286.60 total 808.61 last 75.00) - open/close 25836 (time 1781625296.60 total 818.62 last 69.68) - open/close 26589 (time 1781625306.61 total 828.63 last 75.24) - open/close 27279 (time 1781625316.62 total 838.64 last 68.91) - open/close 27975 (time 1781625326.63 total 848.64 last 69.57) - open/close 28585 (time 1781625336.63 total 858.64 last 61.00) - open/close 29295 (time 1781625346.65 total 868.66 last 70.87) - open/close 29957 (time 1781625356.66 total 878.67 last 66.14) - open/close 30684 (time 1781625366.68 total 888.69 last 72.55) - open/close 31411 (time 1781625376.68 total 898.69 last 72.68) - open/close 32148 (time 1781625386.68 total 908.70 last 73.65) - open/close 32848 (time 1781625396.69 total 918.70 last 69.98) - open/close 33620 (time 1781625406.70 total 928.71 last 77.13) - open/close 34377 (time 1781625416.71 total 938.72 last 75.63) - open/close 35105 (time 1781625426.71 total 948.72 last 72.80) - open/close 35893 (time 1781625436.71 total 958.72 last 78.80) - open/close 36747 (time 1781625446.72 total 968.73 last 85.34) - open/close 37501 (time 1781625456.73 total 978.75 last 75.28) - open/close 38246 (time 1781625466.74 total 988.76 last 74.40) - open/close 38977 (time 1781625476.75 total 998.76 last 73.07) - open/close 39803 (time 1781625486.76 total 1008.77 last 82.54) - open/close 40000 (time 1781625489.16 total 1011.17 last 82.08) - open/close 40741 (time 1781625499.17 total 1021.18 last 74.02) - open/close 41478 (time 1781625509.18 total 1031.19 last 73.63) - open/close 42182 (time 1781625519.18 total 1041.20 last 70.37) - open/close 42905 (time 1781625529.19 total 1051.21 last 72.20) - open/close 43690 (time 1781625539.20 total 1061.22 last 78.43) - open/close 44383 (time 1781625549.21 total 1071.23 last 69.23) - open/close 45132 (time 1781625559.23 total 1081.24 last 74.79) - open/close 45844 (time 1781625569.23 total 1091.24 last 71.20) - open/close 46620 (time 1781625579.24 total 1101.25 last 77.52) - open/close 47424 (time 1781625589.24 total 1111.26 last 80.35) - open/close 48207 (time 1781625599.25 total 1121.27 last 78.25) - open/close 48997 (time 1781625609.26 total 1131.27 last 78.96) - open/close 49773 (time 1781625619.26 total 1141.27 last 77.59) total: 50000 open/close in 1144.09 seconds: 43.70 ops/second - unlinked 0 (time 1781625625 ; total 0 ; last 0) - unlinked 10000 (time 1781625749 ; total 124 ; last 124) - unlinked 20000 (time 1781625877 ; total 252 ; last 128) - unlinked 30000 (time 1781625981 ; total 356 ; last 104) - unlinked 40000 (time 1781626078 ; total 453 ; last 97) total: 50000 unlinks in 512 seconds: 97.656250 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg139-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg139-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg139-server stop mds service on oleg139-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg139-server unloading modules via unload_modules_local on: 'oleg139-server' oleg139-server: oleg139-server.virtnet: executing unload_modules_local oleg139-server: modules unloaded. pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-client: ssh exited with exit code 2 pdsh@oleg139-client: oleg139-server: ssh exited with exit code 2