== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:45:02 (1783140302) start mds service on oleg353-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg353-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg353-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1783140379.82 total 10.31 last 21.81) - open/close 522 (time 1783140389.82 total 20.32 last 29.69) - open/close 709 (time 1783140399.83 total 30.32 last 18.69) - open/close 950 (time 1783140412.45 total 42.95 last 19.09) - open/close 1143 (time 1783140422.47 total 52.96 last 19.28) - open/close 1353 (time 1783140432.47 total 62.97 last 20.98) - open/close 1610 (time 1783140442.49 total 72.98 last 25.67) - open/close 1854 (time 1783140453.15 total 83.65 last 22.87) - open/close 2119 (time 1783140463.16 total 93.65 last 26.50) - open/close 2325 (time 1783140473.25 total 103.75 last 20.40) - open/close 2565 (time 1783140483.27 total 113.77 last 23.96) - open/close 2850 (time 1783140496.12 total 126.62 last 22.18) - open/close 3100 (time 1783140506.95 total 137.45 last 23.08) - open/close 3350 (time 1783140517.23 total 147.73 last 24.33) - open/close 3635 (time 1783140527.97 total 158.46 last 26.54) - open/close 3896 (time 1783140538.40 total 168.90 last 25.01) - open/close 4162 (time 1783140549.10 total 179.60 last 24.87) - open/close 4399 (time 1783140559.27 total 189.77 last 23.30) - open/close 4574 (time 1783140569.56 total 200.06 last 17.00) - open/close 4876 (time 1783140579.57 total 210.07 last 30.19) - open/close 5093 (time 1783140589.58 total 220.08 last 21.68) - open/close 5300 (time 1783140599.59 total 230.08 last 20.69) - open/close 5554 (time 1783140609.84 total 240.33 last 24.78) - open/close 5827 (time 1783140619.85 total 250.34 last 27.27) - open/close 5988 (time 1783140629.86 total 260.35 last 16.08) - open/close 6242 (time 1783140639.97 total 270.46 last 25.13) - open/close 6450 (time 1783140650.15 total 280.64 last 20.43) - open/close 6648 (time 1783140660.19 total 290.69 last 19.71) - open/close 6830 (time 1783140670.20 total 300.69 last 18.19) - open/close 7017 (time 1783140680.21 total 310.71 last 18.67) - open/close 7157 (time 1783140690.22 total 320.72 last 13.99) - open/close 7336 (time 1783140700.89 total 331.39 last 16.77) - open/close 7541 (time 1783140711.02 total 341.52 last 20.24) - open/close 7726 (time 1783140721.04 total 351.53 last 18.48) - open/close 7907 (time 1783140731.90 total 362.40 last 16.66) - open/close 8093 (time 1783140742.81 total 373.31 last 17.04) - open/close 8256 (time 1783140753.52 total 384.02 last 15.22) - open/close 8451 (time 1783140763.53 total 394.02 last 19.49) - open/close 8643 (time 1783140773.53 total 404.03 last 19.19) - open/close 8783 (time 1783140783.71 total 414.21 last 13.75) - open/close 8988 (time 1783140793.72 total 424.22 last 20.49) - open/close 9216 (time 1783140804.00 total 434.49 last 22.18) - open/close 9408 (time 1783140814.75 total 445.25 last 17.85) - open/close 9646 (time 1783140825.32 total 455.81 last 22.53) - open/close 9832 (time 1783140835.33 total 465.83 last 18.57) - open/close 10000 (time 1783140843.88 total 474.38 last 19.65) - open/close 10150 (time 1783140854.31 total 484.80 last 14.39) - open/close 10335 (time 1783140864.31 total 494.81 last 18.50) - open/close 10509 (time 1783140875.11 total 505.61 last 16.11) - open/close 10695 (time 1783140885.92 total 516.41 last 17.21) - open/close 10874 (time 1783140896.64 total 527.14 last 16.70) - open/close 11087 (time 1783140906.66 total 537.16 last 21.26) - open/close 11239 (time 1783140916.98 total 547.47 last 14.73) - open/close 11361 (time 1783140927.78 total 558.28 last 11.29) - open/close 11479 (time 1783140938.53 total 569.02 last 10.98) - open/close 11640 (time 1783140948.89 total 579.38 last 15.54) - open/close 11766 (time 1783140959.66 total 590.16 last 11.70) - open/close 11933 (time 1783140969.83 total 600.33 last 16.42) - open/close 12046 (time 1783140980.51 total 611.01 last 10.58) - open/close 12162 (time 1783140991.21 total 621.71 last 10.84) - open/close 12298 (time 1783141002.14 total 632.64 last 12.45) - open/close 12440 (time 1783141013.22 total 643.71 last 12.82) - open/close 12548 (time 1783141023.84 total 654.34 last 10.16) - open/close 12678 (time 1783141033.96 total 664.46 last 12.85) - open/close 12750 (time 1783141044.12 total 674.62 last 7.08) - open/close 12899 (time 1783141054.25 total 684.75 last 14.71) - open/close 12965 (time 1783141065.46 total 695.96 last 5.89) - open/close 13008 (time 1783141076.25 total 706.75 last 3.99) - open/close 13144 (time 1783141086.44 total 716.93 last 13.36) - open/close 14076 (time 1783141096.44 total 726.93 last 93.18) - open/close 15162 (time 1783141106.44 total 736.94 last 108.57) - open/close 16237 (time 1783141116.44 total 746.94 last 107.46) - open/close 17293 (time 1783141126.45 total 756.95 last 105.48) - open/close 18347 (time 1783141136.45 total 766.95 last 105.39) - open/close 19513 (time 1783141146.46 total 776.96 last 116.49) - open/close 20000 (time 1783141150.86 total 781.35 last 110.82) - open/close 21113 (time 1783141160.86 total 791.36 last 111.28) - open/close 22340 (time 1783141170.87 total 801.37 last 122.60) - open/close 23589 (time 1783141180.87 total 811.37 last 124.90) - open/close 24663 (time 1783141190.87 total 821.37 last 107.36) - open/close 25776 (time 1783141200.88 total 831.37 last 111.25) - open/close 26890 (time 1783141210.88 total 841.38 last 111.33) - open/close 28233 (time 1783141220.89 total 851.39 last 134.21) - open/close 29539 (time 1783141230.89 total 861.39 last 130.56) - open/close 30000 (time 1783141235.23 total 865.72 last 106.35) - open/close 31209 (time 1783141245.23 total 875.73 last 120.81) - open/close 32335 (time 1783141255.24 total 885.74 last 112.54) - open/close 33648 (time 1783141265.25 total 895.74 last 131.19) - open/close 35147 (time 1783141275.25 total 905.75 last 149.88) - open/close 36611 (time 1783141285.26 total 915.75 last 146.31) - open/close 38012 (time 1783141295.26 total 925.76 last 140.05) - open/close 39316 (time 1783141305.26 total 935.76 last 130.36) - open/close 40000 (time 1783141310.37 total 940.87 last 133.86) - open/close 41590 (time 1783141320.38 total 950.87 last 158.93) - open/close 42936 (time 1783141330.38 total 960.88 last 134.53) - open/close 44327 (time 1783141340.38 total 970.88 last 139.09) - open/close 45622 (time 1783141350.39 total 980.89 last 129.42) - open/close 46941 (time 1783141360.40 total 990.89 last 131.83) - open/close 48423 (time 1783141370.40 total 1000.90 last 148.09) - open/close 49940 (time 1783141380.41 total 1010.91 last 151.56) total: 50000 open/close in 1011.32 seconds: 49.44 ops/second - unlinked 0 (time 1783141382 ; total 0 ; last 0) - unlinked 10000 (time 1783141473 ; total 91 ; last 91) - unlinked 20000 (time 1783141573 ; total 191 ; last 100) - unlinked 30000 (time 1783141658 ; total 276 ; last 85) - unlinked 40000 (time 1783141743 ; total 361 ; last 85) total: 50000 unlinks in 449 seconds: 111.358574 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server unloading modules via unload_modules_local on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local oleg353-server: modules unloaded. pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2