== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:22:36 (1783563756) start mds service on oleg440-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg440-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 353 (time 1783563797.54 total 10.88 last 32.44) - open/close 519 (time 1783563807.59 total 20.93 last 16.52) - open/close 694 (time 1783563818.21 total 31.56 last 16.47) - open/close 921 (time 1783563829.09 total 42.44 last 20.86) - open/close 1157 (time 1783563839.59 total 52.93 last 22.49) - open/close 1397 (time 1783563850.23 total 63.57 last 22.55) - open/close 1609 (time 1783563860.23 total 73.57 last 21.20) - open/close 1788 (time 1783563870.57 total 83.92 last 17.30) - open/close 1994 (time 1783563880.58 total 93.93 last 20.58) - open/close 2079 (time 1783563890.60 total 103.94 last 8.49) - open/close 2289 (time 1783563900.62 total 113.96 last 20.95) - open/close 2474 (time 1783563910.65 total 123.99 last 18.45) - open/close 2720 (time 1783563921.45 total 134.80 last 22.77) - open/close 2934 (time 1783563931.88 total 145.22 last 20.52) - open/close 3087 (time 1783563942.57 total 155.91 last 14.31) - open/close 3194 (time 1783563953.45 total 166.79 last 9.84) - open/close 3381 (time 1783563964.07 total 177.41 last 17.60) - open/close 3536 (time 1783563974.07 total 187.42 last 15.49) - open/close 3604 (time 1783563984.30 total 197.65 last 6.65) - open/close 3660 (time 1783563994.34 total 207.69 last 5.58) - open/close 3850 (time 1783564004.59 total 217.93 last 18.55) - open/close 4035 (time 1783564014.89 total 228.24 last 17.95) - open/close 4194 (time 1783564024.90 total 238.24 last 15.89) - open/close 4439 (time 1783564034.90 total 248.24 last 24.50) - open/close 4597 (time 1783564045.86 total 259.21 last 14.41) - open/close 4708 (time 1783564056.55 total 269.89 last 10.39) - open/close 4881 (time 1783564067.24 total 280.58 last 16.18) - open/close 5063 (time 1783564078.05 total 291.40 last 16.83) - open/close 5118 (time 1783564088.07 total 301.41 last 5.49) - open/close 5238 (time 1783564098.80 total 312.14 last 11.18) - open/close 5377 (time 1783564108.97 total 322.31 last 13.67) - open/close 5520 (time 1783564119.28 total 332.63 last 13.86) - open/close 5681 (time 1783564129.65 total 342.99 last 15.53) - open/close 5827 (time 1783564140.08 total 353.43 last 13.99) - open/close 5986 (time 1783564150.63 total 363.98 last 15.07) - open/close 6125 (time 1783564161.00 total 374.34 last 13.41) - open/close 6326 (time 1783564171.63 total 384.97 last 18.91) - open/close 6457 (time 1783564181.64 total 394.98 last 13.09) - open/close 6541 (time 1783564191.87 total 405.22 last 8.21) - open/close 6685 (time 1783564201.88 total 415.22 last 14.39) - open/close 6793 (time 1783564214.58 total 427.92 last 8.51) - open/close 7000 (time 1783564224.95 total 438.29 last 19.95) - open/close 7310 (time 1783564235.44 total 448.78 last 29.56) - open/close 7414 (time 1783564245.95 total 459.29 last 9.90) - open/close 7552 (time 1783564256.93 total 470.28 last 12.56) - open/close 7657 (time 1783564267.69 total 481.03 last 9.76) - open/close 7859 (time 1783564278.51 total 491.85 last 18.68) - open/close 7965 (time 1783564289.07 total 502.41 last 10.04) - open/close 8046 (time 1783564299.43 total 512.78 last 7.81) - open/close 8230 (time 1783564309.86 total 523.20 last 17.64) - open/close 8294 (time 1783564319.98 total 533.32 last 6.33) - open/close 8437 (time 1783564330.74 total 544.08 last 13.29) - open/close 8599 (time 1783564340.74 total 554.09 last 16.19) - open/close 8680 (time 1783564351.92 total 565.26 last 7.25) - open/close 9120 (time 1783564361.93 total 575.27 last 43.97) - open/close 10000 (time 1783564371.14 total 584.49 last 95.47) - open/close 11002 (time 1783564381.14 total 594.49 last 100.20) - open/close 11952 (time 1783564391.16 total 604.50 last 94.89) - open/close 12913 (time 1783564401.16 total 614.50 last 96.08) - open/close 13857 (time 1783564411.16 total 624.51 last 94.35) - open/close 14647 (time 1783564421.17 total 634.51 last 78.97) - open/close 15560 (time 1783564431.17 total 644.52 last 91.23) - open/close 16341 (time 1783564441.18 total 654.52 last 78.09) - open/close 17219 (time 1783564451.18 total 664.52 last 87.80) - open/close 18086 (time 1783564461.18 total 674.52 last 86.69) - open/close 18957 (time 1783564471.18 total 684.53 last 87.05) - open/close 19828 (time 1783564481.19 total 694.54 last 87.02) - open/close 20791 (time 1783564491.19 total 704.54 last 96.29) - open/close 21768 (time 1783564501.20 total 714.54 last 97.63) - open/close 22685 (time 1783564511.21 total 724.56 last 91.57) - open/close 23588 (time 1783564521.23 total 734.57 last 90.17) - open/close 24515 (time 1783564531.23 total 744.57 last 92.68) - open/close 25461 (time 1783564541.24 total 754.58 last 94.50) - open/close 26355 (time 1783564551.25 total 764.59 last 89.32) - open/close 27200 (time 1783564561.25 total 774.59 last 84.50) - open/close 28210 (time 1783564571.26 total 784.60 last 100.91) - open/close 29190 (time 1783564581.26 total 794.61 last 97.94) - open/close 30000 (time 1783564589.83 total 803.17 last 94.57) - open/close 30932 (time 1783564599.84 total 813.18 last 93.11) - open/close 31798 (time 1783564609.84 total 823.19 last 86.55) - open/close 32742 (time 1783564619.85 total 833.19 last 94.35) - open/close 33603 (time 1783564629.86 total 843.21 last 86.00) - open/close 34444 (time 1783564639.87 total 853.21 last 84.02) - open/close 35262 (time 1783564649.87 total 863.22 last 81.77) - open/close 36119 (time 1783564659.88 total 873.22 last 85.69) - open/close 37060 (time 1783564669.88 total 883.22 last 94.09) - open/close 37979 (time 1783564679.88 total 893.22 last 91.90) - open/close 38926 (time 1783564689.89 total 903.23 last 94.61) - open/close 39952 (time 1783564699.90 total 913.24 last 102.51) - open/close 41064 (time 1783564709.90 total 923.24 last 111.17) - open/close 42039 (time 1783564719.91 total 933.25 last 97.43) - open/close 43005 (time 1783564729.91 total 943.25 last 96.58) - open/close 43944 (time 1783564739.92 total 953.26 last 93.81) - open/close 44920 (time 1783564749.93 total 963.27 last 97.47) - open/close 46093 (time 1783564759.94 total 973.28 last 117.20) - open/close 47260 (time 1783564769.95 total 983.29 last 116.62) - open/close 48311 (time 1783564779.95 total 993.29 last 105.04) - open/close 49301 (time 1783564789.96 total 1003.31 last 98.88) total: 50000 open/close in 1010.07 seconds: 49.50 ops/second - unlinked 0 (time 1783564799 ; total 0 ; last 0) - unlinked 10000 (time 1783564874 ; total 75 ; last 75) - unlinked 20000 (time 1783564920 ; total 121 ; last 46) - unlinked 30000 (time 1783564972 ; total 173 ; last 52) - unlinked 40000 (time 1783565022 ; total 223 ; last 50) total: 50000 unlinks in 275 seconds: 181.818176 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: modules unloaded. pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2