== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:43:56 (1787708636) start mds service on oleg354-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing load_modules_local oleg354-server: Loading modules from /home/green/git/lustre-release/lustre oleg354-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg354-server: detected 4 online CPUs by sysfs oleg354-server: MODOPTS_LIBCFS= oleg354-server: Force libcfs to create 2 CPU partitions oleg354-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg354-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg354-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1787708714.35 total 10.63 last 22.39) - open/close 468 (time 1787708724.92 total 21.20 last 21.76) - open/close 693 (time 1787708734.92 total 31.20 last 22.50) - open/close 939 (time 1787708744.92 total 41.20 last 24.60) - open/close 1156 (time 1787708755.52 total 51.80 last 20.47) - open/close 1384 (time 1787708765.75 total 62.03 last 22.29) - open/close 1625 (time 1787708776.12 total 72.40 last 23.25) - open/close 1875 (time 1787708786.36 total 82.64 last 24.41) - open/close 2104 (time 1787708796.48 total 92.76 last 22.63) - open/close 2348 (time 1787708807.47 total 103.75 last 22.20) - open/close 2598 (time 1787708817.48 total 113.76 last 24.99) - open/close 2828 (time 1787708828.01 total 124.29 last 21.83) - open/close 3103 (time 1787708838.12 total 134.40 last 27.20) - open/close 3347 (time 1787708848.24 total 144.52 last 24.11) - open/close 3639 (time 1787708858.25 total 154.53 last 29.16) - open/close 3882 (time 1787708868.28 total 164.56 last 24.24) - open/close 4154 (time 1787708878.28 total 174.56 last 27.19) - open/close 4409 (time 1787708888.29 total 184.57 last 25.49) - open/close 4670 (time 1787708898.30 total 194.58 last 26.06) - open/close 4807 (time 1787708908.32 total 204.60 last 13.68) - open/close 4962 (time 1787708918.91 total 215.19 last 14.64) - open/close 5181 (time 1787708928.91 total 225.19 last 21.90) - open/close 5346 (time 1787708938.91 total 235.19 last 16.50) - open/close 5602 (time 1787708948.91 total 245.19 last 25.59) - open/close 5823 (time 1787708958.92 total 255.20 last 22.08) - open/close 6085 (time 1787708969.79 total 266.07 last 24.09) - open/close 6330 (time 1787708980.08 total 276.36 last 23.81) - open/close 6562 (time 1787708990.09 total 286.37 last 23.19) - open/close 6795 (time 1787709000.63 total 296.91 last 22.11) - open/close 7035 (time 1787709010.99 total 307.27 last 23.16) - open/close 7291 (time 1787709021.69 total 317.97 last 23.93) - open/close 7534 (time 1787709032.31 total 328.59 last 22.87) - open/close 7816 (time 1787709043.24 total 339.52 last 25.80) - open/close 8088 (time 1787709053.26 total 349.54 last 27.14) - open/close 8209 (time 1787709063.28 total 359.56 last 12.08) - open/close 8458 (time 1787709074.16 total 370.44 last 22.88) - open/close 8737 (time 1787709084.27 total 380.55 last 27.62) - open/close 9010 (time 1787709094.27 total 390.55 last 27.28) - open/close 9289 (time 1787709104.94 total 401.22 last 26.16) - open/close 9549 (time 1787709115.77 total 412.05 last 23.99) - open/close 9789 (time 1787709126.01 total 422.29 last 23.44) - open/close 10000 (time 1787709133.98 total 430.26 last 26.48) - open/close 10183 (time 1787709144.19 total 440.46 last 17.93) - open/close 10440 (time 1787709154.41 total 450.69 last 25.14) - open/close 10650 (time 1787709164.41 total 460.69 last 21.00) - open/close 10799 (time 1787709175.09 total 471.37 last 13.94) - open/close 11025 (time 1787709185.79 total 482.07 last 21.12) - open/close 11183 (time 1787709195.80 total 492.08 last 15.79) - open/close 11395 (time 1787709205.81 total 502.09 last 21.17) - open/close 11632 (time 1787709216.68 total 512.96 last 21.80) - open/close 11904 (time 1787709227.14 total 523.42 last 26.01) - open/close 12143 (time 1787709237.15 total 533.43 last 23.87) - open/close 12381 (time 1787709247.16 total 543.44 last 23.77) - open/close 12615 (time 1787709257.16 total 553.44 last 23.39) - open/close 12841 (time 1787709267.18 total 563.46 last 22.57) - open/close 13067 (time 1787709277.18 total 573.46 last 22.59) - open/close 13296 (time 1787709287.54 total 583.82 last 22.10) - open/close 13583 (time 1787709298.41 total 594.69 last 26.41) - open/close 13812 (time 1787709308.42 total 604.70 last 22.88) - open/close 14029 (time 1787709318.46 total 614.74 last 21.61) - open/close 14289 (time 1787709329.46 total 625.74 last 23.64) - open/close 14562 (time 1787709340.05 total 636.32 last 25.78) - open/close 14823 (time 1787709350.90 total 647.18 last 24.05) - open/close 15054 (time 1787709361.14 total 657.42 last 22.56) - open/close 15298 (time 1787709371.56 total 667.84 last 23.40) - open/close 15544 (time 1787709381.75 total 678.03 last 24.16) - open/close 15795 (time 1787709391.88 total 688.16 last 24.77) - open/close 16041 (time 1787709401.89 total 698.17 last 24.59) - open/close 16252 (time 1787709412.40 total 708.68 last 20.07) - open/close 16477 (time 1787709422.59 total 718.87 last 22.08) - open/close 16741 (time 1787709432.60 total 728.87 last 26.39) - open/close 16946 (time 1787709443.00 total 739.27 last 19.71) - open/close 17213 (time 1787709453.80 total 750.08 last 24.70) - open/close 17459 (time 1787709463.82 total 760.09 last 24.57) - open/close 17692 (time 1787709473.82 total 770.10 last 23.29) - open/close 17926 (time 1787709484.02 total 780.30 last 22.93) - open/close 18180 (time 1787709494.91 total 791.19 last 23.33) - open/close 18433 (time 1787709505.54 total 801.82 last 23.80) - open/close 18660 (time 1787709515.58 total 811.85 last 22.61) - open/close 18883 (time 1787709525.89 total 822.17 last 21.63) - open/close 19186 (time 1787709536.06 total 832.34 last 29.79) - open/close 19474 (time 1787709546.29 total 842.56 last 28.16) - open/close 19636 (time 1787709556.29 total 852.57 last 16.19) - open/close 19860 (time 1787709566.30 total 862.57 last 22.39) - open/close 20000 (time 1787709569.40 total 865.68 last 45.07) - open/close 20775 (time 1787709579.40 total 875.68 last 77.50) - open/close 21530 (time 1787709589.41 total 885.69 last 75.47) - open/close 22256 (time 1787709599.42 total 895.69 last 72.53) - open/close 23002 (time 1787709609.42 total 905.70 last 74.55) - open/close 23601 (time 1787709619.43 total 915.71 last 59.84) - open/close 24419 (time 1787709629.43 total 925.71 last 81.78) - open/close 25194 (time 1787709639.44 total 935.72 last 77.47) - open/close 25955 (time 1787709649.46 total 945.73 last 75.96) - open/close 26696 (time 1787709659.47 total 955.75 last 74.01) - open/close 27451 (time 1787709669.47 total 965.75 last 75.45) - open/close 28256 (time 1787709679.49 total 975.77 last 80.38) - open/close 29080 (time 1787709689.50 total 985.78 last 82.31) - open/close 29861 (time 1787709699.50 total 995.78 last 78.07) - open/close 30684 (time 1787709709.51 total 1005.79 last 82.23) - open/close 31458 (time 1787709719.52 total 1015.80 last 77.35) - open/close 32202 (time 1787709729.52 total 1025.80 last 74.38) - open/close 32924 (time 1787709739.53 total 1035.81 last 72.12) - open/close 33706 (time 1787709749.54 total 1045.82 last 78.11) - open/close 34497 (time 1787709759.56 total 1055.83 last 79.01) - open/close 35255 (time 1787709769.56 total 1065.84 last 75.78) - open/close 35972 (time 1787709779.60 total 1075.88 last 71.40) - open/close 36754 (time 1787709789.61 total 1085.89 last 78.12) - open/close 37511 (time 1787709799.62 total 1095.90 last 75.60) - open/close 38198 (time 1787709809.63 total 1105.91 last 68.64) - open/close 38984 (time 1787709819.64 total 1115.92 last 78.55) - open/close 39796 (time 1787709829.64 total 1125.92 last 81.19) - open/close 40000 (time 1787709832.59 total 1128.87 last 69.14) - open/close 40778 (time 1787709842.60 total 1138.88 last 77.73) - open/close 41548 (time 1787709852.61 total 1148.89 last 76.88) - open/close 42325 (time 1787709862.62 total 1158.90 last 77.64) - open/close 43054 (time 1787709872.62 total 1168.90 last 72.90) - open/close 43751 (time 1787709882.65 total 1178.93 last 69.51) - open/close 44430 (time 1787709892.65 total 1188.93 last 67.89) - open/close 45111 (time 1787709902.65 total 1198.93 last 68.08) - open/close 45908 (time 1787709912.68 total 1208.96 last 79.52) - open/close 46719 (time 1787709922.68 total 1218.96 last 81.09) - open/close 47507 (time 1787709932.68 total 1228.96 last 78.75) - open/close 48306 (time 1787709942.70 total 1238.98 last 79.80) - open/close 49105 (time 1787709952.70 total 1248.98 last 79.85) - open/close 49872 (time 1787709962.70 total 1258.98 last 76.69) total: 50000 open/close in 1260.89 seconds: 39.65 ops/second - unlinked 0 (time 1787709968 ; total 0 ; last 0) - unlinked 10000 (time 1787710102 ; total 134 ; last 134) - unlinked 20000 (time 1787710242 ; total 274 ; last 140) - unlinked 30000 (time 1787710384 ; total 416 ; last 142) - unlinked 40000 (time 1787710526 ; total 558 ; last 142) total: 50000 unlinks in 694 seconds: 72.046112 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server unloading modules via unload_modules_local on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local oleg354-server: modules unloaded. pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2