== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:06:29 (1780855589) start mds service on oleg424-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg424-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1780855659.81 total 10.34 last 23.02) - open/close 478 (time 1780855670.06 total 20.59 last 23.42) - open/close 648 (time 1780855680.16 total 30.69 last 16.83) - open/close 890 (time 1780855690.73 total 41.26 last 22.89) - open/close 1094 (time 1780855701.22 total 51.75 last 19.45) - open/close 1316 (time 1780855712.23 total 62.76 last 20.16) - open/close 1550 (time 1780855722.60 total 73.13 last 22.55) - open/close 1787 (time 1780855732.61 total 83.14 last 23.68) - open/close 2000 (time 1780855742.95 total 93.49 last 20.59) - open/close 2216 (time 1780855752.99 total 103.52 last 21.53) - open/close 2462 (time 1780855763.88 total 114.41 last 22.58) - open/close 2698 (time 1780855774.00 total 124.53 last 23.31) - open/close 2902 (time 1780855784.02 total 134.55 last 20.37) - open/close 3097 (time 1780855794.02 total 144.55 last 19.50) - open/close 3333 (time 1780855804.12 total 154.65 last 23.35) - open/close 3545 (time 1780855815.07 total 165.60 last 19.37) - open/close 3796 (time 1780855825.44 total 175.97 last 24.21) - open/close 3988 (time 1780855835.45 total 185.98 last 19.19) - open/close 4128 (time 1780855845.66 total 196.19 last 13.71) - open/close 4304 (time 1780855856.54 total 207.08 last 16.17) - open/close 4532 (time 1780855866.55 total 217.08 last 22.79) - open/close 4743 (time 1780855877.48 total 228.01 last 19.31) - open/close 4967 (time 1780855887.48 total 238.01 last 22.38) - open/close 5161 (time 1780855897.69 total 248.23 last 19.00) - open/close 5368 (time 1780855908.70 total 259.23 last 18.80) - open/close 5582 (time 1780855919.13 total 269.66 last 20.52) - open/close 5813 (time 1780855929.13 total 279.66 last 23.09) - open/close 5978 (time 1780855939.37 total 289.90 last 16.12) - open/close 6154 (time 1780855949.37 total 299.90 last 17.60) - open/close 6285 (time 1780855959.38 total 309.91 last 13.08) - open/close 6461 (time 1780855969.39 total 319.92 last 17.58) - open/close 6647 (time 1780855979.40 total 329.93 last 18.59) - open/close 6822 (time 1780855990.44 total 340.97 last 15.84) - open/close 6953 (time 1780856000.60 total 351.13 last 12.89) - open/close 7083 (time 1780856010.85 total 361.38 last 12.69) - open/close 7262 (time 1780856021.47 total 372.00 last 16.85) - open/close 7452 (time 1780856032.49 total 383.02 last 17.25) - open/close 7639 (time 1780856043.49 total 394.02 last 16.99) - open/close 7851 (time 1780856054.24 total 404.77 last 19.72) - open/close 8093 (time 1780856065.11 total 415.64 last 22.26) - open/close 8277 (time 1780856076.00 total 426.53 last 16.90) - open/close 8455 (time 1780856086.00 total 436.53 last 17.80) - open/close 8680 (time 1780856096.66 total 447.20 last 21.10) - open/close 8889 (time 1780856107.54 total 458.08 last 19.21) - open/close 9098 (time 1780856118.24 total 468.77 last 19.55) - open/close 9330 (time 1780856128.24 total 478.77 last 23.20) - open/close 9552 (time 1780856138.26 total 488.79 last 22.14) - open/close 9765 (time 1780856148.89 total 499.42 last 20.05) - open/close 10000 (time 1780856156.42 total 506.95 last 31.19) - open/close 10269 (time 1780856166.82 total 517.35 last 25.88) - open/close 10469 (time 1780856177.50 total 528.03 last 18.72) - open/close 10698 (time 1780856187.51 total 538.04 last 22.89) - open/close 10898 (time 1780856198.17 total 548.70 last 18.76) - open/close 11114 (time 1780856209.12 total 559.65 last 19.73) - open/close 11322 (time 1780856219.75 total 570.28 last 19.57) - open/close 11565 (time 1780856229.98 total 580.51 last 23.74) - open/close 11795 (time 1780856240.99 total 591.52 last 20.90) - open/close 11981 (time 1780856251.09 total 601.62 last 18.41) - open/close 12178 (time 1780856261.28 total 611.81 last 19.35) - open/close 12340 (time 1780856272.09 total 622.62 last 14.97) - open/close 12522 (time 1780856282.22 total 632.75 last 17.97) - open/close 12701 (time 1780856292.32 total 642.85 last 17.73) - open/close 12888 (time 1780856302.49 total 653.02 last 18.38) - open/close 13019 (time 1780856312.93 total 663.46 last 12.55) - open/close 13191 (time 1780856323.75 total 674.28 last 15.90) - open/close 13394 (time 1780856333.75 total 684.28 last 20.30) - open/close 13562 (time 1780856344.22 total 694.75 last 16.04) - open/close 13748 (time 1780856354.23 total 704.76 last 18.59) - open/close 13893 (time 1780856364.96 total 715.49 last 13.51) - open/close 14079 (time 1780856375.85 total 726.38 last 17.08) - open/close 14292 (time 1780856385.85 total 736.39 last 21.30) - open/close 14948 (time 1780856395.86 total 746.39 last 65.59) - open/close 15762 (time 1780856405.86 total 756.39 last 81.39) - open/close 16640 (time 1780856415.87 total 766.40 last 87.73) - open/close 17690 (time 1780856425.87 total 776.40 last 104.99) - open/close 18611 (time 1780856435.88 total 786.41 last 91.97) - open/close 19536 (time 1780856445.89 total 796.42 last 92.44) - open/close 20000 (time 1780856451.38 total 801.91 last 84.42) - open/close 20913 (time 1780856461.39 total 811.92 last 91.22) - open/close 21945 (time 1780856471.39 total 821.93 last 103.16) - open/close 22919 (time 1780856481.40 total 831.93 last 97.37) - open/close 23888 (time 1780856491.40 total 841.93 last 96.83) - open/close 24825 (time 1780856501.42 total 851.95 last 93.58) - open/close 25686 (time 1780856511.42 total 861.95 last 86.06) - open/close 26471 (time 1780856521.42 total 871.95 last 78.48) - open/close 27331 (time 1780856531.43 total 881.97 last 85.91) - open/close 28281 (time 1780856541.44 total 891.97 last 94.99) - open/close 29303 (time 1780856551.44 total 901.97 last 102.12) - open/close 30000 (time 1780856557.60 total 908.13 last 113.17) - open/close 31092 (time 1780856567.61 total 918.14 last 109.14) - open/close 32083 (time 1780856577.61 total 928.14 last 99.04) - open/close 33100 (time 1780856587.62 total 938.15 last 101.67) - open/close 34195 (time 1780856597.62 total 948.15 last 109.47) - open/close 35219 (time 1780856607.63 total 958.16 last 102.34) - open/close 36156 (time 1780856617.64 total 968.17 last 93.61) - open/close 37070 (time 1780856627.65 total 978.18 last 91.31) - open/close 37978 (time 1780856637.65 total 988.19 last 90.72) - open/close 38912 (time 1780856647.67 total 998.20 last 93.25) - open/close 39819 (time 1780856657.68 total 1008.21 last 90.66) - open/close 40000 (time 1780856659.73 total 1010.26 last 88.00) - open/close 40887 (time 1780856669.73 total 1020.27 last 88.68) - open/close 41835 (time 1780856679.74 total 1030.27 last 94.78) - open/close 42925 (time 1780856689.74 total 1040.27 last 108.93) - open/close 43845 (time 1780856699.75 total 1050.28 last 91.93) - open/close 44963 (time 1780856709.76 total 1060.29 last 111.74) - open/close 46020 (time 1780856719.77 total 1070.30 last 105.60) - open/close 47124 (time 1780856729.77 total 1080.30 last 110.38) - open/close 48240 (time 1780856739.77 total 1090.30 last 111.60) - open/close 49803 (time 1780856749.78 total 1100.31 last 156.09) total: 50000 open/close in 1101.50 seconds: 45.39 ops/second - unlinked 0 (time 1780856752 ; total 0 ; last 0) - unlinked 10000 (time 1780856839 ; total 87 ; last 87) - unlinked 20000 (time 1780856898 ; total 146 ; last 59) - unlinked 30000 (time 1780856946 ; total 194 ; last 48) - unlinked 40000 (time 1780857002 ; total 250 ; last 56) total: 50000 unlinks in 297 seconds: 168.350174 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2