== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:05:35 (1780333535) start mds service on oleg323-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg323-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1780333613.42 total 10.98 last 19.49) - open/close 435 (time 1780333623.43 total 21.00 last 22.06) - open/close 632 (time 1780333634.36 total 31.93 last 18.03) - open/close 836 (time 1780333644.37 total 41.94 last 20.38) - open/close 1094 (time 1780333654.89 total 52.45 last 24.53) - open/close 1322 (time 1780333665.21 total 62.78 last 22.09) - open/close 1498 (time 1780333675.23 total 72.79 last 17.58) - open/close 1599 (time 1780333685.81 total 83.37 last 9.54) - open/close 1769 (time 1780333695.82 total 93.38 last 16.99) - open/close 2002 (time 1780333705.84 total 103.40 last 23.26) - open/close 2193 (time 1780333715.85 total 113.41 last 19.08) - open/close 2316 (time 1780333726.09 total 123.65 last 12.01) - open/close 2573 (time 1780333736.88 total 134.44 last 23.82) - open/close 2837 (time 1780333746.90 total 144.46 last 26.37) - open/close 3086 (time 1780333757.04 total 154.60 last 24.55) - open/close 3391 (time 1780333767.53 total 165.10 last 29.06) - open/close 3648 (time 1780333778.09 total 175.65 last 24.34) - open/close 3943 (time 1780333788.09 total 185.65 last 29.50) - open/close 4191 (time 1780333798.84 total 196.40 last 23.09) - open/close 4477 (time 1780333809.60 total 207.16 last 26.58) - open/close 4740 (time 1780333820.41 total 217.97 last 24.31) - open/close 4965 (time 1780333830.65 total 228.21 last 21.98) - open/close 5251 (time 1780333841.28 total 238.84 last 26.90) - open/close 5500 (time 1780333851.33 total 248.89 last 24.76) - open/close 5647 (time 1780333861.52 total 259.08 last 14.43) - open/close 5719 (time 1780333871.54 total 269.11 last 7.18) - open/close 5990 (time 1780333882.49 total 280.05 last 24.76) - open/close 6242 (time 1780333893.29 total 290.85 last 23.33) - open/close 6499 (time 1780333903.81 total 301.37 last 24.43) - open/close 6712 (time 1780333913.82 total 311.38 last 21.28) - open/close 6912 (time 1780333925.38 total 322.94 last 17.31) - open/close 7176 (time 1780333936.18 total 333.74 last 24.43) - open/close 7408 (time 1780333946.41 total 343.97 last 22.68) - open/close 7671 (time 1780333957.11 total 354.67 last 24.58) - open/close 7939 (time 1780333967.93 total 365.50 last 24.76) - open/close 8237 (time 1780333978.48 total 376.04 last 28.27) - open/close 8461 (time 1780333988.59 total 386.16 last 22.14) - open/close 8667 (time 1780333998.60 total 396.16 last 20.59) - open/close 8903 (time 1780334008.60 total 406.17 last 23.59) - open/close 9125 (time 1780334019.24 total 416.80 last 20.87) - open/close 9179 (time 1780334030.26 total 427.82 last 4.90) - open/close 9359 (time 1780334041.08 total 438.64 last 16.64) - open/close 9578 (time 1780334051.43 total 448.99 last 21.15) - open/close 9699 (time 1780334061.75 total 459.31 last 11.73) - open/close 9969 (time 1780334072.06 total 469.62 last 26.18) - open/close 10216 (time 1780334082.08 total 479.64 last 24.66) - open/close 10423 (time 1780334092.42 total 489.98 last 20.01) - open/close 10698 (time 1780334102.44 total 500.00 last 27.46) - open/close 11022 (time 1780334113.27 total 510.83 last 29.92) - open/close 11273 (time 1780334123.28 total 520.84 last 25.07) - open/close 11505 (time 1780334133.29 total 530.85 last 23.18) - open/close 11744 (time 1780334143.29 total 540.85 last 23.89) - open/close 11994 (time 1780334153.30 total 550.86 last 24.97) - open/close 12240 (time 1780334163.35 total 560.91 last 24.50) - open/close 12434 (time 1780334173.84 total 571.40 last 18.49) - open/close 12614 (time 1780334183.84 total 581.40 last 17.99) - open/close 12730 (time 1780334194.69 total 592.25 last 10.70) - open/close 12959 (time 1780334204.79 total 602.35 last 22.66) - open/close 13219 (time 1780334215.41 total 612.97 last 24.48) - open/close 13407 (time 1780334226.10 total 623.66 last 17.59) - open/close 13522 (time 1780334236.11 total 633.67 last 11.49) - open/close 13714 (time 1780334246.12 total 643.68 last 19.18) - open/close 13970 (time 1780334256.13 total 653.70 last 25.56) - open/close 14161 (time 1780334266.43 total 663.99 last 18.55) - open/close 14409 (time 1780334277.07 total 674.63 last 23.31) - open/close 14608 (time 1780334287.07 total 684.63 last 19.90) - open/close 14844 (time 1780334297.68 total 695.24 last 22.25) - open/close 15142 (time 1780334308.21 total 705.77 last 28.30) - open/close 15403 (time 1780334318.64 total 716.20 last 25.01) - open/close 15632 (time 1780334328.67 total 726.23 last 22.84) - open/close 15842 (time 1780334339.06 total 736.62 last 20.22) - open/close 16094 (time 1780334349.23 total 746.79 last 24.78) - open/close 16358 (time 1780334359.87 total 757.43 last 24.81) - open/close 16602 (time 1780334370.55 total 768.11 last 22.84) - open/close 16786 (time 1780334380.56 total 778.12 last 18.39) - open/close 16955 (time 1780334393.47 total 791.03 last 13.09) - open/close 17212 (time 1780334404.21 total 801.77 last 23.92) - open/close 17472 (time 1780334414.61 total 812.17 last 25.00) - open/close 17686 (time 1780334424.61 total 822.18 last 21.39) - open/close 17937 (time 1780334434.94 total 832.50 last 24.31) - open/close 18218 (time 1780334444.94 total 842.50 last 28.10) - open/close 18438 (time 1780334455.08 total 852.64 last 21.69) - open/close 18647 (time 1780334465.90 total 863.46 last 19.31) - open/close 18881 (time 1780334476.00 total 873.56 last 23.18) - open/close 19143 (time 1780334486.00 total 883.56 last 26.20) - open/close 19389 (time 1780334496.11 total 893.68 last 24.32) - open/close 19651 (time 1780334507.88 total 905.44 last 22.28) - open/close 19851 (time 1780334518.16 total 915.72 last 19.45) - open/close 20000 (time 1780334523.07 total 920.63 last 30.33) - open/close 20767 (time 1780334533.08 total 930.64 last 76.66) - open/close 21521 (time 1780334543.09 total 940.65 last 75.28) - open/close 22300 (time 1780334553.09 total 950.65 last 77.89) - open/close 23082 (time 1780334563.10 total 960.66 last 78.13) - open/close 23900 (time 1780334573.12 total 970.68 last 81.66) - open/close 24689 (time 1780334583.13 total 980.69 last 78.85) - open/close 25457 (time 1780334593.13 total 990.69 last 76.77) - open/close 26227 (time 1780334603.13 total 1000.70 last 76.96) - open/close 26986 (time 1780334613.14 total 1010.70 last 75.86) - open/close 27725 (time 1780334623.14 total 1020.70 last 73.88) - open/close 28493 (time 1780334633.15 total 1030.71 last 76.72) - open/close 29287 (time 1780334643.16 total 1040.72 last 79.33) - open/close 30000 (time 1780334651.50 total 1049.06 last 85.53) - open/close 30842 (time 1780334661.50 total 1059.07 last 84.14) - open/close 31700 (time 1780334671.51 total 1069.07 last 85.74) - open/close 32565 (time 1780334681.52 total 1079.08 last 86.45) - open/close 33416 (time 1780334691.52 total 1089.08 last 85.09) - open/close 34291 (time 1780334701.58 total 1099.14 last 86.95) - open/close 35185 (time 1780334711.58 total 1109.14 last 89.40) - open/close 35963 (time 1780334721.60 total 1119.17 last 77.63) - open/close 36749 (time 1780334731.61 total 1129.17 last 78.56) - open/close 37567 (time 1780334741.61 total 1139.17 last 81.79) - open/close 38386 (time 1780334751.61 total 1149.17 last 81.89) - open/close 39229 (time 1780334761.61 total 1159.18 last 84.28) - open/close 39972 (time 1780334771.62 total 1169.18 last 74.26) - open/close 40750 (time 1780334781.63 total 1179.19 last 77.71) - open/close 41511 (time 1780334791.65 total 1189.21 last 75.97) - open/close 42296 (time 1780334801.65 total 1199.21 last 78.47) - open/close 43047 (time 1780334811.66 total 1209.22 last 75.07) - open/close 43860 (time 1780334821.68 total 1219.24 last 81.13) - open/close 44693 (time 1780334831.68 total 1229.24 last 83.29) - open/close 45540 (time 1780334841.69 total 1239.25 last 84.64) - open/close 46388 (time 1780334851.70 total 1249.26 last 84.69) - open/close 47183 (time 1780334861.71 total 1259.27 last 79.43) - open/close 48078 (time 1780334871.71 total 1269.28 last 89.44) - open/close 48939 (time 1780334881.73 total 1279.29 last 85.99) - open/close 49768 (time 1780334891.73 total 1289.29 last 82.88) total: 50000 open/close in 1292.00 seconds: 38.70 ops/second - unlinked 0 (time 1780334898 ; total 0 ; last 0) - unlinked 10000 (time 1780335028 ; total 130 ; last 130) - unlinked 20000 (time 1780335158 ; total 260 ; last 130) - unlinked 30000 (time 1780335293 ; total 395 ; last 135) - unlinked 40000 (time 1780335431 ; total 533 ; last 138) total: 50000 unlinks in 670 seconds: 74.626869 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2