== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:24:53 (1782231893) start mds service on oleg433-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: MODOPTS_LIBCFS= oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg433-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg433-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock 192.168.204.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1782231973.35 total 10.69 last 21.88) - open/close 423 (time 1782231983.36 total 20.71 last 18.88) - open/close 667 (time 1782231993.69 total 31.03 last 23.64) - open/close 897 (time 1782232003.70 total 41.04 last 22.97) - open/close 1151 (time 1782232014.76 total 52.10 last 22.97) - open/close 1371 (time 1782232024.80 total 62.14 last 21.91) - open/close 1703 (time 1782232035.31 total 72.65 last 31.60) - open/close 1949 (time 1782232045.31 total 82.65 last 24.60) - open/close 2182 (time 1782232056.17 total 93.51 last 21.45) - open/close 2423 (time 1782232066.71 total 104.06 last 22.85) - open/close 2669 (time 1782232076.74 total 114.08 last 24.55) - open/close 2920 (time 1782232087.31 total 124.66 last 23.73) - open/close 3185 (time 1782232097.32 total 134.66 last 26.49) - open/close 3446 (time 1782232107.74 total 145.08 last 25.05) - open/close 3693 (time 1782232117.74 total 155.09 last 24.69) - open/close 3920 (time 1782232128.81 total 166.15 last 20.52) - open/close 4143 (time 1782232138.82 total 176.16 last 22.27) - open/close 4373 (time 1782232148.90 total 186.24 last 22.82) - open/close 4613 (time 1782232159.18 total 196.53 last 23.34) - open/close 4787 (time 1782232169.94 total 207.28 last 16.18) - open/close 5010 (time 1782232180.19 total 217.54 last 21.75) - open/close 5248 (time 1782232191.21 total 228.56 last 21.60) - open/close 5485 (time 1782232201.21 total 238.56 last 23.70) - open/close 5689 (time 1782232211.22 total 248.57 last 20.38) - open/close 5941 (time 1782232221.40 total 258.75 last 24.76) - open/close 6181 (time 1782232231.77 total 269.11 last 23.16) - open/close 6391 (time 1782232241.77 total 279.11 last 21.00) - open/close 6600 (time 1782232251.79 total 289.13 last 20.86) - open/close 6826 (time 1782232261.79 total 299.13 last 22.59) - open/close 7109 (time 1782232271.80 total 309.14 last 28.28) - open/close 7320 (time 1782232282.01 total 319.35 last 20.67) - open/close 7543 (time 1782232292.96 total 330.31 last 20.35) - open/close 7771 (time 1782232303.64 total 340.98 last 21.36) - open/close 8007 (time 1782232313.64 total 350.98 last 23.60) - open/close 8223 (time 1782232324.13 total 361.47 last 20.60) - open/close 8427 (time 1782232334.62 total 371.96 last 19.44) - open/close 8614 (time 1782232344.63 total 381.97 last 18.68) - open/close 8821 (time 1782232355.61 total 392.95 last 18.85) - open/close 8992 (time 1782232366.25 total 403.59 last 16.08) - open/close 9171 (time 1782232376.27 total 413.61 last 17.86) - open/close 9362 (time 1782232386.40 total 423.75 last 18.85) - open/close 9552 (time 1782232396.42 total 433.77 last 18.96) - open/close 9678 (time 1782232406.43 total 443.77 last 12.60) - open/close 9822 (time 1782232416.66 total 454.00 last 14.07) - open/close 9989 (time 1782232427.38 total 464.73 last 15.57) - open/close 10158 (time 1782232437.96 total 475.31 last 15.98) - open/close 10260 (time 1782232448.72 total 486.07 last 9.48) - open/close 10466 (time 1782232458.79 total 496.13 last 20.46) - open/close 10658 (time 1782232469.65 total 506.99 last 17.69) - open/close 10842 (time 1782232479.77 total 517.12 last 18.17) - open/close 11037 (time 1782232489.88 total 527.22 last 19.30) - open/close 11182 (time 1782232499.99 total 537.34 last 14.33) - open/close 11390 (time 1782232510.01 total 547.35 last 20.77) - open/close 11561 (time 1782232520.66 total 558.00 last 16.05) - open/close 11697 (time 1782232530.67 total 568.01 last 13.59) - open/close 11823 (time 1782232540.67 total 578.02 last 12.59) - open/close 11917 (time 1782232551.32 total 588.67 last 8.83) - open/close 12128 (time 1782232562.02 total 599.36 last 19.73) - open/close 12256 (time 1782232572.03 total 609.38 last 12.78) - open/close 12391 (time 1782232582.04 total 619.38 last 13.50) - open/close 12573 (time 1782232593.00 total 630.34 last 16.60) - open/close 12697 (time 1782232603.93 total 641.27 last 11.35) - open/close 12774 (time 1782232614.74 total 652.08 last 7.12) - open/close 12953 (time 1782232624.76 total 662.10 last 17.87) - open/close 13135 (time 1782232635.44 total 672.79 last 17.03) - open/close 13285 (time 1782232646.39 total 683.73 last 13.71) - open/close 13333 (time 1782232657.11 total 694.45 last 4.48) - open/close 13429 (time 1782232667.71 total 705.05 last 9.06) - open/close 13536 (time 1782232677.74 total 715.08 last 10.67) - open/close 13613 (time 1782232688.69 total 726.03 last 7.03) - open/close 13793 (time 1782232698.69 total 736.03 last 18.00) - open/close 13997 (time 1782232708.70 total 746.04 last 20.39) - open/close 14188 (time 1782232718.70 total 756.04 last 19.09) - open/close 14389 (time 1782232728.71 total 766.06 last 20.07) - open/close 14610 (time 1782232738.78 total 776.12 last 21.97) - open/close 14852 (time 1782232751.19 total 788.53 last 19.49) - open/close 15085 (time 1782232761.20 total 798.54 last 23.28) - open/close 15313 (time 1782232771.54 total 808.88 last 22.04) - open/close 15977 (time 1782232781.54 total 818.88 last 66.39) - open/close 16654 (time 1782232791.55 total 828.89 last 67.67) - open/close 17401 (time 1782232801.56 total 838.90 last 74.61) - open/close 18081 (time 1782232811.56 total 848.90 last 67.98) - open/close 18819 (time 1782232821.62 total 858.96 last 73.37) - open/close 19573 (time 1782232831.62 total 868.97 last 75.38) - open/close 20000 (time 1782232837.64 total 874.98 last 70.97) - open/close 20752 (time 1782232847.64 total 884.98 last 75.18) - open/close 21565 (time 1782232857.68 total 895.03 last 80.96) - open/close 22309 (time 1782232867.75 total 905.09 last 73.90) - open/close 23036 (time 1782232877.78 total 915.12 last 72.48) - open/close 23736 (time 1782232887.79 total 925.13 last 69.95) - open/close 24427 (time 1782232897.80 total 935.15 last 69.00) - open/close 25166 (time 1782232907.81 total 945.15 last 73.88) - open/close 25820 (time 1782232917.82 total 955.16 last 65.31) - open/close 26368 (time 1782232927.83 total 965.18 last 54.73) - open/close 27042 (time 1782232937.85 total 975.19 last 67.31) - open/close 27678 (time 1782232947.86 total 985.20 last 63.52) - open/close 28206 (time 1782232957.87 total 995.21 last 52.76) - open/close 28727 (time 1782232967.88 total 1005.22 last 52.05) - open/close 29216 (time 1782232977.89 total 1015.24 last 48.82) - open/close 29676 (time 1782232987.91 total 1025.26 last 45.91) - open/close 30000 (time 1782232994.03 total 1031.37 last 52.97) - open/close 30640 (time 1782233004.04 total 1041.38 last 63.94) - open/close 31369 (time 1782233014.04 total 1051.39 last 72.87) - open/close 32043 (time 1782233024.10 total 1061.44 last 67.04) - open/close 32612 (time 1782233034.12 total 1071.46 last 56.77) - open/close 33259 (time 1782233044.13 total 1081.47 last 64.66) - open/close 33899 (time 1782233054.13 total 1091.48 last 63.95) - open/close 34512 (time 1782233064.14 total 1101.48 last 61.27) - open/close 35236 (time 1782233074.14 total 1111.48 last 72.40) - open/close 35935 (time 1782233084.15 total 1121.49 last 69.86) - open/close 36619 (time 1782233094.15 total 1131.50 last 68.35) - open/close 37233 (time 1782233104.15 total 1141.50 last 61.40) - open/close 37799 (time 1782233114.16 total 1151.50 last 56.59) - open/close 38491 (time 1782233124.16 total 1161.50 last 69.16) - open/close 39089 (time 1782233134.18 total 1171.52 last 59.68) - open/close 39795 (time 1782233144.20 total 1181.54 last 70.47) - open/close 40000 (time 1782233147.17 total 1184.51 last 69.10) - open/close 40674 (time 1782233157.17 total 1194.52 last 67.35) - open/close 41395 (time 1782233167.18 total 1204.53 last 72.03) - open/close 42024 (time 1782233177.19 total 1214.53 last 62.86) - open/close 42685 (time 1782233187.20 total 1224.54 last 66.03) - open/close 43325 (time 1782233197.20 total 1234.55 last 63.98) - open/close 44002 (time 1782233207.21 total 1244.56 last 67.62) - open/close 44521 (time 1782233217.22 total 1254.57 last 51.85) - open/close 45196 (time 1782233227.22 total 1264.57 last 67.50) - open/close 45888 (time 1782233237.23 total 1274.57 last 69.17) - open/close 46409 (time 1782233247.27 total 1284.61 last 51.89) - open/close 46897 (time 1782233257.28 total 1294.62 last 48.77) - open/close 47529 (time 1782233267.29 total 1304.63 last 63.12) - open/close 48136 (time 1782233277.31 total 1314.66 last 60.54) - open/close 48787 (time 1782233287.32 total 1324.66 last 65.06) - open/close 49434 (time 1782233297.32 total 1334.67 last 64.69) total: 50000 open/close in 1343.20 seconds: 37.22 ops/second - unlinked 0 (time 1782233311 ; total 0 ; last 0) - unlinked 10000 (time 1782233467 ; total 156 ; last 156) - unlinked 20000 (time 1782233630 ; total 319 ; last 163) - unlinked 30000 (time 1782233797 ; total 486 ; last 167) - unlinked 40000 (time 1782233956 ; total 645 ; last 159) total: 50000 unlinks in 802 seconds: 62.344139 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server unloading modules via unload_modules_local on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2