== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:50:19 (1781232619) start mds service on oleg348-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg348-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg348-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1781232702.60 total 10.01 last 18.98) - open/close 372 (time 1781232712.63 total 20.04 last 18.16) - open/close 535 (time 1781232722.64 total 30.04 last 16.29) - open/close 759 (time 1781232733.60 total 41.01 last 20.43) - open/close 901 (time 1781232743.71 total 51.12 last 14.04) - open/close 1137 (time 1781232754.53 total 61.94 last 21.82) - open/close 1348 (time 1781232764.89 total 72.30 last 20.36) - open/close 1577 (time 1781232775.27 total 82.68 last 22.08) - open/close 1829 (time 1781232785.83 total 93.23 last 23.87) - open/close 1996 (time 1781232796.85 total 104.26 last 15.15) - open/close 2245 (time 1781232806.87 total 114.28 last 24.85) - open/close 2481 (time 1781232816.89 total 124.30 last 23.54) - open/close 2715 (time 1781232827.74 total 135.15 last 21.57) - open/close 2951 (time 1781232837.75 total 145.15 last 23.59) - open/close 3154 (time 1781232848.56 total 155.97 last 18.78) - open/close 3288 (time 1781232858.58 total 165.99 last 13.37) - open/close 3488 (time 1781232868.58 total 175.99 last 20.00) - open/close 3672 (time 1781232878.64 total 186.05 last 18.29) - open/close 3927 (time 1781232889.38 total 196.79 last 23.74) - open/close 4153 (time 1781232899.87 total 207.28 last 21.55) - open/close 4410 (time 1781232909.88 total 217.29 last 25.68) - open/close 4488 (time 1781232919.97 total 227.38 last 7.73) - open/close 4617 (time 1781232930.27 total 237.68 last 12.52) - open/close 4809 (time 1781232940.32 total 247.73 last 19.10) - open/close 5059 (time 1781232950.34 total 257.75 last 24.97) - open/close 5279 (time 1781232960.99 total 268.40 last 20.65) - open/close 5431 (time 1781232974.06 total 281.46 last 11.64) - open/close 5562 (time 1781232984.24 total 291.65 last 12.86) - open/close 5750 (time 1781232995.11 total 302.52 last 17.30) - open/close 5972 (time 1781233007.78 total 315.18 last 17.52) - open/close 6213 (time 1781233018.31 total 325.72 last 22.87) - open/close 6444 (time 1781233029.06 total 336.47 last 21.50) - open/close 6673 (time 1781233039.39 total 346.80 last 22.16) - open/close 6861 (time 1781233049.47 total 356.88 last 18.65) - open/close 6934 (time 1781233060.51 total 367.92 last 6.61) - open/close 7159 (time 1781233072.89 total 380.29 last 18.18) - open/close 7347 (time 1781233082.90 total 390.30 last 18.78) - open/close 7522 (time 1781233092.90 total 400.31 last 17.49) - open/close 7656 (time 1781233103.14 total 410.55 last 13.09) - open/close 7936 (time 1781233113.71 total 421.12 last 26.49) - open/close 8117 (time 1781233123.72 total 431.13 last 18.07) - open/close 8286 (time 1781233134.63 total 442.04 last 15.50) - open/close 8421 (time 1781233145.20 total 452.61 last 12.77) - open/close 8605 (time 1781233155.29 total 462.70 last 18.23) - open/close 8829 (time 1781233166.26 total 473.67 last 20.42) - open/close 9019 (time 1781233176.36 total 483.77 last 18.82) - open/close 9208 (time 1781233187.25 total 494.65 last 17.36) - open/close 9474 (time 1781233198.05 total 505.46 last 24.62) - open/close 9694 (time 1781233208.07 total 515.48 last 21.96) - open/close 9937 (time 1781233218.07 total 525.48 last 24.30) - open/close 10000 (time 1781233220.39 total 527.80 last 27.08) - open/close 10147 (time 1781233230.88 total 538.29 last 14.02) - open/close 10397 (time 1781233241.82 total 549.23 last 22.85) - open/close 10658 (time 1781233251.85 total 559.26 last 26.02) - open/close 10770 (time 1781233261.88 total 569.29 last 11.17) - open/close 11021 (time 1781233271.89 total 579.30 last 25.06) - open/close 11203 (time 1781233282.02 total 589.42 last 17.98) - open/close 11237 (time 1781233292.39 total 599.80 last 3.28) - open/close 11456 (time 1781233302.82 total 610.22 last 21.00) - open/close 11696 (time 1781233312.83 total 620.24 last 23.96) - open/close 11864 (time 1781233322.85 total 630.26 last 16.77) - open/close 12028 (time 1781233332.86 total 640.27 last 16.39) - open/close 12256 (time 1781233342.87 total 650.28 last 22.76) - open/close 12490 (time 1781233353.38 total 660.79 last 22.27) - open/close 12712 (time 1781233363.39 total 670.80 last 22.18) - open/close 12925 (time 1781233373.41 total 680.82 last 21.26) - open/close 13124 (time 1781233383.43 total 690.83 last 19.87) - open/close 13263 (time 1781233393.43 total 700.84 last 13.89) - open/close 13403 (time 1781233406.02 total 713.43 last 11.12) - open/close 13617 (time 1781233416.35 total 723.76 last 20.72) - open/close 13801 (time 1781233426.73 total 734.14 last 17.73) - open/close 13979 (time 1781233437.68 total 745.09 last 16.26) - open/close 14216 (time 1781233448.43 total 755.84 last 22.04) - open/close 14485 (time 1781233458.44 total 765.85 last 26.87) - open/close 14718 (time 1781233468.46 total 775.86 last 23.26) - open/close 14921 (time 1781233478.76 total 786.17 last 19.70) - open/close 15197 (time 1781233489.80 total 797.21 last 24.99) - open/close 15377 (time 1781233500.70 total 808.11 last 16.52) - open/close 15583 (time 1781233511.07 total 818.48 last 19.86) - open/close 15800 (time 1781233521.83 total 829.23 last 20.18) - open/close 16038 (time 1781233531.85 total 839.26 last 23.74) - open/close 16275 (time 1781233542.56 total 849.97 last 22.12) - open/close 16508 (time 1781233552.56 total 859.97 last 23.30) - open/close 16725 (time 1781233563.43 total 870.84 last 19.97) - open/close 16912 (time 1781233575.86 total 883.27 last 15.05) - open/close 17100 (time 1781233586.94 total 894.35 last 16.97) - open/close 17324 (time 1781233596.96 total 904.37 last 22.35) - open/close 17569 (time 1781233606.98 total 914.39 last 24.46) - open/close 17801 (time 1781233617.38 total 924.79 last 22.30) - open/close 18018 (time 1781233627.57 total 934.98 last 21.30) - open/close 18163 (time 1781233638.46 total 945.87 last 13.31) - open/close 18326 (time 1781233648.47 total 955.88 last 16.30) - open/close 18546 (time 1781233658.47 total 965.88 last 22.00) - open/close 18820 (time 1781233668.47 total 975.88 last 27.40) - open/close 19453 (time 1781233678.47 total 985.88 last 63.29) - open/close 20000 (time 1781233686.52 total 993.92 last 67.99) - open/close 20659 (time 1781233696.52 total 1003.93 last 65.89) - open/close 21313 (time 1781233706.53 total 1013.93 last 65.35) - open/close 21916 (time 1781233716.53 total 1023.94 last 60.26) - open/close 22581 (time 1781233726.54 total 1033.95 last 66.43) - open/close 23211 (time 1781233736.55 total 1043.96 last 62.94) - open/close 23809 (time 1781233746.57 total 1053.97 last 59.73) - open/close 24451 (time 1781233756.63 total 1064.03 last 63.82) - open/close 25056 (time 1781233766.63 total 1074.04 last 60.46) - open/close 25644 (time 1781233776.65 total 1084.05 last 58.72) - open/close 26365 (time 1781233786.65 total 1094.06 last 72.05) - open/close 26985 (time 1781233796.65 total 1104.06 last 61.99) - open/close 27624 (time 1781233806.67 total 1114.07 last 63.82) - open/close 28209 (time 1781233816.67 total 1124.08 last 58.50) - open/close 28872 (time 1781233826.68 total 1134.09 last 66.22) - open/close 29498 (time 1781233836.69 total 1144.09 last 62.56) - open/close 30000 (time 1781233844.27 total 1151.68 last 66.18) - open/close 30628 (time 1781233854.29 total 1161.70 last 62.68) - open/close 31223 (time 1781233864.30 total 1171.70 last 59.47) - open/close 31846 (time 1781233874.32 total 1181.73 last 62.17) - open/close 32493 (time 1781233884.33 total 1191.73 last 64.65) - open/close 33120 (time 1781233894.33 total 1201.74 last 62.66) - open/close 33800 (time 1781233904.34 total 1211.75 last 67.94) - open/close 34433 (time 1781233914.34 total 1221.75 last 63.30) - open/close 35102 (time 1781233924.36 total 1231.77 last 66.75) - open/close 35741 (time 1781233934.37 total 1241.78 last 63.85) - open/close 36413 (time 1781233944.38 total 1251.78 last 67.17) - open/close 37062 (time 1781233954.38 total 1261.79 last 64.88) - open/close 37688 (time 1781233964.39 total 1271.80 last 62.51) - open/close 38353 (time 1781233974.41 total 1281.81 last 66.42) - open/close 38972 (time 1781233984.42 total 1291.83 last 61.81) - open/close 39576 (time 1781233994.43 total 1301.84 last 60.34) - open/close 40000 (time 1781234001.22 total 1308.63 last 62.44) - open/close 40643 (time 1781234011.23 total 1318.64 last 64.22) - open/close 41317 (time 1781234021.26 total 1328.67 last 67.22) - open/close 41928 (time 1781234031.26 total 1338.67 last 61.09) - open/close 42632 (time 1781234041.27 total 1348.68 last 70.37) - open/close 43210 (time 1781234051.27 total 1358.68 last 57.78) - open/close 43823 (time 1781234061.29 total 1368.70 last 61.15) - open/close 44400 (time 1781234071.30 total 1378.71 last 57.69) - open/close 44818 (time 1781234081.32 total 1388.73 last 41.71) - open/close 45362 (time 1781234091.34 total 1398.75 last 54.29) - open/close 45973 (time 1781234101.35 total 1408.76 last 61.01) - open/close 46590 (time 1781234111.36 total 1418.77 last 61.64) - open/close 47232 (time 1781234121.38 total 1428.79 last 64.11) - open/close 47874 (time 1781234131.39 total 1438.80 last 64.11) - open/close 48473 (time 1781234141.42 total 1448.83 last 59.74) - open/close 49120 (time 1781234151.43 total 1458.84 last 64.60) - open/close 49707 (time 1781234161.44 total 1468.85 last 58.65) total: 50000 open/close in 1473.21 seconds: 33.94 ops/second - unlinked 0 (time 1781234170 ; total 0 ; last 0) - unlinked 10000 (time 1781234347 ; total 177 ; last 177) - unlinked 20000 (time 1781234525 ; total 355 ; last 178) - unlinked 30000 (time 1781234705 ; total 535 ; last 180) - unlinked 40000 (time 1781234885 ; total 715 ; last 180) total: 50000 unlinks in 885 seconds: 56.497173 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local oleg348-server: modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2