== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:31:35 (1787844695) start mds service on oleg157-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing load_modules_local oleg157-server: Loading modules from /home/green/git/lustre-release/lustre oleg157-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg157-server: detected 4 online CPUs by sysfs oleg157-server: MODOPTS_LIBCFS= oleg157-server: Force libcfs to create 2 CPU partitions oleg157-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg157-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg157-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg157-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg157-server: oleg157-server.virtnet: executing set_default_debug -1 all pdsh@oleg157-client: oleg157-server: ssh exited with exit code 1 Started lustre-OST0000 oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg157-client.virtnet: -o user_xattr,flock 192.168.201.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1787844787.53 total 10.32 last 10.18) - open/close 293 (time 1787844797.65 total 20.44 last 18.57) - open/close 531 (time 1787844809.75 total 32.54 last 19.68) - open/close 602 (time 1787844820.74 total 43.53 last 6.46) - open/close 716 (time 1787844830.75 total 53.54 last 11.39) - open/close 859 (time 1787844841.28 total 64.06 last 13.58) - open/close 1075 (time 1787844852.05 total 74.84 last 20.04) - open/close 1235 (time 1787844862.06 total 84.85 last 15.99) - open/close 1425 (time 1787844872.07 total 94.86 last 18.98) - open/close 1526 (time 1787844883.02 total 105.80 last 9.23) - open/close 1655 (time 1787844893.19 total 115.97 last 12.68) - open/close 1844 (time 1787844903.19 total 125.98 last 18.89) - open/close 2086 (time 1787844913.66 total 136.45 last 23.12) - open/close 2277 (time 1787844924.23 total 147.02 last 18.07) - open/close 2470 (time 1787844934.23 total 157.02 last 19.29) - open/close 2642 (time 1787844944.25 total 167.03 last 17.18) - open/close 2801 (time 1787844954.52 total 177.31 last 15.47) - open/close 2974 (time 1787844965.20 total 187.98 last 16.21) - open/close 3162 (time 1787844975.84 total 198.63 last 17.67) - open/close 3298 (time 1787844985.89 total 208.67 last 13.54) - open/close 3506 (time 1787844996.98 total 219.77 last 18.74) - open/close 3692 (time 1787845007.87 total 230.66 last 17.08) - open/close 3923 (time 1787845018.58 total 241.37 last 21.58) - open/close 3982 (time 1787845030.20 total 252.99 last 5.08) - open/close 4172 (time 1787845040.22 total 263.01 last 18.96) - open/close 4329 (time 1787845050.63 total 273.42 last 15.08) - open/close 4488 (time 1787845061.13 total 283.92 last 15.14) - open/close 4674 (time 1787845071.35 total 294.14 last 18.21) - open/close 4837 (time 1787845081.54 total 304.33 last 16.00) - open/close 5024 (time 1787845092.52 total 315.31 last 17.03) - open/close 5162 (time 1787845103.07 total 325.86 last 13.08) - open/close 5268 (time 1787845113.07 total 335.86 last 10.60) - open/close 5404 (time 1787845123.83 total 346.61 last 12.65) - open/close 5660 (time 1787845135.95 total 358.73 last 21.12) - open/close 5894 (time 1787845145.97 total 368.76 last 23.34) - open/close 6104 (time 1787845156.56 total 379.35 last 19.83) - open/close 6296 (time 1787845166.58 total 389.37 last 19.17) - open/close 6430 (time 1787845177.61 total 400.39 last 12.15) - open/close 6702 (time 1787845187.61 total 410.40 last 27.19) - open/close 6875 (time 1787845198.62 total 421.41 last 15.71) - open/close 7062 (time 1787845208.75 total 431.54 last 18.47) - open/close 7300 (time 1787845219.76 total 442.54 last 21.62) - open/close 7424 (time 1787845230.08 total 452.86 last 12.02) - open/close 7553 (time 1787845240.30 total 463.09 last 12.62) - open/close 7600 (time 1787845250.97 total 473.76 last 4.40) - open/close 7745 (time 1787845260.98 total 483.77 last 14.49) - open/close 7900 (time 1787845271.02 total 493.81 last 15.43) - open/close 8140 (time 1787845283.23 total 506.02 last 19.65) - open/close 8257 (time 1787845294.24 total 517.03 last 10.63) - open/close 8486 (time 1787845305.04 total 527.83 last 21.20) - open/close 8545 (time 1787845315.80 total 538.59 last 5.48) - open/close 8622 (time 1787845325.96 total 548.75 last 7.58) - open/close 8848 (time 1787845335.96 total 558.75 last 22.60) - open/close 8988 (time 1787845346.09 total 568.88 last 13.83) - open/close 9276 (time 1787845356.96 total 579.75 last 26.50) - open/close 9416 (time 1787845366.97 total 589.75 last 13.99) - open/close 9544 (time 1787845377.54 total 600.33 last 12.11) - open/close 9673 (time 1787845388.04 total 610.83 last 12.28) - open/close 9916 (time 1787845398.80 total 621.58 last 22.60) - open/close 10000 (time 1787845404.26 total 627.05 last 15.37) - open/close 10138 (time 1787845414.99 total 637.78 last 12.87) - open/close 10249 (time 1787845425.64 total 648.42 last 10.42) - open/close 10418 (time 1787845436.37 total 659.15 last 15.75) - open/close 10488 (time 1787845446.67 total 669.46 last 6.79) - open/close 10587 (time 1787845457.16 total 679.95 last 9.44) - open/close 10763 (time 1787845467.22 total 690.01 last 17.49) - open/close 10853 (time 1787845477.45 total 700.23 last 8.80) - open/close 11085 (time 1787845487.46 total 710.25 last 23.16) - open/close 11193 (time 1787845498.15 total 720.94 last 10.10) - open/close 11365 (time 1787845508.90 total 731.69 last 16.00) - open/close 11515 (time 1787845519.83 total 742.62 last 13.72) - open/close 11642 (time 1787845529.87 total 752.65 last 12.66) - open/close 11832 (time 1787845539.87 total 762.66 last 18.98) - open/close 11941 (time 1787845550.42 total 773.20 last 10.34) - open/close 12175 (time 1787845560.43 total 783.22 last 23.35) - open/close 12448 (time 1787845570.45 total 793.24 last 27.26) - open/close 12679 (time 1787845580.48 total 803.26 last 23.04) - open/close 12806 (time 1787845591.34 total 814.12 last 11.70) - open/close 12926 (time 1787845601.61 total 824.40 last 11.68) - open/close 13160 (time 1787845611.62 total 834.41 last 23.38) - open/close 13321 (time 1787845623.05 total 845.84 last 14.08) - open/close 13519 (time 1787845633.06 total 855.84 last 19.79) - open/close 13702 (time 1787845643.47 total 866.26 last 17.57) - open/close 13943 (time 1787845653.48 total 876.27 last 24.09) - open/close 14118 (time 1787845663.48 total 886.27 last 17.49) - open/close 14222 (time 1787845673.77 total 896.56 last 10.11) - open/close 14458 (time 1787845683.78 total 906.57 last 23.57) - open/close 14615 (time 1787845694.10 total 916.89 last 15.21) - open/close 14875 (time 1787845704.65 total 927.43 last 24.66) - open/close 15105 (time 1787845715.29 total 938.08 last 21.61) - open/close 15203 (time 1787845725.30 total 948.09 last 9.79) - open/close 15283 (time 1787845735.37 total 958.15 last 7.95) - open/close 15379 (time 1787845745.38 total 968.16 last 9.59) - open/close 15499 (time 1787845758.48 total 981.27 last 9.16) - open/close 15711 (time 1787845769.18 total 991.97 last 19.81) - open/close 15922 (time 1787845779.20 total 1001.98 last 21.07) - open/close 16102 (time 1787845789.51 total 1012.30 last 17.45) - open/close 16252 (time 1787845799.52 total 1022.31 last 14.99) - open/close 16352 (time 1787845809.54 total 1032.33 last 9.98) - open/close 16379 (time 1787845819.79 total 1042.58 last 2.63) - open/close 16428 (time 1787845830.41 total 1053.20 last 4.61) - open/close 16602 (time 1787845840.42 total 1063.21 last 17.38) - open/close 17161 (time 1787845850.42 total 1073.21 last 55.90) - open/close 17828 (time 1787845860.43 total 1083.22 last 66.65) - open/close 18573 (time 1787845870.44 total 1093.23 last 74.40) - open/close 19327 (time 1787845880.45 total 1103.24 last 75.33) - open/close 20000 (time 1787845888.96 total 1111.75 last 79.08) - open/close 20713 (time 1787845898.97 total 1121.76 last 71.24) - open/close 21438 (time 1787845908.97 total 1131.76 last 72.47) - open/close 22150 (time 1787845918.99 total 1141.77 last 71.10) - open/close 22848 (time 1787845928.99 total 1151.78 last 69.75) - open/close 23564 (time 1787845939.00 total 1161.79 last 71.57) - open/close 24245 (time 1787845949.00 total 1171.79 last 68.09) - open/close 25025 (time 1787845959.00 total 1181.79 last 77.99) - open/close 25826 (time 1787845969.05 total 1191.83 last 79.75) - open/close 26520 (time 1787845979.06 total 1201.85 last 69.32) - open/close 27356 (time 1787845989.07 total 1211.86 last 83.46) - open/close 28120 (time 1787845999.11 total 1221.90 last 76.13) - open/close 28824 (time 1787846009.11 total 1231.90 last 70.39) - open/close 29532 (time 1787846019.12 total 1241.91 last 70.76) - open/close 30000 (time 1787846025.52 total 1248.31 last 73.13) - open/close 30816 (time 1787846035.52 total 1258.31 last 81.54) - open/close 31573 (time 1787846045.53 total 1268.32 last 75.64) - open/close 32245 (time 1787846055.53 total 1278.32 last 67.19) - open/close 33090 (time 1787846065.54 total 1288.33 last 84.42) - open/close 33849 (time 1787846075.55 total 1298.34 last 75.85) - open/close 34647 (time 1787846085.56 total 1308.34 last 79.76) - open/close 35452 (time 1787846095.56 total 1318.35 last 80.48) - open/close 36240 (time 1787846105.56 total 1328.35 last 78.79) - open/close 37012 (time 1787846115.56 total 1338.35 last 77.17) - open/close 37618 (time 1787846125.59 total 1348.37 last 60.47) - open/close 38217 (time 1787846135.60 total 1358.38 last 59.83) - open/close 38963 (time 1787846145.60 total 1368.39 last 74.59) - open/close 39690 (time 1787846155.60 total 1378.39 last 72.70) - open/close 40000 (time 1787846159.85 total 1382.64 last 72.95) - open/close 40825 (time 1787846169.85 total 1392.64 last 82.45) - open/close 41521 (time 1787846179.85 total 1402.64 last 69.59) - open/close 42149 (time 1787846189.86 total 1412.65 last 62.74) - open/close 42911 (time 1787846199.89 total 1422.68 last 76.00) - open/close 43629 (time 1787846209.90 total 1432.69 last 71.75) - open/close 44362 (time 1787846219.90 total 1442.69 last 73.29) - open/close 45222 (time 1787846229.90 total 1452.69 last 85.98) - open/close 46058 (time 1787846239.91 total 1462.70 last 83.54) - open/close 46887 (time 1787846249.92 total 1472.71 last 82.81) - open/close 47540 (time 1787846259.92 total 1482.71 last 65.28) - open/close 48359 (time 1787846269.93 total 1492.72 last 81.86) - open/close 49081 (time 1787846279.93 total 1502.72 last 72.17) - open/close 49888 (time 1787846289.93 total 1512.72 last 80.69) total: 50000 open/close in 1514.13 seconds: 33.02 ops/second - unlinked 0 (time 1787846295 ; total 0 ; last 0) - unlinked 10000 (time 1787846432 ; total 137 ; last 137) - unlinked 20000 (time 1787846584 ; total 289 ; last 152) - unlinked 30000 (time 1787846742 ; total 447 ; last 158) - unlinked 40000 (time 1787846892 ; total 597 ; last 150) total: 50000 unlinks in 728 seconds: 68.681320 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg157-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg157-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg157-server stop mds service on oleg157-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg157-server unloading modules via unload_modules_local on: 'oleg157-server' oleg157-server: oleg157-server.virtnet: executing unload_modules_local oleg157-server: modules unloaded. pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-server: ssh exited with exit code 2 pdsh@oleg157-client: oleg157-client: ssh exited with exit code 2