== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:04:14 (1787385854) start mds service on oleg138-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing load_modules_local oleg138-server: Loading modules from /home/green/git/lustre-release/lustre oleg138-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg138-server: detected 4 online CPUs by sysfs oleg138-server: MODOPTS_LIBCFS= oleg138-server: Force libcfs to create 2 CPU partitions oleg138-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg138-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg138-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg138-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug -1 all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 15 (time 1787385944.43 total 10.85 last 1.38) - open/close 264 (time 1787385956.43 total 22.85 last 20.75) - open/close 514 (time 1787385967.22 total 33.64 last 23.17) - open/close 642 (time 1787385977.23 total 43.65 last 12.80) - open/close 872 (time 1787385987.31 total 53.73 last 22.81) - open/close 1126 (time 1787385997.45 total 63.87 last 25.06) - open/close 1391 (time 1787386007.74 total 74.16 last 25.76) - open/close 1664 (time 1787386018.23 total 84.65 last 26.01) - open/close 1890 (time 1787386028.23 total 94.65 last 22.59) - open/close 1962 (time 1787386038.28 total 104.70 last 7.17) - open/close 2180 (time 1787386048.31 total 114.73 last 21.73) - open/close 2355 (time 1787386059.25 total 125.67 last 16.01) - open/close 2540 (time 1787386069.26 total 135.68 last 18.48) - open/close 2692 (time 1787386079.92 total 146.34 last 14.26) - open/close 2834 (time 1787386090.50 total 156.92 last 13.42) - open/close 2927 (time 1787386100.53 total 166.95 last 9.27) - open/close 3079 (time 1787386110.90 total 177.32 last 14.66) - open/close 3269 (time 1787386121.33 total 187.75 last 18.21) - open/close 3466 (time 1787386131.77 total 198.19 last 18.87) - open/close 3717 (time 1787386141.79 total 208.21 last 25.06) - open/close 3955 (time 1787386151.80 total 218.22 last 23.76) - open/close 4173 (time 1787386161.81 total 228.23 last 21.77) - open/close 4299 (time 1787386172.36 total 238.78 last 11.95) - open/close 4398 (time 1787386183.04 total 249.46 last 9.27) - open/close 4629 (time 1787386193.84 total 260.26 last 21.38) - open/close 4841 (time 1787386204.49 total 270.91 last 19.91) - open/close 5023 (time 1787386214.49 total 280.91 last 18.19) - open/close 5186 (time 1787386224.68 total 291.10 last 16.00) - open/close 5428 (time 1787386235.38 total 301.80 last 22.61) - open/close 5610 (time 1787386245.40 total 311.82 last 18.17) - open/close 5823 (time 1787386255.41 total 321.83 last 21.28) - open/close 6051 (time 1787386265.53 total 331.95 last 22.52) - open/close 6287 (time 1787386275.55 total 341.97 last 23.57) - open/close 6504 (time 1787386285.76 total 352.18 last 21.26) - open/close 6713 (time 1787386295.93 total 362.35 last 20.55) - open/close 6910 (time 1787386306.42 total 372.84 last 18.77) - open/close 7076 (time 1787386316.99 total 383.40 last 15.71) - open/close 7295 (time 1787386327.03 total 393.45 last 21.80) - open/close 7572 (time 1787386337.15 total 403.57 last 27.36) - open/close 7801 (time 1787386348.14 total 414.56 last 20.84) - open/close 8053 (time 1787386358.15 total 424.57 last 25.19) - open/close 8192 (time 1787386368.95 total 435.37 last 12.86) - open/close 8345 (time 1787386380.23 total 446.65 last 13.56) - open/close 8595 (time 1787386392.18 total 458.60 last 20.93) - open/close 8829 (time 1787386402.55 total 468.97 last 22.55) - open/close 9012 (time 1787386412.56 total 478.98 last 18.29) - open/close 9271 (time 1787386422.56 total 488.98 last 25.89) - open/close 9445 (time 1787386432.57 total 498.99 last 17.40) - open/close 9575 (time 1787386442.75 total 509.17 last 12.76) - open/close 9822 (time 1787386453.38 total 519.80 last 23.24) - open/close 9931 (time 1787386463.40 total 529.82 last 10.88) - open/close 10000 (time 1787386469.18 total 535.60 last 11.94) - open/close 10124 (time 1787386479.74 total 546.16 last 11.74) - open/close 10393 (time 1787386490.11 total 556.53 last 25.94) - open/close 10561 (time 1787386501.00 total 567.42 last 15.42) - open/close 10841 (time 1787386511.01 total 577.43 last 27.98) - open/close 10991 (time 1787386521.01 total 587.43 last 15.00) - open/close 11200 (time 1787386531.02 total 597.44 last 20.89) - open/close 11416 (time 1787386541.63 total 608.05 last 20.35) - open/close 11650 (time 1787386551.93 total 618.35 last 22.71) - open/close 11782 (time 1787386562.39 total 628.81 last 12.63) - open/close 12062 (time 1787386572.42 total 638.84 last 27.91) - open/close 12122 (time 1787386582.44 total 648.86 last 5.99) - open/close 12309 (time 1787386593.09 total 659.51 last 17.56) - open/close 12586 (time 1787386603.09 total 669.51 last 27.68) - open/close 12790 (time 1787386616.24 total 682.65 last 15.52) - open/close 13088 (time 1787386627.32 total 693.74 last 26.88) - open/close 13290 (time 1787386637.36 total 703.78 last 20.12) - open/close 13476 (time 1787386647.38 total 713.80 last 18.57) - open/close 13701 (time 1787386658.23 total 724.65 last 20.74) - open/close 13945 (time 1787386668.23 total 734.65 last 24.39) - open/close 14142 (time 1787386678.77 total 745.19 last 18.69) - open/close 14233 (time 1787386689.48 total 755.90 last 8.50) - open/close 14334 (time 1787386700.02 total 766.44 last 9.58) - open/close 14582 (time 1787386710.02 total 776.44 last 24.80) - open/close 14787 (time 1787386720.58 total 787.00 last 19.43) - open/close 15023 (time 1787386731.43 total 797.85 last 21.75) - open/close 15257 (time 1787386741.88 total 808.30 last 22.39) - open/close 15342 (time 1787386751.90 total 818.32 last 8.48) - open/close 15545 (time 1787386761.91 total 828.33 last 20.28) - open/close 15786 (time 1787386772.15 total 838.57 last 23.54) - open/close 16063 (time 1787386782.19 total 848.61 last 27.60) - open/close 16352 (time 1787386792.49 total 858.91 last 28.05) - open/close 16564 (time 1787386802.50 total 868.92 last 21.19) - open/close 16776 (time 1787386813.00 total 879.42 last 20.19) - open/close 17003 (time 1787386823.85 total 890.27 last 20.91) - open/close 17180 (time 1787386833.87 total 900.29 last 17.68) - open/close 17344 (time 1787386844.49 total 910.91 last 15.43) - open/close 17459 (time 1787386854.52 total 920.94 last 11.46) - open/close 17647 (time 1787386864.53 total 930.95 last 18.78) - open/close 17878 (time 1787386875.55 total 941.97 last 20.97) - open/close 18011 (time 1787386885.57 total 951.99 last 13.27) - open/close 18210 (time 1787386895.59 total 962.01 last 19.85) - open/close 18415 (time 1787386905.62 total 972.04 last 20.45) - open/close 18686 (time 1787386915.62 total 982.04 last 27.09) - open/close 18845 (time 1787386926.51 total 992.93 last 14.60) - open/close 18946 (time 1787386936.54 total 1002.96 last 10.07) - open/close 19405 (time 1787386946.55 total 1012.97 last 45.87) - open/close 20000 (time 1787386953.85 total 1020.27 last 81.43) - open/close 20755 (time 1787386963.86 total 1030.28 last 75.46) - open/close 21504 (time 1787386973.87 total 1040.29 last 74.81) - open/close 22344 (time 1787386983.87 total 1050.29 last 83.99) - open/close 23065 (time 1787386993.88 total 1060.30 last 72.05) - open/close 23740 (time 1787387003.89 total 1070.31 last 67.46) - open/close 24305 (time 1787387013.92 total 1080.34 last 56.32) - open/close 25010 (time 1787387023.93 total 1090.35 last 70.43) - open/close 25706 (time 1787387033.93 total 1100.35 last 69.59) - open/close 26453 (time 1787387043.93 total 1110.35 last 74.67) - open/close 26948 (time 1787387053.94 total 1120.36 last 49.49) - open/close 27587 (time 1787387063.94 total 1130.36 last 63.87) - open/close 28176 (time 1787387073.95 total 1140.37 last 58.82) - open/close 28763 (time 1787387083.96 total 1150.38 last 58.66) - open/close 29456 (time 1787387093.99 total 1160.41 last 69.14) - open/close 30000 (time 1787387101.62 total 1168.04 last 71.22) - open/close 30669 (time 1787387111.62 total 1178.04 last 66.89) - open/close 31460 (time 1787387121.63 total 1188.05 last 79.03) - open/close 32162 (time 1787387131.64 total 1198.06 last 70.18) - open/close 32881 (time 1787387141.65 total 1208.07 last 71.77) - open/close 33662 (time 1787387151.66 total 1218.08 last 78.03) - open/close 34325 (time 1787387161.67 total 1228.09 last 66.26) - open/close 34939 (time 1787387171.68 total 1238.10 last 61.35) - open/close 35610 (time 1787387181.69 total 1248.11 last 67.00) - open/close 36315 (time 1787387191.69 total 1258.11 last 70.50) - open/close 37213 (time 1787387201.69 total 1268.11 last 89.80) - open/close 38025 (time 1787387211.70 total 1278.12 last 81.14) - open/close 38759 (time 1787387221.70 total 1288.12 last 73.38) - open/close 39506 (time 1787387231.71 total 1298.13 last 74.67) - open/close 40000 (time 1787387239.23 total 1305.65 last 65.69) - open/close 40717 (time 1787387249.23 total 1315.65 last 71.68) - open/close 41506 (time 1787387259.23 total 1325.65 last 78.89) - open/close 42206 (time 1787387269.23 total 1335.65 last 69.99) - open/close 42897 (time 1787387279.23 total 1345.65 last 69.09) - open/close 43564 (time 1787387289.24 total 1355.66 last 66.68) - open/close 44242 (time 1787387299.24 total 1365.66 last 67.76) - open/close 44939 (time 1787387309.25 total 1375.67 last 69.62) - open/close 45625 (time 1787387319.26 total 1385.68 last 68.57) - open/close 46323 (time 1787387329.27 total 1395.69 last 69.72) - open/close 47097 (time 1787387339.28 total 1405.70 last 77.31) - open/close 47797 (time 1787387349.28 total 1415.70 last 69.99) - open/close 48493 (time 1787387359.30 total 1425.72 last 69.52) - open/close 49364 (time 1787387369.31 total 1435.73 last 86.98) total: 50000 open/close in 1444.40 seconds: 34.62 ops/second - unlinked 0 (time 1787387384 ; total 0 ; last 0) - unlinked 10000 (time 1787387539 ; total 155 ; last 155) - unlinked 20000 (time 1787387700 ; total 316 ; last 161) - unlinked 30000 (time 1787387842 ; total 458 ; last 142) - unlinked 40000 (time 1787388020 ; total 636 ; last 178) total: 50000 unlinks in 793 seconds: 63.051701 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg138-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg138-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg138-server stop mds service on oleg138-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg138-server unloading modules via unload_modules_local on: 'oleg138-server' oleg138-server: oleg138-server.virtnet: executing unload_modules_local oleg138-server: modules unloaded. pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-server: ssh exited with exit code 2 pdsh@oleg138-client: oleg138-client: ssh exited with exit code 2