== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:27:03 (1786062423) start mds service on oleg413-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing load_modules_local oleg413-server: Loading modules from /home/green/git/lustre-release/lustre oleg413-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg413-server: detected 4 online CPUs by sysfs oleg413-server: MODOPTS_LIBCFS= oleg413-server: Force libcfs to create 2 CPU partitions oleg413-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg413-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg413-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg413-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug -1 all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 161 (time 1786062525.80 total 10.02 last 16.06) - open/close 325 (time 1786062535.82 total 20.04 last 16.37) - open/close 435 (time 1786062545.84 total 30.06 last 10.98) - open/close 631 (time 1786062555.86 total 40.08 last 19.56) - open/close 731 (time 1786062565.91 total 50.13 last 9.95) - open/close 890 (time 1786062575.92 total 60.14 last 15.89) - open/close 957 (time 1786062588.40 total 72.62 last 5.37) - open/close 1065 (time 1786062599.37 total 83.60 last 9.84) - open/close 1261 (time 1786062609.67 total 93.90 last 19.03) - open/close 1335 (time 1786062619.69 total 103.92 last 7.38) - open/close 1451 (time 1786062630.15 total 114.37 last 11.09) - open/close 1545 (time 1786062640.64 total 124.87 last 8.96) - open/close 1654 (time 1786062651.93 total 136.15 last 9.66) - open/close 1822 (time 1786062662.36 total 146.58 last 16.11) - open/close 1928 (time 1786062672.36 total 156.58 last 10.60) - open/close 2058 (time 1786062684.23 total 168.45 last 10.96) - open/close 2216 (time 1786062695.13 total 179.36 last 14.49) - open/close 2357 (time 1786062709.17 total 193.39 last 10.05) - open/close 2479 (time 1786062719.63 total 203.85 last 11.66) - open/close 2672 (time 1786062729.64 total 213.86 last 19.28) - open/close 2843 (time 1786062739.65 total 223.87 last 17.08) - open/close 3023 (time 1786062750.23 total 234.46 last 17.01) - open/close 3214 (time 1786062760.79 total 245.02 last 18.09) - open/close 3382 (time 1786062770.96 total 255.18 last 16.53) - open/close 3487 (time 1786062781.92 total 266.15 last 9.57) - open/close 3536 (time 1786062792.98 total 277.20 last 4.43) - open/close 3715 (time 1786062803.03 total 287.25 last 17.82) - open/close 3869 (time 1786062813.04 total 297.26 last 15.38) - open/close 4076 (time 1786062823.05 total 307.27 last 20.68) - open/close 4222 (time 1786062833.72 total 317.95 last 13.68) - open/close 4401 (time 1786062844.12 total 328.34 last 17.22) - open/close 4550 (time 1786062854.13 total 338.35 last 14.89) - open/close 4674 (time 1786062864.34 total 348.56 last 12.15) - open/close 4813 (time 1786062875.20 total 359.42 last 12.80) - open/close 4925 (time 1786062886.30 total 370.52 last 10.09) - open/close 5136 (time 1786062898.13 total 382.35 last 17.84) - open/close 5268 (time 1786062908.62 total 392.84 last 12.59) - open/close 5446 (time 1786062918.74 total 402.96 last 17.58) - open/close 5644 (time 1786062930.12 total 414.35 last 17.39) - open/close 5796 (time 1786062940.35 total 424.58 last 14.86) - open/close 5996 (time 1786062951.04 total 435.26 last 18.72) - open/close 6199 (time 1786062961.07 total 445.30 last 20.22) - open/close 6428 (time 1786062971.79 total 456.01 last 21.38) - open/close 6629 (time 1786062983.39 total 467.61 last 17.32) - open/close 6764 (time 1786062993.41 total 477.64 last 13.47) - open/close 6847 (time 1786063004.06 total 488.29 last 7.79) - open/close 7015 (time 1786063014.09 total 498.32 last 16.75) - open/close 7169 (time 1786063024.09 total 508.32 last 15.40) - open/close 7380 (time 1786063035.18 total 519.40 last 19.03) - open/close 7487 (time 1786063045.66 total 529.88 last 10.21) - open/close 7579 (time 1786063056.53 total 540.75 last 8.46) - open/close 7663 (time 1786063066.55 total 550.77 last 8.39) - open/close 7777 (time 1786063076.55 total 560.78 last 11.39) - open/close 7883 (time 1786063086.57 total 570.79 last 10.59) - open/close 8092 (time 1786063096.58 total 580.81 last 20.86) - open/close 8329 (time 1786063107.22 total 591.45 last 22.28) - open/close 8460 (time 1786063117.82 total 602.05 last 12.36) - open/close 8535 (time 1786063128.15 total 612.37 last 7.27) - open/close 8728 (time 1786063138.57 total 622.80 last 18.51) - open/close 8901 (time 1786063148.60 total 632.83 last 17.25) - open/close 9093 (time 1786063159.43 total 643.65 last 17.73) - open/close 9218 (time 1786063169.66 total 653.88 last 12.22) - open/close 9289 (time 1786063180.00 total 664.22 last 6.87) - open/close 9421 (time 1786063190.09 total 674.31 last 13.08) - open/close 9569 (time 1786063200.19 total 684.42 last 14.64) - open/close 9716 (time 1786063211.03 total 695.25 last 13.56) - open/close 9811 (time 1786063222.13 total 706.36 last 8.55) - open/close 9891 (time 1786063232.16 total 716.39 last 7.98) - open/close 10000 (time 1786063240.21 total 724.44 last 13.54) - open/close 10156 (time 1786063250.22 total 734.44 last 15.59) - open/close 10291 (time 1786063260.23 total 744.45 last 13.49) - open/close 10453 (time 1786063270.59 total 754.82 last 15.63) - open/close 10538 (time 1786063281.48 total 765.71 last 7.81) - open/close 10639 (time 1786063291.53 total 775.76 last 10.05) - open/close 10776 (time 1786063301.72 total 785.95 last 13.44) - open/close 10876 (time 1786063311.84 total 796.07 last 9.88) - open/close 11017 (time 1786063321.85 total 806.07 last 14.10) - open/close 11169 (time 1786063331.86 total 816.09 last 15.17) - open/close 11275 (time 1786063342.24 total 826.47 last 10.21) - open/close 11401 (time 1786063352.40 total 836.62 last 12.41) - open/close 11558 (time 1786063364.27 total 848.49 last 13.23) - open/close 11733 (time 1786063374.28 total 858.51 last 17.47) - open/close 11834 (time 1786063384.83 total 869.06 last 9.57) - open/close 11946 (time 1786063395.24 total 879.46 last 10.77) - open/close 12044 (time 1786063405.25 total 889.47 last 9.79) - open/close 12191 (time 1786063415.26 total 899.49 last 14.68) - open/close 12284 (time 1786063430.30 total 914.52 last 6.19) - open/close 12408 (time 1786063440.45 total 924.68 last 12.21) - open/close 12510 (time 1786063451.08 total 935.30 last 9.60) - open/close 12658 (time 1786063461.10 total 945.32 last 14.78) - open/close 12833 (time 1786063471.20 total 955.42 last 17.33) - open/close 12944 (time 1786063481.20 total 965.42 last 11.10) - open/close 13071 (time 1786063491.34 total 975.57 last 12.52) - open/close 13181 (time 1786063501.77 total 985.99 last 10.55) - open/close 13322 (time 1786063512.78 total 997.00 last 12.81) - open/close 13422 (time 1786063522.79 total 1007.01 last 9.99) - open/close 13548 (time 1786063532.79 total 1017.01 last 12.60) - open/close 13717 (time 1786063543.20 total 1027.42 last 16.24) - open/close 13966 (time 1786063554.06 total 1038.29 last 22.92) - open/close 14120 (time 1786063564.07 total 1048.29 last 15.39) - open/close 14282 (time 1786063574.87 total 1059.09 last 15.00) - open/close 14396 (time 1786063587.14 total 1071.36 last 9.29) - open/close 14504 (time 1786063600.27 total 1084.49 last 8.23) - open/close 14616 (time 1786063610.27 total 1094.50 last 11.20) - open/close 14731 (time 1786063621.06 total 1105.29 last 10.66) - open/close 14912 (time 1786063631.07 total 1115.30 last 18.08) - open/close 15052 (time 1786063641.09 total 1125.31 last 13.98) - open/close 15225 (time 1786063651.59 total 1135.81 last 16.47) - open/close 15333 (time 1786063662.09 total 1146.31 last 10.29) - open/close 15514 (time 1786063672.09 total 1156.32 last 18.09) - open/close 15613 (time 1786063682.39 total 1166.61 last 9.62) - open/close 15748 (time 1786063693.22 total 1177.44 last 12.46) - open/close 15921 (time 1786063703.77 total 1187.99 last 16.41) - open/close 16209 (time 1786063713.77 total 1197.99 last 28.79) - open/close 16916 (time 1786063723.78 total 1208.00 last 70.65) - open/close 17611 (time 1786063733.79 total 1218.01 last 69.42) - open/close 18297 (time 1786063743.80 total 1228.03 last 68.50) - open/close 18994 (time 1786063753.81 total 1238.03 last 69.67) - open/close 19665 (time 1786063763.82 total 1248.04 last 67.02) - open/close 20000 (time 1786063768.62 total 1252.84 last 69.83) - open/close 20666 (time 1786063778.63 total 1262.85 last 66.52) - open/close 21360 (time 1786063788.63 total 1272.86 last 69.38) - open/close 22030 (time 1786063798.63 total 1282.86 last 67.00) - open/close 22705 (time 1786063808.64 total 1292.86 last 67.45) - open/close 23456 (time 1786063818.64 total 1302.87 last 75.08) - open/close 24203 (time 1786063828.65 total 1312.87 last 74.63) - open/close 24916 (time 1786063838.67 total 1322.89 last 71.16) - open/close 25607 (time 1786063848.67 total 1332.90 last 69.08) - open/close 26331 (time 1786063858.68 total 1342.91 last 72.34) - open/close 26994 (time 1786063868.69 total 1352.91 last 66.26) - open/close 27603 (time 1786063878.70 total 1362.92 last 60.84) - open/close 28287 (time 1786063888.70 total 1372.92 last 68.38) - open/close 28978 (time 1786063898.71 total 1382.93 last 69.06) - open/close 29675 (time 1786063908.72 total 1392.94 last 69.63) - open/close 30000 (time 1786063913.44 total 1397.66 last 68.87) - open/close 30671 (time 1786063923.44 total 1407.66 last 67.08) - open/close 31391 (time 1786063933.44 total 1417.66 last 71.98) - open/close 32082 (time 1786063943.44 total 1427.67 last 69.08) - open/close 32772 (time 1786063953.45 total 1437.67 last 68.95) - open/close 33443 (time 1786063963.46 total 1447.68 last 67.06) - open/close 34104 (time 1786063973.47 total 1457.70 last 66.00) - open/close 34806 (time 1786063983.49 total 1467.71 last 70.09) - open/close 35528 (time 1786063993.50 total 1477.72 last 72.14) - open/close 36235 (time 1786064003.50 total 1487.72 last 70.68) - open/close 36957 (time 1786064013.50 total 1497.73 last 72.18) - open/close 37658 (time 1786064023.51 total 1507.73 last 70.04) - open/close 38367 (time 1786064033.51 total 1517.74 last 70.89) - open/close 39081 (time 1786064043.53 total 1527.75 last 71.26) - open/close 39791 (time 1786064053.54 total 1537.77 last 70.92) - open/close 40000 (time 1786064056.38 total 1540.60 last 73.66) - open/close 40668 (time 1786064066.39 total 1550.61 last 66.73) - open/close 41348 (time 1786064076.41 total 1560.63 last 67.88) - open/close 42126 (time 1786064086.42 total 1570.64 last 77.70) - open/close 42758 (time 1786064096.43 total 1580.65 last 63.15) - open/close 43444 (time 1786064106.44 total 1590.66 last 68.53) - open/close 44129 (time 1786064116.45 total 1600.67 last 68.44) - open/close 44823 (time 1786064126.46 total 1610.68 last 69.34) - open/close 45500 (time 1786064136.47 total 1620.69 last 67.62) - open/close 46208 (time 1786064146.48 total 1630.70 last 70.73) - open/close 46720 (time 1786064156.48 total 1640.71 last 51.17) - open/close 47417 (time 1786064166.49 total 1650.71 last 69.65) - open/close 48119 (time 1786064176.51 total 1660.74 last 70.05) - open/close 48809 (time 1786064186.52 total 1670.75 last 68.93) - open/close 49453 (time 1786064196.53 total 1680.75 last 64.38) total: 50000 open/close in 1688.84 seconds: 29.61 ops/second - unlinked 0 (time 1786064208 ; total 0 ; last 0) - unlinked 10000 (time 1786064366 ; total 158 ; last 158) - unlinked 20000 (time 1786064524 ; total 316 ; last 158) - unlinked 30000 (time 1786064679 ; total 471 ; last 155) - unlinked 40000 (time 1786064837 ; total 629 ; last 158) total: 50000 unlinks in 788 seconds: 63.451778 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg413-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg413-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg413-server stop mds service on oleg413-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg413-server unloading modules via unload_modules_local on: 'oleg413-server' oleg413-server: oleg413-server.virtnet: executing unload_modules_local oleg413-server: modules unloaded. pdsh@oleg413-client: oleg413-client: ssh exited with exit code 2 pdsh@oleg413-client: oleg413-server: ssh exited with exit code 2 pdsh@oleg413-client: oleg413-client: ssh exited with exit code 2 pdsh@oleg413-client: oleg413-server: ssh exited with exit code 2