== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:59:06 (1785311946) start mds service on oleg106-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg106-server: detected 4 online CPUs by sysfs oleg106-server: MODOPTS_LIBCFS= oleg106-server: Force libcfs to create 2 CPU partitions oleg106-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg106-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg106-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1785312027.72 total 10.01 last 24.78) - open/close 483 (time 1785312038.49 total 20.78 last 21.82) - open/close 738 (time 1785312048.52 total 30.81 last 25.41) - open/close 954 (time 1785312059.49 total 41.77 last 19.70) - open/close 1072 (time 1785312069.51 total 51.80 last 11.77) - open/close 1285 (time 1785312079.65 total 61.94 last 21.02) - open/close 1493 (time 1785312090.34 total 72.63 last 19.44) - open/close 1779 (time 1785312100.34 total 82.63 last 28.60) - open/close 1910 (time 1785312110.35 total 92.64 last 13.10) - open/close 2112 (time 1785312120.37 total 102.66 last 20.16) - open/close 2357 (time 1785312131.27 total 113.56 last 22.48) - open/close 2560 (time 1785312141.43 total 123.72 last 19.98) - open/close 2735 (time 1785312151.43 total 133.72 last 17.50) - open/close 2861 (time 1785312161.44 total 143.72 last 12.59) - open/close 3182 (time 1785312171.93 total 154.22 last 30.59) - open/close 3329 (time 1785312182.54 total 164.83 last 13.85) - open/close 3529 (time 1785312193.30 total 175.59 last 18.59) - open/close 3758 (time 1785312203.31 total 185.60 last 22.88) - open/close 4018 (time 1785312213.37 total 195.66 last 25.85) - open/close 4294 (time 1785312223.37 total 205.66 last 27.59) - open/close 4515 (time 1785312233.43 total 215.72 last 21.97) - open/close 4768 (time 1785312243.44 total 225.73 last 25.28) - open/close 4953 (time 1785312254.07 total 236.36 last 17.41) - open/close 5106 (time 1785312264.08 total 246.37 last 15.28) - open/close 5373 (time 1785312274.40 total 256.68 last 25.88) - open/close 5606 (time 1785312284.70 total 266.99 last 22.62) - open/close 5743 (time 1785312294.70 total 276.99 last 13.70) - open/close 5952 (time 1785312304.74 total 287.03 last 20.82) - open/close 6209 (time 1785312315.69 total 297.98 last 23.47) - open/close 6431 (time 1785312326.43 total 308.72 last 20.66) - open/close 6609 (time 1785312336.44 total 318.73 last 17.80) - open/close 6826 (time 1785312346.45 total 328.74 last 21.68) - open/close 6913 (time 1785312356.52 total 338.81 last 8.64) - open/close 7155 (time 1785312367.06 total 349.35 last 22.95) - open/close 7411 (time 1785312377.76 total 360.04 last 23.94) - open/close 7627 (time 1785312388.31 total 370.60 last 20.47) - open/close 7841 (time 1785312398.33 total 380.62 last 21.34) - open/close 8119 (time 1785312408.37 total 390.66 last 27.70) - open/close 8334 (time 1785312418.37 total 400.66 last 21.49) - open/close 8583 (time 1785312428.37 total 410.66 last 24.89) - open/close 8844 (time 1785312438.49 total 420.78 last 25.80) - open/close 9076 (time 1785312450.80 total 433.09 last 18.85) - open/close 9323 (time 1785312460.80 total 443.09 last 24.70) - open/close 9581 (time 1785312470.84 total 453.13 last 25.70) - open/close 9714 (time 1785312480.85 total 463.14 last 13.28) - open/close 9935 (time 1785312490.88 total 473.17 last 22.05) - open/close 10000 (time 1785312493.27 total 475.55 last 27.23) - open/close 10239 (time 1785312503.40 total 485.68 last 23.59) - open/close 10444 (time 1785312513.50 total 495.79 last 20.28) - open/close 10662 (time 1785312524.26 total 506.55 last 20.26) - open/close 10920 (time 1785312534.26 total 516.55 last 25.79) - open/close 11148 (time 1785312544.55 total 526.84 last 22.16) - open/close 11352 (time 1785312554.57 total 536.86 last 20.37) - open/close 11592 (time 1785312564.84 total 547.13 last 23.36) - open/close 11846 (time 1785312575.62 total 557.91 last 23.57) - open/close 12138 (time 1785312585.62 total 567.91 last 29.20) - open/close 12382 (time 1785312595.69 total 577.97 last 24.24) - open/close 12585 (time 1785312605.69 total 587.98 last 20.30) - open/close 12932 (time 1785312615.70 total 597.98 last 34.67) - open/close 13143 (time 1785312625.95 total 608.23 last 20.59) - open/close 13366 (time 1785312636.95 total 619.23 last 20.27) - open/close 13621 (time 1785312647.83 total 630.12 last 23.43) - open/close 13820 (time 1785312658.01 total 640.30 last 19.54) - open/close 14048 (time 1785312668.03 total 650.32 last 22.77) - open/close 14288 (time 1785312678.03 total 660.32 last 24.00) - open/close 14520 (time 1785312688.73 total 671.02 last 21.67) - open/close 14738 (time 1785312699.55 total 681.84 last 20.15) - open/close 15029 (time 1785312709.59 total 691.88 last 29.00) - open/close 15218 (time 1785312720.10 total 702.39 last 17.97) - open/close 15487 (time 1785312730.53 total 712.82 last 25.81) - open/close 15752 (time 1785312740.53 total 722.82 last 26.50) - open/close 15956 (time 1785312750.53 total 732.82 last 20.40) - open/close 16183 (time 1785312760.53 total 742.82 last 22.69) - open/close 16421 (time 1785312770.55 total 752.84 last 23.77) - open/close 16648 (time 1785312781.46 total 763.75 last 20.80) - open/close 16941 (time 1785312792.28 total 774.57 last 27.09) - open/close 17148 (time 1785312802.78 total 785.07 last 19.72) - open/close 17409 (time 1785312813.67 total 795.96 last 23.96) - open/close 17654 (time 1785312824.09 total 806.38 last 23.52) - open/close 17901 (time 1785312834.10 total 816.39 last 24.68) - open/close 18117 (time 1785312844.76 total 827.05 last 20.26) - open/close 18308 (time 1785312854.88 total 837.17 last 18.87) - open/close 18505 (time 1785312864.89 total 847.18 last 19.67) - open/close 18684 (time 1785312875.03 total 857.32 last 17.66) - open/close 19401 (time 1785312885.04 total 867.32 last 71.64) - open/close 20000 (time 1785312891.54 total 873.83 last 92.14) - open/close 20931 (time 1785312901.55 total 883.84 last 92.98) - open/close 21699 (time 1785312911.56 total 893.85 last 76.70) - open/close 22138 (time 1785312921.57 total 903.86 last 43.85) - open/close 22575 (time 1785312931.58 total 913.87 last 43.66) - open/close 22843 (time 1785312941.60 total 923.89 last 26.76) - open/close 22927 (time 1785312951.62 total 933.91 last 8.38) - open/close 23235 (time 1785312962.31 total 944.60 last 28.82) - open/close 23497 (time 1785312972.33 total 954.62 last 26.15) - open/close 23701 (time 1785312982.34 total 964.63 last 20.37) - open/close 23908 (time 1785312992.36 total 974.65 last 20.67) - open/close 24166 (time 1785313002.37 total 984.66 last 25.77) - open/close 24441 (time 1785313012.37 total 994.66 last 27.49) - open/close 24731 (time 1785313022.38 total 1004.67 last 29.00) - open/close 25061 (time 1785313032.40 total 1014.69 last 32.92) - open/close 25538 (time 1785313042.44 total 1024.73 last 47.50) - open/close 26076 (time 1785313052.45 total 1034.74 last 53.75) - open/close 26684 (time 1785313062.48 total 1044.77 last 60.64) - open/close 27321 (time 1785313072.48 total 1054.77 last 63.67) - open/close 27856 (time 1785313082.49 total 1064.78 last 53.48) - open/close 28661 (time 1785313092.51 total 1074.80 last 80.32) - open/close 29255 (time 1785313102.53 total 1084.82 last 59.28) - open/close 29902 (time 1785313112.53 total 1094.82 last 64.67) - open/close 30582 (time 1785313122.55 total 1104.83 last 67.92) - open/close 31191 (time 1785313132.55 total 1114.84 last 60.87) - open/close 31925 (time 1785313142.55 total 1124.84 last 73.39) - open/close 32543 (time 1785313152.55 total 1134.84 last 61.79) - open/close 33276 (time 1785313162.55 total 1144.84 last 73.30) - open/close 33875 (time 1785313172.57 total 1154.86 last 59.82) - open/close 34545 (time 1785313182.59 total 1164.88 last 66.85) - open/close 35119 (time 1785313192.59 total 1174.88 last 57.40) - open/close 35834 (time 1785313202.59 total 1184.88 last 71.49) - open/close 36695 (time 1785313212.60 total 1194.89 last 86.02) - open/close 37132 (time 1785313222.61 total 1204.89 last 43.68) - open/close 37785 (time 1785313232.61 total 1214.90 last 65.28) - open/close 38528 (time 1785313242.61 total 1224.90 last 74.30) - open/close 39185 (time 1785313252.62 total 1234.91 last 65.64) - open/close 39598 (time 1785313262.63 total 1244.92 last 41.25) - open/close 40000 (time 1785313268.77 total 1251.06 last 65.42) - open/close 40692 (time 1785313278.77 total 1261.06 last 69.20) - open/close 41388 (time 1785313288.77 total 1271.06 last 69.60) - open/close 42089 (time 1785313298.79 total 1281.08 last 70.00) - open/close 42515 (time 1785313308.80 total 1291.09 last 42.54) - open/close 42981 (time 1785313318.84 total 1301.13 last 46.44) - open/close 43509 (time 1785313328.84 total 1311.13 last 52.76) - open/close 44315 (time 1785313338.85 total 1321.14 last 80.58) - open/close 44901 (time 1785313348.85 total 1331.14 last 58.59) - open/close 45582 (time 1785313358.85 total 1341.14 last 68.06) - open/close 46195 (time 1785313368.86 total 1351.15 last 61.28) - open/close 46817 (time 1785313378.86 total 1361.15 last 62.16) - open/close 47415 (time 1785313388.87 total 1371.16 last 59.79) - open/close 48054 (time 1785313398.87 total 1381.16 last 63.85) - open/close 48709 (time 1785313408.89 total 1391.18 last 65.42) - open/close 49258 (time 1785313418.91 total 1401.20 last 54.76) total: 50000 open/close in 1410.46 seconds: 35.45 ops/second - unlinked 0 (time 1785313433 ; total 0 ; last 0) - unlinked 10000 (time 1785313609 ; total 176 ; last 176) - unlinked 20000 (time 1785313770 ; total 337 ; last 161) - unlinked 30000 (time 1785313943 ; total 510 ; last 173) - unlinked 40000 (time 1785314123 ; total 690 ; last 180) total: 50000 unlinks in 840 seconds: 59.523811 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local oleg106-server: modules unloaded. pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2