== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:02:10 (1787785330) start mds service on oleg659-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing load_modules_local oleg659-server: Loading modules from /home/green/git/lustre-release/lustre oleg659-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg659-server: detected 4 online CPUs by sysfs oleg659-server: MODOPTS_LIBCFS= oleg659-server: Force libcfs to create 2 CPU partitions oleg659-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg659-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg659-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg659-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg659-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg659-server: oleg659-server.virtnet: executing set_default_debug -1 all pdsh@oleg659-client: oleg659-server: ssh exited with exit code 1 Started lustre-OST0000 oleg659-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg659-client.virtnet: -o user_xattr,flock 192.168.206.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1787785407.89 total 10.00 last 20.40) - open/close 432 (time 1787785418.38 total 20.50 last 21.72) - open/close 678 (time 1787785428.95 total 31.06 last 23.28) - open/close 889 (time 1787785438.96 total 41.07 last 21.08) - open/close 1161 (time 1787785448.96 total 51.07 last 27.19) - open/close 1393 (time 1787785459.35 total 61.46 last 22.33) - open/close 1661 (time 1787785470.36 total 72.47 last 24.35) - open/close 1865 (time 1787785480.36 total 82.47 last 20.40) - open/close 2100 (time 1787785490.51 total 92.62 last 23.15) - open/close 2310 (time 1787785500.88 total 102.99 last 20.25) - open/close 2505 (time 1787785511.18 total 113.29 last 18.92) - open/close 2730 (time 1787785521.48 total 123.60 last 21.84) - open/close 2991 (time 1787785531.87 total 133.98 last 25.13) - open/close 3257 (time 1787785542.86 total 144.97 last 24.20) - open/close 3478 (time 1787785553.62 total 155.73 last 20.55) - open/close 3653 (time 1787785564.49 total 166.60 last 16.09) - open/close 3892 (time 1787785575.50 total 177.61 last 21.71) - open/close 4099 (time 1787785586.52 total 188.64 last 18.78) - open/close 4361 (time 1787785596.53 total 198.65 last 26.17) - open/close 4476 (time 1787785607.38 total 209.49 last 10.61) - open/close 4708 (time 1787785618.00 total 220.11 last 21.84) - open/close 4964 (time 1787785628.00 total 230.11 last 25.60) - open/close 5146 (time 1787785638.00 total 240.11 last 18.20) - open/close 5419 (time 1787785648.35 total 250.46 last 26.39) - open/close 5567 (time 1787785658.59 total 260.70 last 14.45) - open/close 5785 (time 1787785669.26 total 271.37 last 20.42) - open/close 5865 (time 1787785679.31 total 281.42 last 7.96) - open/close 6120 (time 1787785690.14 total 292.25 last 23.53) - open/close 6359 (time 1787785700.15 total 302.27 last 23.87) - open/close 6556 (time 1787785710.17 total 312.28 last 19.67) - open/close 6846 (time 1787785721.18 total 323.29 last 26.33) - open/close 7016 (time 1787785731.87 total 333.98 last 15.91) - open/close 7098 (time 1787785742.90 total 345.01 last 7.43) - open/close 7269 (time 1787785753.57 total 355.68 last 16.03) - open/close 7357 (time 1787785763.93 total 366.04 last 8.49) - open/close 7611 (time 1787785774.31 total 376.42 last 24.47) - open/close 7748 (time 1787785785.17 total 387.28 last 12.62) - open/close 7962 (time 1787785795.43 total 397.54 last 20.85) - open/close 8164 (time 1787785806.43 total 408.54 last 18.36) - open/close 8324 (time 1787785816.67 total 418.78 last 15.63) - open/close 8516 (time 1787785826.68 total 428.79 last 19.18) - open/close 8745 (time 1787785837.46 total 439.58 last 21.24) - open/close 8934 (time 1787785849.85 total 451.96 last 15.26) - open/close 9097 (time 1787785860.63 total 462.75 last 15.11) - open/close 9232 (time 1787785870.87 total 472.98 last 13.19) - open/close 9421 (time 1787785880.88 total 482.99 last 18.88) - open/close 9625 (time 1787785891.54 total 493.65 last 19.13) - open/close 9926 (time 1787785901.54 total 503.65 last 30.10) - open/close 10000 (time 1787785903.76 total 505.88 last 33.31) - open/close 10229 (time 1787785913.77 total 515.89 last 22.88) - open/close 10492 (time 1787785923.78 total 525.89 last 26.28) - open/close 10599 (time 1787785934.04 total 536.15 last 10.43) - open/close 10770 (time 1787785944.21 total 546.32 last 16.81) - open/close 10984 (time 1787785954.22 total 556.33 last 21.38) - open/close 11207 (time 1787785964.23 total 566.34 last 22.28) - open/close 11425 (time 1787785974.24 total 576.35 last 21.77) - open/close 11667 (time 1787785984.87 total 586.99 last 22.76) - open/close 11889 (time 1787785994.89 total 597.00 last 22.17) - open/close 12103 (time 1787786005.64 total 607.75 last 19.90) - open/close 12367 (time 1787786017.87 total 619.99 last 21.58) - open/close 12639 (time 1787786027.90 total 630.02 last 27.12) - open/close 12783 (time 1787786037.92 total 640.03 last 14.38) - open/close 12978 (time 1787786047.94 total 650.05 last 19.47) - open/close 13201 (time 1787786058.23 total 660.34 last 21.67) - open/close 13464 (time 1787786068.35 total 670.46 last 25.98) - open/close 13721 (time 1787786078.38 total 680.50 last 25.62) - open/close 13959 (time 1787786088.91 total 691.02 last 22.61) - open/close 14214 (time 1787786098.93 total 701.04 last 25.44) - open/close 14427 (time 1787786109.01 total 711.12 last 21.13) - open/close 14628 (time 1787786119.37 total 721.48 last 19.40) - open/close 14847 (time 1787786129.38 total 731.49 last 21.88) - open/close 15075 (time 1787786139.39 total 741.50 last 22.79) - open/close 15295 (time 1787786149.39 total 751.50 last 22.00) - open/close 15524 (time 1787786159.41 total 761.52 last 22.84) - open/close 15791 (time 1787786169.42 total 771.53 last 26.69) - open/close 15993 (time 1787786179.42 total 781.53 last 20.19) - open/close 16159 (time 1787786192.09 total 794.20 last 13.10) - open/close 16375 (time 1787786202.09 total 804.20 last 21.59) - open/close 16592 (time 1787786213.03 total 815.14 last 19.83) - open/close 16843 (time 1787786223.43 total 825.55 last 24.13) - open/close 17054 (time 1787786234.06 total 836.18 last 19.85) - open/close 17275 (time 1787786244.07 total 846.18 last 22.08) - open/close 17481 (time 1787786254.08 total 856.19 last 20.58) - open/close 17756 (time 1787786264.92 total 867.03 last 25.37) - open/close 18020 (time 1787786275.48 total 877.59 last 25.01) - open/close 18251 (time 1787786286.30 total 888.41 last 21.35) - open/close 18485 (time 1787786297.37 total 899.48 last 21.13) - open/close 18718 (time 1787786307.37 total 909.48 last 23.30) - open/close 19216 (time 1787786317.38 total 919.49 last 49.76) - open/close 19929 (time 1787786327.42 total 929.53 last 70.99) - open/close 20690 (time 1787786337.43 total 939.54 last 76.06) - open/close 21462 (time 1787786347.43 total 949.55 last 77.15) - open/close 22174 (time 1787786357.44 total 959.55 last 71.16) - open/close 22896 (time 1787786367.45 total 969.56 last 72.16) - open/close 23659 (time 1787786377.45 total 979.57 last 76.24) - open/close 24368 (time 1787786387.46 total 989.57 last 70.86) - open/close 25081 (time 1787786397.46 total 999.57 last 71.28) - open/close 25795 (time 1787786407.47 total 1009.58 last 71.35) - open/close 26532 (time 1787786417.48 total 1019.59 last 73.66) - open/close 27274 (time 1787786427.49 total 1029.60 last 74.12) - open/close 27989 (time 1787786437.49 total 1039.60 last 71.48) - open/close 28642 (time 1787786447.50 total 1049.61 last 65.23) - open/close 29255 (time 1787786457.50 total 1059.61 last 61.29) - open/close 29791 (time 1787786467.51 total 1069.62 last 53.55) - open/close 30000 (time 1787786470.75 total 1072.86 last 64.58) - open/close 30698 (time 1787786480.76 total 1082.87 last 69.72) - open/close 31326 (time 1787786490.76 total 1092.88 last 62.75) - open/close 32003 (time 1787786500.77 total 1102.88 last 67.69) - open/close 32750 (time 1787786510.77 total 1112.89 last 74.65) - open/close 33462 (time 1787786520.78 total 1122.89 last 71.17) - open/close 34162 (time 1787786530.79 total 1132.90 last 69.92) - open/close 34858 (time 1787786540.79 total 1142.90 last 69.60) - open/close 35579 (time 1787786550.80 total 1152.91 last 72.04) - open/close 36320 (time 1787786560.82 total 1162.93 last 73.98) - open/close 36868 (time 1787786570.82 total 1172.93 last 54.80) - open/close 37519 (time 1787786580.83 total 1182.94 last 65.03) - open/close 38175 (time 1787786590.83 total 1192.94 last 65.59) - open/close 38875 (time 1787786600.83 total 1202.94 last 69.99) - open/close 39505 (time 1787786610.83 total 1212.95 last 62.97) - open/close 40000 (time 1787786618.21 total 1220.32 last 67.11) - open/close 40682 (time 1787786628.22 total 1230.33 last 68.11) - open/close 41359 (time 1787786638.23 total 1240.34 last 67.65) - open/close 42062 (time 1787786648.24 total 1250.35 last 70.27) - open/close 42799 (time 1787786658.24 total 1260.35 last 73.64) - open/close 43487 (time 1787786668.25 total 1270.36 last 68.74) - open/close 44207 (time 1787786678.26 total 1280.37 last 71.96) - open/close 44913 (time 1787786688.26 total 1290.37 last 70.56) - open/close 45663 (time 1787786698.27 total 1300.38 last 74.94) - open/close 46344 (time 1787786708.29 total 1310.40 last 67.99) - open/close 47076 (time 1787786718.29 total 1320.40 last 73.17) - open/close 47747 (time 1787786728.29 total 1330.40 last 67.08) - open/close 48410 (time 1787786738.30 total 1340.41 last 66.27) - open/close 49066 (time 1787786748.31 total 1350.42 last 65.54) - open/close 49751 (time 1787786758.32 total 1360.43 last 68.40) total: 50000 open/close in 1363.60 seconds: 36.67 ops/second - unlinked 0 (time 1787786765 ; total 0 ; last 0) - unlinked 10000 (time 1787786915 ; total 150 ; last 150) - unlinked 20000 (time 1787787060 ; total 295 ; last 145) - unlinked 30000 (time 1787787204 ; total 439 ; last 144) - unlinked 40000 (time 1787787336 ; total 571 ; last 132) total: 50000 unlinks in 700 seconds: 71.428574 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg659-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg659-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg659-server stop mds service on oleg659-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg659-server unloading modules via unload_modules_local on: 'oleg659-server' oleg659-server: oleg659-server.virtnet: executing unload_modules_local oleg659-server: modules unloaded. pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-server: ssh exited with exit code 2 pdsh@oleg659-client: oleg659-client: ssh exited with exit code 2