== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:38:19 (1779313099) start mds service on oleg425-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg425-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg425-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg425-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1779313181.68 total 10.02 last 22.06) - open/close 432 (time 1779313191.74 total 20.08 last 20.96) - open/close 652 (time 1779313202.69 total 31.03 last 20.10) - open/close 862 (time 1779313213.17 total 41.51 last 20.03) - open/close 1053 (time 1779313223.42 total 51.76 last 18.64) - open/close 1289 (time 1779313233.92 total 62.26 last 22.47) - open/close 1532 (time 1779313244.42 total 72.76 last 23.15) - open/close 1760 (time 1779313254.43 total 82.77 last 22.78) - open/close 1884 (time 1779313264.97 total 93.31 last 11.76) - open/close 2013 (time 1779313274.98 total 103.32 last 12.88) - open/close 2227 (time 1779313285.64 total 113.98 last 20.07) - open/close 2449 (time 1779313295.66 total 124.00 last 22.16) - open/close 2661 (time 1779313305.67 total 134.01 last 21.18) - open/close 2906 (time 1779313315.90 total 144.24 last 23.95) - open/close 3083 (time 1779313325.91 total 154.25 last 17.69) - open/close 3238 (time 1779313335.92 total 164.26 last 15.47) - open/close 3413 (time 1779313346.81 total 175.15 last 16.08) - open/close 3596 (time 1779313356.81 total 185.15 last 18.30) - open/close 3779 (time 1779313367.67 total 196.01 last 16.85) - open/close 3999 (time 1779313378.78 total 207.12 last 19.79) - open/close 4209 (time 1779313388.80 total 217.14 last 20.96) - open/close 4403 (time 1779313399.81 total 228.15 last 17.63) - open/close 4619 (time 1779313410.32 total 238.66 last 20.54) - open/close 4813 (time 1779313420.34 total 248.67 last 19.37) - open/close 5062 (time 1779313430.34 total 258.68 last 24.89) - open/close 5316 (time 1779313440.36 total 268.70 last 25.34) - open/close 5551 (time 1779313451.25 total 279.59 last 21.58) - open/close 5718 (time 1779313461.31 total 289.65 last 16.61) - open/close 5930 (time 1779313472.19 total 300.53 last 19.48) - open/close 6137 (time 1779313482.27 total 310.61 last 20.54) - open/close 6366 (time 1779313492.64 total 320.98 last 22.08) - open/close 6545 (time 1779313503.71 total 332.04 last 16.18) - open/close 6651 (time 1779313514.52 total 342.86 last 9.80) - open/close 6776 (time 1779313524.53 total 352.87 last 12.49) - open/close 6967 (time 1779313534.54 total 362.88 last 19.07) - open/close 7166 (time 1779313544.71 total 373.05 last 19.56) - open/close 7289 (time 1779313554.72 total 383.05 last 12.30) - open/close 7397 (time 1779313565.06 total 393.40 last 10.44) - open/close 7566 (time 1779313575.08 total 403.42 last 16.87) - open/close 7784 (time 1779313585.68 total 414.02 last 20.56) - open/close 7989 (time 1779313595.90 total 424.24 last 20.07) - open/close 8237 (time 1779313605.90 total 434.24 last 24.78) - open/close 8451 (time 1779313616.25 total 444.59 last 20.67) - open/close 8649 (time 1779313627.21 total 455.55 last 18.07) - open/close 8830 (time 1779313637.23 total 465.57 last 18.06) - open/close 9017 (time 1779313647.30 total 475.64 last 18.57) - open/close 9211 (time 1779313657.31 total 485.65 last 19.38) - open/close 9423 (time 1779313667.38 total 495.72 last 21.06) - open/close 9628 (time 1779313677.39 total 505.72 last 20.48) - open/close 9841 (time 1779313687.39 total 515.72 last 21.30) - open/close 10000 (time 1779313694.41 total 522.75 last 22.62) - open/close 10186 (time 1779313705.36 total 533.70 last 16.99) - open/close 10327 (time 1779313716.05 total 544.38 last 13.20) - open/close 10496 (time 1779313726.06 total 554.40 last 16.87) - open/close 10600 (time 1779313736.06 total 564.40 last 10.40) - open/close 10706 (time 1779313746.07 total 574.41 last 10.59) - open/close 10827 (time 1779313756.08 total 584.42 last 12.08) - open/close 10978 (time 1779313766.97 total 595.31 last 13.87) - open/close 11166 (time 1779313777.17 total 605.50 last 18.44) - open/close 11376 (time 1779313787.45 total 615.79 last 20.41) - open/close 11606 (time 1779313797.47 total 625.81 last 22.97) - open/close 11808 (time 1779313807.47 total 635.81 last 20.20) - open/close 12028 (time 1779313818.56 total 646.90 last 19.84) - open/close 12241 (time 1779313829.49 total 657.83 last 19.48) - open/close 12417 (time 1779313839.55 total 667.89 last 17.50) - open/close 12589 (time 1779313850.06 total 678.40 last 16.37) - open/close 12722 (time 1779313860.07 total 688.41 last 13.28) - open/close 12847 (time 1779313870.08 total 698.42 last 12.48) - open/close 13012 (time 1779313880.14 total 708.48 last 16.40) - open/close 13239 (time 1779313890.75 total 719.09 last 21.40) - open/close 13463 (time 1779313901.24 total 729.58 last 21.35) - open/close 13668 (time 1779313911.44 total 739.78 last 20.11) - open/close 13908 (time 1779313924.42 total 752.76 last 18.48) - open/close 14087 (time 1779313934.61 total 762.95 last 17.57) - open/close 14210 (time 1779313945.30 total 773.63 last 11.51) - open/close 14403 (time 1779313955.46 total 783.80 last 18.98) - open/close 14555 (time 1779313965.53 total 793.87 last 15.10) - open/close 14736 (time 1779313975.54 total 803.88 last 18.07) - open/close 14955 (time 1779313986.05 total 814.39 last 20.84) - open/close 15154 (time 1779313996.06 total 824.40 last 19.88) - open/close 15372 (time 1779314006.07 total 834.41 last 21.78) - open/close 15478 (time 1779314016.07 total 844.41 last 10.59) - open/close 15638 (time 1779314027.16 total 855.50 last 14.43) - open/close 15803 (time 1779314037.17 total 865.51 last 16.50) - open/close 15999 (time 1779314047.57 total 875.91 last 18.84) - open/close 16234 (time 1779314057.58 total 885.92 last 23.48) - open/close 16426 (time 1779314067.58 total 895.92 last 19.19) - open/close 16665 (time 1779314078.16 total 906.50 last 22.60) - open/close 16895 (time 1779314088.17 total 916.51 last 22.98) - open/close 17134 (time 1779314098.17 total 926.51 last 23.90) - open/close 17453 (time 1779314108.19 total 936.53 last 31.84) - open/close 18193 (time 1779314118.19 total 946.53 last 73.96) - open/close 18919 (time 1779314128.20 total 956.54 last 72.53) - open/close 19633 (time 1779314138.21 total 966.55 last 71.35) - open/close 20000 (time 1779314143.31 total 971.65 last 71.92) - open/close 20713 (time 1779314153.32 total 981.66 last 71.26) - open/close 21353 (time 1779314163.32 total 991.66 last 63.96) - open/close 21985 (time 1779314173.34 total 1001.68 last 63.09) - open/close 22676 (time 1779314183.35 total 1011.69 last 69.04) - open/close 23331 (time 1779314193.38 total 1021.72 last 65.29) - open/close 23938 (time 1779314203.39 total 1031.73 last 60.66) - open/close 24551 (time 1779314213.40 total 1041.74 last 61.25) - open/close 25271 (time 1779314223.40 total 1051.74 last 71.98) - open/close 25927 (time 1779314233.40 total 1061.74 last 65.60) - open/close 26544 (time 1779314243.42 total 1071.76 last 61.58) - open/close 27151 (time 1779314253.42 total 1081.76 last 60.68) - open/close 27862 (time 1779314263.42 total 1091.76 last 71.09) - open/close 28649 (time 1779314273.43 total 1101.77 last 78.67) - open/close 29391 (time 1779314283.44 total 1111.78 last 74.14) - open/close 30000 (time 1779314290.86 total 1119.20 last 81.98) - open/close 30846 (time 1779314300.91 total 1129.25 last 84.19) - open/close 31547 (time 1779314310.92 total 1139.26 last 70.08) - open/close 32328 (time 1779314320.92 total 1149.26 last 78.05) - open/close 33203 (time 1779314330.93 total 1159.27 last 87.40) - open/close 33833 (time 1779314340.94 total 1169.28 last 62.95) - open/close 34364 (time 1779314350.96 total 1179.30 last 53.01) - open/close 35175 (time 1779314360.96 total 1189.30 last 81.10) - open/close 36122 (time 1779314370.96 total 1199.30 last 94.65) - open/close 36832 (time 1779314380.97 total 1209.31 last 70.98) - open/close 37563 (time 1779314390.97 total 1219.31 last 73.07) - open/close 38318 (time 1779314400.97 total 1229.31 last 75.50) - open/close 39045 (time 1779314410.98 total 1239.32 last 72.67) - open/close 39659 (time 1779314420.98 total 1249.32 last 61.36) - open/close 40000 (time 1779314429.29 total 1257.63 last 41.06) - open/close 40647 (time 1779314439.29 total 1267.63 last 64.67) - open/close 41357 (time 1779314449.31 total 1277.64 last 70.91) - open/close 42218 (time 1779314459.31 total 1287.65 last 86.03) - open/close 42930 (time 1779314469.33 total 1297.66 last 71.12) - open/close 43738 (time 1779314479.33 total 1307.67 last 80.79) - open/close 44522 (time 1779314489.33 total 1317.67 last 78.36) - open/close 45297 (time 1779314499.34 total 1327.68 last 77.43) - open/close 46087 (time 1779314509.35 total 1337.69 last 78.95) - open/close 46900 (time 1779314519.35 total 1347.69 last 81.24) - open/close 47639 (time 1779314529.36 total 1357.70 last 73.84) - open/close 48260 (time 1779314539.40 total 1367.74 last 61.89) - open/close 48894 (time 1779314549.40 total 1377.74 last 63.39) - open/close 49642 (time 1779314559.41 total 1387.75 last 74.72) total: 50000 open/close in 1392.84 seconds: 35.90 ops/second - unlinked 0 (time 1779314568 ; total 0 ; last 0) - unlinked 10000 (time 1779314735 ; total 167 ; last 167) - unlinked 20000 (time 1779314913 ; total 345 ; last 178) - unlinked 30000 (time 1779315102 ; total 534 ; last 189) - unlinked 40000 (time 1779315278 ; total 710 ; last 176) total: 50000 unlinks in 879 seconds: 56.882820 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local oleg425-server: modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2