== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:27:10 (1779690430) start mds service on oleg427-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 154 (time 1779690499.35 total 10.38 last 14.84) - open/close 329 (time 1779690510.16 total 21.19 last 16.19) - open/close 476 (time 1779690520.34 total 31.37 last 14.44) - open/close 651 (time 1779690530.98 total 42.01 last 16.44) - open/close 909 (time 1779690542.03 total 53.07 last 23.33) - open/close 1085 (time 1779690552.78 total 63.81 last 16.38) - open/close 1245 (time 1779690563.01 total 74.05 last 15.64) - open/close 1381 (time 1779690573.02 total 84.05 last 13.59) - open/close 1515 (time 1779690583.03 total 94.06 last 13.39) - open/close 1642 (time 1779690593.03 total 104.06 last 12.70) - open/close 1788 (time 1779690603.21 total 114.24 last 14.34) - open/close 2017 (time 1779690613.52 total 124.55 last 22.23) - open/close 2258 (time 1779690624.20 total 135.23 last 22.56) - open/close 2436 (time 1779690634.96 total 145.99 last 16.55) - open/close 2594 (time 1779690645.14 total 156.17 last 15.51) - open/close 2741 (time 1779690655.24 total 166.27 last 14.55) - open/close 2911 (time 1779690665.74 total 176.77 last 16.19) - open/close 3015 (time 1779690676.23 total 187.27 last 9.91) - open/close 3179 (time 1779690686.79 total 197.83 last 15.53) - open/close 3375 (time 1779690697.61 total 208.64 last 18.13) - open/close 3579 (time 1779690707.61 total 218.64 last 20.40) - open/close 3749 (time 1779690718.16 total 229.19 last 16.12) - open/close 3897 (time 1779690728.27 total 239.30 last 14.63) - open/close 4057 (time 1779690738.45 total 249.48 last 15.72) - open/close 4216 (time 1779690748.75 total 259.78 last 15.44) - open/close 4415 (time 1779690759.70 total 270.73 last 18.17) - open/close 4553 (time 1779690770.76 total 281.79 last 12.47) - open/close 4699 (time 1779690780.88 total 291.91 last 14.43) - open/close 4801 (time 1779690791.37 total 302.41 last 9.72) - open/close 4937 (time 1779690802.25 total 313.29 last 12.50) - open/close 5056 (time 1779690812.87 total 323.90 last 11.21) - open/close 5124 (time 1779690823.05 total 334.08 last 6.68) - open/close 5229 (time 1779690833.61 total 344.64 last 9.94) - open/close 5346 (time 1779690844.42 total 355.46 last 10.82) - open/close 5437 (time 1779690854.86 total 365.89 last 8.72) - open/close 5562 (time 1779690865.42 total 376.45 last 11.83) - open/close 5660 (time 1779690875.92 total 386.95 last 9.34) - open/close 5846 (time 1779690886.79 total 397.82 last 17.11) - open/close 5999 (time 1779690897.74 total 408.77 last 13.98) - open/close 6122 (time 1779690908.62 total 419.65 last 11.30) - open/close 6253 (time 1779690918.80 total 429.83 last 12.87) - open/close 6357 (time 1779690929.54 total 440.57 last 9.68) - open/close 6528 (time 1779690939.65 total 450.69 last 16.91) - open/close 6647 (time 1779690950.53 total 461.57 last 10.94) - open/close 6796 (time 1779690960.78 total 471.81 last 14.54) - open/close 6996 (time 1779690970.78 total 481.82 last 19.99) - open/close 7133 (time 1779690980.79 total 491.82 last 13.69) - open/close 7221 (time 1779690991.31 total 502.34 last 8.37) - open/close 7379 (time 1779691001.43 total 512.46 last 15.62) - open/close 7555 (time 1779691011.94 total 522.97 last 16.75) - open/close 7725 (time 1779691022.93 total 533.96 last 15.47) - open/close 7899 (time 1779691033.64 total 544.67 last 16.25) - open/close 8111 (time 1779691043.66 total 554.69 last 21.15) - open/close 8282 (time 1779691054.36 total 565.39 last 15.98) - open/close 8376 (time 1779691065.30 total 576.33 last 8.59) - open/close 8576 (time 1779691075.31 total 586.35 last 19.97) - open/close 8761 (time 1779691088.10 total 599.13 last 14.47) - open/close 9029 (time 1779691098.25 total 609.28 last 26.39) - open/close 9297 (time 1779691109.36 total 620.40 last 24.11) - open/close 9572 (time 1779691119.37 total 630.40 last 27.49) - open/close 9791 (time 1779691129.39 total 640.42 last 21.86) - open/close 10000 (time 1779691137.51 total 648.55 last 25.72) - open/close 10178 (time 1779691148.05 total 659.08 last 16.90) - open/close 10403 (time 1779691158.06 total 669.09 last 22.47) - open/close 10612 (time 1779691168.07 total 679.10 last 20.88) - open/close 10831 (time 1779691178.07 total 689.10 last 21.90) - open/close 11504 (time 1779691188.10 total 699.13 last 67.09) - open/close 12208 (time 1779691198.12 total 709.15 last 70.27) - open/close 12844 (time 1779691208.12 total 719.15 last 63.58) - open/close 13557 (time 1779691218.13 total 729.16 last 71.24) - open/close 14280 (time 1779691228.13 total 739.17 last 72.26) - open/close 15070 (time 1779691238.14 total 749.17 last 78.97) - open/close 15862 (time 1779691248.14 total 759.18 last 79.14) - open/close 16633 (time 1779691258.15 total 769.18 last 77.05) - open/close 17400 (time 1779691268.16 total 779.20 last 76.61) - open/close 18114 (time 1779691278.18 total 789.21 last 71.31) - open/close 18845 (time 1779691288.18 total 799.21 last 73.07) - open/close 19605 (time 1779691298.20 total 809.24 last 75.83) - open/close 20000 (time 1779691303.47 total 814.50 last 75.01) - open/close 20775 (time 1779691313.48 total 824.51 last 77.45) - open/close 21550 (time 1779691323.49 total 834.52 last 77.43) - open/close 22268 (time 1779691333.49 total 844.52 last 71.75) - open/close 23048 (time 1779691343.50 total 854.53 last 77.95) - open/close 23817 (time 1779691353.51 total 864.54 last 76.84) - open/close 24633 (time 1779691363.52 total 874.55 last 81.52) - open/close 25405 (time 1779691373.52 total 884.55 last 77.19) - open/close 26195 (time 1779691383.52 total 894.55 last 78.96) - open/close 27022 (time 1779691393.53 total 904.56 last 82.66) - open/close 27857 (time 1779691403.53 total 914.56 last 83.46) - open/close 28808 (time 1779691413.54 total 924.58 last 94.99) - open/close 29943 (time 1779691423.55 total 934.58 last 113.40) - open/close 30946 (time 1779691433.55 total 944.59 last 100.29) - open/close 32004 (time 1779691443.56 total 954.59 last 105.77) - open/close 33201 (time 1779691453.56 total 964.59 last 119.67) - open/close 34398 (time 1779691463.56 total 974.59 last 119.66) - open/close 35809 (time 1779691473.57 total 984.60 last 141.02) - open/close 36922 (time 1779691483.57 total 994.61 last 111.24) - open/close 38170 (time 1779691493.57 total 1004.61 last 124.79) - open/close 39357 (time 1779691503.58 total 1014.61 last 118.59) - open/close 40000 (time 1779691509.93 total 1020.96 last 101.31) - open/close 41097 (time 1779691519.93 total 1030.97 last 109.65) - open/close 42096 (time 1779691529.94 total 1040.97 last 99.84) - open/close 43162 (time 1779691539.94 total 1050.97 last 106.57) - open/close 44230 (time 1779691549.95 total 1060.98 last 106.74) - open/close 45353 (time 1779691559.95 total 1070.99 last 112.21) - open/close 46457 (time 1779691569.96 total 1080.99 last 110.39) - open/close 47574 (time 1779691579.96 total 1090.99 last 111.70) - open/close 48685 (time 1779691589.96 total 1100.99 last 111.09) - open/close 49879 (time 1779691599.96 total 1110.99 last 119.39) total: 50000 open/close in 1111.94 seconds: 44.97 ops/second - unlinked 0 (time 1779691603 ; total 0 ; last 0) - unlinked 10000 (time 1779691674 ; total 71 ; last 71) - unlinked 20000 (time 1779691715 ; total 112 ; last 41) - unlinked 30000 (time 1779691751 ; total 148 ; last 36) - unlinked 40000 (time 1779691791 ; total 188 ; last 40) total: 50000 unlinks in 233 seconds: 214.592270 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded. pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2