== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:45:01 (1787856301) start mds service on oleg340-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg340-server: detected 4 online CPUs by sysfs oleg340-server: MODOPTS_LIBCFS= oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg340-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg340-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 175 (time 1787856384.40 total 10.01 last 17.49) - open/close 355 (time 1787856395.11 total 20.72 last 16.81) - open/close 540 (time 1787856405.79 total 31.40 last 17.32) - open/close 778 (time 1787856415.80 total 41.41 last 23.78) - open/close 915 (time 1787856426.11 total 51.71 last 13.29) - open/close 1136 (time 1787856437.15 total 62.76 last 20.01) - open/close 1381 (time 1787856447.16 total 72.77 last 24.48) - open/close 1538 (time 1787856457.66 total 83.27 last 14.94) - open/close 1786 (time 1787856468.22 total 93.83 last 23.48) - open/close 2010 (time 1787856479.68 total 105.29 last 19.56) - open/close 2236 (time 1787856489.68 total 115.29 last 22.59) - open/close 2385 (time 1787856499.68 total 125.29 last 14.90) - open/close 2564 (time 1787856510.25 total 135.85 last 16.95) - open/close 2646 (time 1787856523.50 total 149.11 last 6.19) - open/close 2847 (time 1787856533.61 total 159.22 last 19.87) - open/close 3134 (time 1787856543.63 total 169.24 last 28.66) - open/close 3374 (time 1787856553.64 total 179.25 last 23.97) - open/close 3616 (time 1787856563.65 total 189.26 last 24.18) - open/close 3819 (time 1787856573.68 total 199.29 last 20.24) - open/close 4038 (time 1787856583.69 total 209.30 last 21.88) - open/close 4240 (time 1787856593.69 total 219.30 last 20.20) - open/close 4488 (time 1787856603.72 total 229.33 last 24.74) - open/close 4722 (time 1787856616.57 total 242.17 last 18.21) - open/close 4909 (time 1787856626.61 total 252.22 last 18.61) - open/close 5194 (time 1787856637.35 total 262.96 last 26.54) - open/close 5416 (time 1787856648.19 total 273.79 last 20.50) - open/close 5635 (time 1787856658.73 total 284.34 last 20.76) - open/close 5910 (time 1787856668.74 total 294.34 last 27.50) - open/close 6133 (time 1787856678.75 total 304.36 last 22.26) - open/close 6335 (time 1787856689.52 total 315.13 last 18.75) - open/close 6572 (time 1787856700.46 total 326.06 last 21.68) - open/close 6831 (time 1787856710.64 total 336.25 last 25.44) - open/close 7118 (time 1787856721.45 total 347.06 last 26.53) - open/close 7333 (time 1787856731.47 total 357.08 last 21.47) - open/close 7577 (time 1787856742.02 total 367.63 last 23.12) - open/close 7872 (time 1787856752.88 total 378.49 last 27.16) - open/close 8103 (time 1787856763.75 total 389.36 last 21.26) - open/close 8299 (time 1787856773.75 total 399.36 last 19.60) - open/close 8443 (time 1787856784.74 total 410.35 last 13.11) - open/close 8599 (time 1787856795.31 total 420.92 last 14.76) - open/close 8719 (time 1787856806.18 total 431.79 last 11.04) - open/close 8881 (time 1787856817.20 total 442.81 last 14.70) - open/close 8985 (time 1787856827.71 total 453.32 last 9.89) - open/close 9199 (time 1787856837.73 total 463.34 last 21.37) - open/close 9307 (time 1787856848.57 total 474.17 last 9.96) - open/close 9531 (time 1787856859.03 total 484.64 last 21.40) - open/close 9772 (time 1787856869.08 total 494.69 last 23.99) - open/close 9965 (time 1787856879.12 total 504.73 last 19.21) - open/close 10183 (time 1787856889.55 total 515.16 last 20.90) - open/close 10421 (time 1787856899.56 total 525.17 last 23.78) - open/close 10623 (time 1787856910.01 total 535.62 last 19.33) - open/close 10870 (time 1787856920.87 total 546.48 last 22.74) - open/close 11087 (time 1787856931.25 total 556.85 last 20.92) - open/close 11241 (time 1787856941.93 total 567.54 last 14.41) - open/close 11445 (time 1787856952.89 total 578.50 last 18.62) - open/close 11698 (time 1787856962.89 total 588.50 last 25.29) - open/close 11846 (time 1787856973.18 total 598.79 last 14.38) - open/close 12006 (time 1787856983.80 total 609.41 last 15.07) - open/close 12129 (time 1787856993.93 total 619.54 last 12.14) - open/close 12274 (time 1787857004.42 total 630.03 last 13.82) - open/close 12476 (time 1787857014.42 total 640.03 last 20.19) - open/close 12689 (time 1787857025.00 total 650.61 last 20.14) - open/close 12913 (time 1787857035.95 total 661.56 last 20.46) - open/close 13103 (time 1787857046.25 total 671.86 last 18.45) - open/close 13291 (time 1787857057.33 total 682.94 last 16.97) - open/close 13494 (time 1787857067.33 total 692.94 last 20.29) - open/close 13707 (time 1787857077.37 total 702.98 last 21.22) - open/close 13935 (time 1787857087.98 total 713.59 last 21.48) - open/close 14074 (time 1787857097.99 total 723.60 last 13.88) - open/close 14289 (time 1787857108.26 total 733.87 last 20.94) - open/close 14519 (time 1787857118.58 total 744.19 last 22.30) - open/close 14780 (time 1787857128.95 total 754.56 last 25.17) - open/close 15002 (time 1787857138.96 total 764.57 last 22.18) - open/close 15131 (time 1787857148.98 total 774.59 last 12.87) - open/close 15278 (time 1787857159.80 total 785.41 last 13.59) - open/close 15380 (time 1787857170.65 total 796.26 last 9.40) - open/close 15471 (time 1787857181.06 total 806.67 last 8.74) - open/close 15686 (time 1787857191.08 total 816.68 last 21.48) - open/close 15834 (time 1787857201.08 total 826.69 last 14.80) - open/close 16074 (time 1787857211.10 total 836.70 last 23.96) - open/close 16310 (time 1787857221.74 total 847.35 last 22.17) - open/close 16471 (time 1787857232.04 total 857.65 last 15.63) - open/close 16664 (time 1787857243.69 total 869.30 last 16.57) - open/close 16921 (time 1787857254.56 total 880.17 last 23.64) - open/close 17177 (time 1787857264.57 total 890.18 last 25.58) - open/close 17390 (time 1787857274.58 total 900.19 last 21.27) - open/close 17596 (time 1787857284.78 total 910.38 last 20.21) - open/close 17820 (time 1787857295.01 total 920.62 last 21.88) - open/close 18074 (time 1787857305.71 total 931.32 last 23.75) - open/close 18563 (time 1787857315.72 total 941.33 last 48.83) - open/close 19205 (time 1787857325.73 total 951.34 last 64.14) - open/close 19858 (time 1787857335.75 total 961.35 last 65.22) - open/close 20000 (time 1787857337.92 total 963.53 last 65.28) - open/close 20613 (time 1787857347.93 total 973.54 last 61.24) - open/close 21239 (time 1787857357.93 total 983.54 last 62.59) - open/close 21925 (time 1787857367.94 total 993.55 last 68.54) - open/close 22549 (time 1787857377.95 total 1003.56 last 62.37) - open/close 23149 (time 1787857387.95 total 1013.56 last 59.97) - open/close 23772 (time 1787857397.97 total 1023.57 last 62.22) - open/close 24386 (time 1787857407.98 total 1033.58 last 61.34) - open/close 25040 (time 1787857417.98 total 1043.59 last 65.34) - open/close 25714 (time 1787857427.99 total 1053.60 last 67.35) - open/close 26363 (time 1787857438.00 total 1063.60 last 64.87) - open/close 27025 (time 1787857448.02 total 1073.63 last 66.02) - open/close 27672 (time 1787857458.02 total 1083.63 last 64.69) - open/close 28313 (time 1787857468.03 total 1093.64 last 64.07) - open/close 28964 (time 1787857478.06 total 1103.67 last 64.90) - open/close 29637 (time 1787857488.11 total 1113.72 last 66.98) - open/close 30000 (time 1787857493.97 total 1119.58 last 61.90) - open/close 30688 (time 1787857504.02 total 1129.63 last 68.47) - open/close 31304 (time 1787857514.03 total 1139.64 last 61.53) - open/close 31865 (time 1787857524.04 total 1149.65 last 56.03) - open/close 32451 (time 1787857534.05 total 1159.66 last 58.58) - open/close 33022 (time 1787857544.07 total 1169.68 last 56.98) - open/close 33598 (time 1787857554.07 total 1179.68 last 57.58) - open/close 34297 (time 1787857564.08 total 1189.69 last 69.87) - open/close 34963 (time 1787857574.08 total 1199.69 last 66.56) - open/close 35582 (time 1787857584.09 total 1209.69 last 61.88) - open/close 36244 (time 1787857594.09 total 1219.69 last 66.20) - open/close 36892 (time 1787857604.09 total 1229.70 last 64.77) - open/close 37548 (time 1787857614.10 total 1239.70 last 65.57) - open/close 38195 (time 1787857624.10 total 1249.71 last 64.66) - open/close 38787 (time 1787857634.11 total 1259.71 last 59.17) - open/close 39405 (time 1787857644.11 total 1269.72 last 61.79) - open/close 39991 (time 1787857654.14 total 1279.74 last 58.44) - open/close 40523 (time 1787857664.16 total 1289.76 last 53.09) - open/close 41056 (time 1787857674.16 total 1299.77 last 53.28) - open/close 41633 (time 1787857684.17 total 1309.78 last 57.66) - open/close 42165 (time 1787857694.18 total 1319.79 last 53.12) - open/close 42795 (time 1787857704.20 total 1329.80 last 62.92) - open/close 43435 (time 1787857714.20 total 1339.81 last 63.98) - open/close 44065 (time 1787857724.20 total 1349.81 last 62.97) - open/close 44725 (time 1787857734.21 total 1359.81 last 65.98) - open/close 45356 (time 1787857744.22 total 1369.83 last 62.99) - open/close 45999 (time 1787857754.24 total 1379.85 last 64.17) - open/close 46521 (time 1787857764.27 total 1389.88 last 52.06) - open/close 47117 (time 1787857774.28 total 1399.89 last 59.51) - open/close 47738 (time 1787857784.30 total 1409.91 last 62.00) - open/close 48362 (time 1787857794.31 total 1419.92 last 62.35) - open/close 49066 (time 1787857804.31 total 1429.92 last 70.36) - open/close 49723 (time 1787857814.33 total 1439.93 last 65.62) total: 50000 open/close in 1444.58 seconds: 34.61 ops/second - unlinked 0 (time 1787857824 ; total 0 ; last 0) - unlinked 10000 (time 1787857995 ; total 171 ; last 171) - unlinked 20000 (time 1787858166 ; total 342 ; last 171) - unlinked 30000 (time 1787858326 ; total 502 ; last 160) - unlinked 40000 (time 1787858486 ; total 662 ; last 160) total: 50000 unlinks in 822 seconds: 60.827251 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server unloading modules via unload_modules_local on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local oleg340-server: modules unloaded. pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2