== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:25:41 (1781220341) start mds service on oleg307-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: MODOPTS_LIBCFS= oleg307-server: Force libcfs to create 2 CPU partitions oleg307-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg307-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg307-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 99 (time 1781220427.50 total 10.30 last 9.61) - open/close 339 (time 1781220438.41 total 21.21 last 21.99) - open/close 517 (time 1781220448.46 total 31.26 last 17.72) - open/close 718 (time 1781220458.48 total 41.28 last 20.06) - open/close 944 (time 1781220468.48 total 51.28 last 22.60) - open/close 1153 (time 1781220478.60 total 61.40 last 20.64) - open/close 1270 (time 1781220489.36 total 72.16 last 10.88) - open/close 1403 (time 1781220499.46 total 82.26 last 13.17) - open/close 1612 (time 1781220512.28 total 95.08 last 16.30) - open/close 1760 (time 1781220522.28 total 105.08 last 14.80) - open/close 1849 (time 1781220532.30 total 115.10 last 8.88) - open/close 2076 (time 1781220542.98 total 125.78 last 21.26) - open/close 2344 (time 1781220552.99 total 135.78 last 26.79) - open/close 2604 (time 1781220563.00 total 145.80 last 25.96) - open/close 2783 (time 1781220573.01 total 155.81 last 17.89) - open/close 2944 (time 1781220583.02 total 165.82 last 16.09) - open/close 2994 (time 1781220595.22 total 178.02 last 4.10) - open/close 3151 (time 1781220605.27 total 188.07 last 15.62) - open/close 3297 (time 1781220615.30 total 198.10 last 14.56) - open/close 3541 (time 1781220625.73 total 208.53 last 23.39) - open/close 3672 (time 1781220636.71 total 219.51 last 11.94) - open/close 3800 (time 1781220647.06 total 229.86 last 12.36) - open/close 4020 (time 1781220657.95 total 240.75 last 20.20) - open/close 4221 (time 1781220668.14 total 250.94 last 19.72) - open/close 4394 (time 1781220678.95 total 261.75 last 16.01) - open/close 4612 (time 1781220688.95 total 271.75 last 21.78) - open/close 4834 (time 1781220699.80 total 282.60 last 20.46) - open/close 5066 (time 1781220709.82 total 292.61 last 23.17) - open/close 5233 (time 1781220720.47 total 303.27 last 15.67) - open/close 5431 (time 1781220730.69 total 313.49 last 19.37) - open/close 5696 (time 1781220740.71 total 323.51 last 26.46) - open/close 5921 (time 1781220751.30 total 334.10 last 21.24) - open/close 6105 (time 1781220761.91 total 344.71 last 17.35) - open/close 6317 (time 1781220772.44 total 355.24 last 20.13) - open/close 6545 (time 1781220782.45 total 365.25 last 22.76) - open/close 6743 (time 1781220792.80 total 375.60 last 19.15) - open/close 6944 (time 1781220803.33 total 386.13 last 19.08) - open/close 7053 (time 1781220813.35 total 396.15 last 10.88) - open/close 7156 (time 1781220823.90 total 406.70 last 9.76) - open/close 7310 (time 1781220834.27 total 417.07 last 14.86) - open/close 7488 (time 1781220844.27 total 427.07 last 17.80) - open/close 7639 (time 1781220854.28 total 437.08 last 15.08) - open/close 7823 (time 1781220864.40 total 447.20 last 18.18) - open/close 7964 (time 1781220875.16 total 457.96 last 13.11) - open/close 8107 (time 1781220885.49 total 468.28 last 13.85) - open/close 8294 (time 1781220896.27 total 479.06 last 17.35) - open/close 8530 (time 1781220907.09 total 489.89 last 21.81) - open/close 8770 (time 1781220917.82 total 500.62 last 22.35) - open/close 8998 (time 1781220928.79 total 511.59 last 20.79) - open/close 9267 (time 1781220938.80 total 521.60 last 26.87) - open/close 9389 (time 1781220948.84 total 531.63 last 12.16) - open/close 9591 (time 1781220958.86 total 541.66 last 20.15) - open/close 9815 (time 1781220971.68 total 554.48 last 17.46) - open/close 10000 (time 1781220981.26 total 564.06 last 19.31) - open/close 10234 (time 1781220991.27 total 574.07 last 23.38) - open/close 10406 (time 1781221001.50 total 584.29 last 16.82) - open/close 10618 (time 1781221012.04 total 594.84 last 20.10) - open/close 10804 (time 1781221022.29 total 605.09 last 18.15) - open/close 10983 (time 1781221032.61 total 615.41 last 17.35) - open/close 11242 (time 1781221043.03 total 625.82 last 24.87) - open/close 11372 (time 1781221053.06 total 635.85 last 12.96) - open/close 11602 (time 1781221063.07 total 645.87 last 22.97) - open/close 11779 (time 1781221073.30 total 656.10 last 17.30) - open/close 11920 (time 1781221083.32 total 666.12 last 14.08) - open/close 12051 (time 1781221093.45 total 676.25 last 12.93) - open/close 12258 (time 1781221103.46 total 686.26 last 20.68) - open/close 12406 (time 1781221114.08 total 696.88 last 13.94) - open/close 12664 (time 1781221124.09 total 706.89 last 25.79) - open/close 12899 (time 1781221135.06 total 717.86 last 21.42) - open/close 12975 (time 1781221145.06 total 727.86 last 7.60) - open/close 13160 (time 1781221155.47 total 738.27 last 17.78) - open/close 13243 (time 1781221165.91 total 748.71 last 7.95) - open/close 13476 (time 1781221176.65 total 759.45 last 21.71) - open/close 13582 (time 1781221187.17 total 769.97 last 10.07) - open/close 13749 (time 1781221197.18 total 779.98 last 16.69) - open/close 13907 (time 1781221207.20 total 790.00 last 15.76) - open/close 14054 (time 1781221218.08 total 800.87 last 13.52) - open/close 14211 (time 1781221228.76 total 811.56 last 14.70) - open/close 14394 (time 1781221239.00 total 821.80 last 17.87) - open/close 14541 (time 1781221250.44 total 833.24 last 12.85) - open/close 14840 (time 1781221260.46 total 843.26 last 29.85) - open/close 15028 (time 1781221270.63 total 853.43 last 18.48) - open/close 15179 (time 1781221281.17 total 863.97 last 14.33) - open/close 15394 (time 1781221292.02 total 874.82 last 19.81) - open/close 15601 (time 1781221302.26 total 885.06 last 20.21) - open/close 15816 (time 1781221312.27 total 895.07 last 21.48) - open/close 15974 (time 1781221322.52 total 905.32 last 15.42) - open/close 16160 (time 1781221334.62 total 917.42 last 15.37) - open/close 16368 (time 1781221344.85 total 927.65 last 20.33) - open/close 16569 (time 1781221354.86 total 937.66 last 20.08) - open/close 16736 (time 1781221364.86 total 947.66 last 16.70) - open/close 16978 (time 1781221374.87 total 957.67 last 24.19) - open/close 17198 (time 1781221385.23 total 968.03 last 21.23) - open/close 17455 (time 1781221395.24 total 978.04 last 25.66) - open/close 17649 (time 1781221405.54 total 988.34 last 18.85) - open/close 18337 (time 1781221415.54 total 998.34 last 68.78) - open/close 18969 (time 1781221425.55 total 1008.35 last 63.17) - open/close 19702 (time 1781221435.55 total 1018.35 last 73.25) - open/close 20000 (time 1781221439.85 total 1022.65 last 69.31) - open/close 20636 (time 1781221449.86 total 1032.66 last 63.55) - open/close 21261 (time 1781221459.86 total 1042.66 last 62.48) - open/close 21850 (time 1781221469.88 total 1052.68 last 58.82) - open/close 22431 (time 1781221479.89 total 1062.69 last 58.01) - open/close 22974 (time 1781221489.90 total 1072.70 last 54.25) - open/close 23507 (time 1781221499.92 total 1082.72 last 53.22) - open/close 24035 (time 1781221509.93 total 1092.73 last 52.73) - open/close 24584 (time 1781221519.95 total 1102.75 last 54.78) - open/close 25198 (time 1781221529.96 total 1112.76 last 61.35) - open/close 25783 (time 1781221539.98 total 1122.78 last 58.37) - open/close 26350 (time 1781221550.00 total 1132.80 last 56.62) - open/close 26907 (time 1781221560.01 total 1142.81 last 55.65) - open/close 27532 (time 1781221570.01 total 1152.81 last 62.46) - open/close 28160 (time 1781221580.03 total 1162.83 last 62.70) - open/close 28824 (time 1781221590.06 total 1172.86 last 66.17) - open/close 29400 (time 1781221600.07 total 1182.87 last 57.56) - open/close 30000 (time 1781221609.33 total 1192.13 last 64.83) - open/close 30679 (time 1781221619.33 total 1202.13 last 67.87) - open/close 31236 (time 1781221629.34 total 1212.13 last 55.67) - open/close 31812 (time 1781221639.34 total 1222.14 last 57.55) - open/close 32399 (time 1781221649.35 total 1232.15 last 58.64) - open/close 32994 (time 1781221659.36 total 1242.16 last 59.45) - open/close 33573 (time 1781221669.37 total 1252.17 last 57.86) - open/close 34176 (time 1781221679.38 total 1262.17 last 60.27) - open/close 34848 (time 1781221689.39 total 1272.19 last 67.12) - open/close 35422 (time 1781221699.40 total 1282.20 last 57.33) - open/close 36030 (time 1781221709.41 total 1292.21 last 60.75) - open/close 36629 (time 1781221719.45 total 1302.25 last 59.67) - open/close 37259 (time 1781221729.47 total 1312.26 last 62.89) - open/close 37864 (time 1781221739.47 total 1322.26 last 60.50) - open/close 38500 (time 1781221749.48 total 1332.28 last 63.50) - open/close 39181 (time 1781221759.49 total 1342.29 last 68.07) - open/close 39907 (time 1781221769.49 total 1352.29 last 72.55) - open/close 40628 (time 1781221779.51 total 1362.30 last 72.02) - open/close 41275 (time 1781221789.51 total 1372.31 last 64.65) - open/close 41927 (time 1781221799.52 total 1382.32 last 65.13) - open/close 42636 (time 1781221809.53 total 1392.33 last 70.88) - open/close 43241 (time 1781221819.53 total 1402.33 last 60.47) - open/close 43895 (time 1781221829.54 total 1412.34 last 65.35) - open/close 44518 (time 1781221839.56 total 1422.36 last 62.18) - open/close 45094 (time 1781221849.57 total 1432.36 last 57.56) - open/close 45730 (time 1781221859.58 total 1442.38 last 63.52) - open/close 46317 (time 1781221869.58 total 1452.38 last 58.67) - open/close 46949 (time 1781221879.59 total 1462.39 last 63.15) - open/close 47529 (time 1781221889.63 total 1472.43 last 57.78) - open/close 48052 (time 1781221899.64 total 1482.44 last 52.25) - open/close 48652 (time 1781221909.64 total 1492.44 last 59.96) - open/close 49201 (time 1781221919.66 total 1502.45 last 54.84) - open/close 49699 (time 1781221929.67 total 1512.47 last 49.72) total: 50000 open/close in 1518.31 seconds: 32.93 ops/second - unlinked 0 (time 1781221940 ; total 1 ; last 1) - unlinked 10000 (time 1781222097 ; total 158 ; last 157) - unlinked 20000 (time 1781222258 ; total 319 ; last 161) - unlinked 30000 (time 1781222433 ; total 494 ; last 175) - unlinked 40000 (time 1781222594 ; total 655 ; last 161) total: 50000 unlinks in 816 seconds: 61.274509 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server unloading modules via unload_modules_local on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2