== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:26:58 (1787038018) start mds service on oleg202-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing load_modules_local oleg202-server: Loading modules from /home/green/git/lustre-release/lustre oleg202-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg202-server: detected 4 online CPUs by sysfs oleg202-server: MODOPTS_LIBCFS= oleg202-server: Force libcfs to create 2 CPU partitions oleg202-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg202-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg202-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg202-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg202-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0000 oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg202-client.virtnet: -o user_xattr,flock 192.168.202.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 175 (time 1787038129.69 total 10.06 last 17.39) - open/close 370 (time 1787038139.70 total 20.06 last 19.49) - open/close 555 (time 1787038150.20 total 30.57 last 17.61) - open/close 771 (time 1787038160.39 total 40.75 last 21.21) - open/close 984 (time 1787038171.04 total 51.40 last 19.99) - open/close 1230 (time 1787038181.16 total 61.53 last 24.30) - open/close 1545 (time 1787038191.54 total 71.90 last 30.36) - open/close 1872 (time 1787038202.40 total 82.77 last 30.10) - open/close 2123 (time 1787038212.55 total 92.91 last 24.75) - open/close 2389 (time 1787038223.11 total 103.48 last 25.17) - open/close 2551 (time 1787038233.15 total 113.51 last 16.14) - open/close 2807 (time 1787038243.68 total 124.04 last 24.32) - open/close 3134 (time 1787038254.43 total 134.80 last 30.39) - open/close 3403 (time 1787038264.84 total 145.21 last 25.85) - open/close 3646 (time 1787038274.84 total 155.21 last 24.30) - open/close 3871 (time 1787038285.03 total 165.40 last 22.08) - open/close 4118 (time 1787038295.04 total 175.40 last 24.69) - open/close 4387 (time 1787038305.95 total 186.32 last 24.64) - open/close 4634 (time 1787038315.97 total 196.34 last 24.65) - open/close 4892 (time 1787038325.97 total 206.34 last 25.79) - open/close 5139 (time 1787038336.44 total 216.81 last 23.59) - open/close 5366 (time 1787038346.64 total 227.00 last 22.27) - open/close 5472 (time 1787038356.64 total 237.00 last 10.60) - open/close 5684 (time 1787038367.16 total 247.53 last 20.14) - open/close 5825 (time 1787038377.32 total 257.68 last 13.89) - open/close 6062 (time 1787038387.42 total 267.79 last 23.46) - open/close 6289 (time 1787038397.80 total 278.17 last 21.87) - open/close 6561 (time 1787038408.47 total 288.84 last 25.49) - open/close 6843 (time 1787038418.48 total 298.84 last 28.19) - open/close 7071 (time 1787038428.49 total 308.85 last 22.78) - open/close 7295 (time 1787038438.83 total 319.19 last 21.66) - open/close 7639 (time 1787038449.25 total 329.62 last 32.99) - open/close 7897 (time 1787038459.29 total 339.65 last 25.72) - open/close 8193 (time 1787038470.06 total 350.42 last 27.48) - open/close 8447 (time 1787038480.06 total 360.43 last 25.38) - open/close 8719 (time 1787038490.69 total 371.06 last 25.59) - open/close 8905 (time 1787038500.71 total 381.08 last 18.57) - open/close 9170 (time 1787038510.72 total 391.08 last 26.49) - open/close 9332 (time 1787038520.73 total 401.10 last 16.17) - open/close 9542 (time 1787038530.86 total 411.22 last 20.75) - open/close 9826 (time 1787038540.86 total 421.22 last 28.40) - open/close 10000 (time 1787038547.39 total 427.76 last 26.63) - open/close 10275 (time 1787038557.40 total 437.76 last 27.48) - open/close 10523 (time 1787038567.97 total 448.34 last 23.46) - open/close 10792 (time 1787038578.67 total 459.03 last 25.14) - open/close 11016 (time 1787038588.68 total 469.04 last 22.38) - open/close 11241 (time 1787038598.69 total 479.06 last 22.47) - open/close 11506 (time 1787038609.52 total 489.89 last 24.47) - open/close 11788 (time 1787038619.53 total 499.89 last 28.17) - open/close 12042 (time 1787038629.93 total 510.29 last 24.42) - open/close 12208 (time 1787038639.95 total 520.31 last 16.57) - open/close 12346 (time 1787038649.95 total 530.32 last 13.79) - open/close 12574 (time 1787038660.53 total 540.90 last 21.56) - open/close 12827 (time 1787038670.54 total 550.91 last 25.27) - open/close 13076 (time 1787038680.55 total 560.91 last 24.89) - open/close 13353 (time 1787038690.55 total 570.92 last 27.69) - open/close 13600 (time 1787038700.56 total 580.92 last 24.69) - open/close 13819 (time 1787038711.34 total 591.70 last 20.32) - open/close 14053 (time 1787038721.35 total 601.71 last 23.37) - open/close 14301 (time 1787038731.62 total 611.99 last 24.14) - open/close 14552 (time 1787038741.62 total 621.99 last 25.09) - open/close 14780 (time 1787038751.64 total 632.00 last 22.78) - open/close 15007 (time 1787038761.64 total 642.00 last 22.70) - open/close 15224 (time 1787038772.07 total 652.44 last 20.80) - open/close 15485 (time 1787038782.07 total 662.44 last 26.09) - open/close 15681 (time 1787038792.08 total 672.44 last 19.59) - open/close 15920 (time 1787038802.81 total 683.17 last 22.27) - open/close 16224 (time 1787038813.81 total 694.17 last 27.64) - open/close 16477 (time 1787038823.82 total 704.18 last 25.28) - open/close 16736 (time 1787038833.82 total 714.19 last 25.89) - open/close 16941 (time 1787038844.20 total 724.57 last 19.75) - open/close 17255 (time 1787038854.21 total 734.58 last 31.36) - open/close 17501 (time 1787038865.13 total 745.49 last 22.54) - open/close 17737 (time 1787038875.96 total 756.32 last 21.79) - open/close 17985 (time 1787038886.25 total 766.61 last 24.09) - open/close 18270 (time 1787038896.69 total 777.06 last 27.29) - open/close 18575 (time 1787038907.62 total 787.99 last 27.91) - open/close 18803 (time 1787038917.73 total 798.10 last 22.56) - open/close 19062 (time 1787038928.44 total 808.80 last 24.19) - open/close 19286 (time 1787038938.45 total 818.82 last 22.37) - open/close 19567 (time 1787038948.46 total 828.82 last 28.07) - open/close 19825 (time 1787038958.63 total 838.99 last 25.38) - open/close 20000 (time 1787038964.49 total 844.86 last 29.84) - open/close 20204 (time 1787038974.63 total 855.00 last 20.12) - open/close 20467 (time 1787038985.59 total 865.95 last 24.01) - open/close 21456 (time 1787038995.59 total 875.95 last 98.89) - open/close 22550 (time 1787039005.59 total 885.95 last 109.39) - open/close 23566 (time 1787039015.59 total 895.96 last 101.57) - open/close 24489 (time 1787039025.60 total 905.96 last 92.26) - open/close 25491 (time 1787039035.60 total 915.97 last 100.12) - open/close 26522 (time 1787039045.61 total 925.97 last 103.04) - open/close 27492 (time 1787039055.61 total 935.98 last 96.95) - open/close 28477 (time 1787039065.62 total 945.99 last 98.42) - open/close 29425 (time 1787039075.63 total 955.99 last 94.73) - open/close 30000 (time 1787039081.09 total 961.46 last 105.30) - open/close 30983 (time 1787039091.10 total 971.46 last 98.24) - open/close 32049 (time 1787039101.10 total 981.46 last 106.57) - open/close 33075 (time 1787039111.11 total 991.47 last 102.52) - open/close 34170 (time 1787039121.11 total 1001.48 last 109.43) - open/close 35095 (time 1787039131.12 total 1011.48 last 92.46) - open/close 36127 (time 1787039141.12 total 1021.49 last 103.14) - open/close 37201 (time 1787039151.13 total 1031.50 last 107.31) - open/close 38177 (time 1787039161.14 total 1041.51 last 97.50) - open/close 39148 (time 1787039171.14 total 1051.51 last 97.08) - open/close 40000 (time 1787039179.18 total 1059.55 last 106.00) - open/close 40990 (time 1787039189.19 total 1069.56 last 98.90) - open/close 42054 (time 1787039199.19 total 1079.56 last 106.36) - open/close 43128 (time 1787039209.19 total 1089.56 last 107.39) - open/close 44076 (time 1787039219.20 total 1099.56 last 94.79) - open/close 45031 (time 1787039229.20 total 1109.56 last 95.48) - open/close 46041 (time 1787039239.20 total 1119.56 last 101.00) - open/close 47097 (time 1787039249.21 total 1129.58 last 105.47) - open/close 48130 (time 1787039259.23 total 1139.59 last 103.12) - open/close 49087 (time 1787039269.24 total 1149.60 last 95.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1155.48 seconds: 42.95 ops/second - unlinked 0 (time 1787039278 ; total 0 ; last 0) - unlinked 10000 (time 1787039375 ; total 97 ; last 97) - unlinked 20000 (time 1787039465 ; total 187 ; last 90) - unlinked 30000 (time 1787039522 ; total 244 ; last 57) - unlinked 40000 (time 1787039581 ; total 303 ; last 59) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 382 seconds: 129.924088 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg202-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server unloading modules via unload_modules_local on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing unload_modules_local oleg202-server: modules unloaded. pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2