== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:31:37 (1787466697) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg152-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg152-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 77 (time 1787466832.75 total 10.12 last 7.61) - open/close 231 (time 1787466843.70 total 21.08 last 14.06) - open/close 364 (time 1787466854.73 total 32.10 last 12.06) - open/close 512 (time 1787466865.37 total 42.75 last 13.91) - open/close 630 (time 1787466875.60 total 52.98 last 11.53) - open/close 719 (time 1787466886.30 total 63.68 last 8.32) - open/close 792 (time 1787466896.66 total 74.04 last 7.05) - open/close 941 (time 1787466907.40 total 84.78 last 13.87) - open/close 1055 (time 1787466917.43 total 94.81 last 11.37) - open/close 1178 (time 1787466927.44 total 104.82 last 12.29) - open/close 1297 (time 1787466937.78 total 115.16 last 11.51) - open/close 1521 (time 1787466947.78 total 125.16 last 22.39) - open/close 1690 (time 1787466957.80 total 135.17 last 16.88) - open/close 1808 (time 1787466968.41 total 145.79 last 11.12) - open/close 1965 (time 1787466978.41 total 155.79 last 15.70) - open/close 2097 (time 1787466988.43 total 165.81 last 13.18) - open/close 2256 (time 1787466998.61 total 175.99 last 15.62) - open/close 2311 (time 1787467009.63 total 187.01 last 4.99) - open/close 2389 (time 1787467019.99 total 197.37 last 7.53) - open/close 2464 (time 1787467030.01 total 207.39 last 7.48) - open/close 2560 (time 1787467040.60 total 217.98 last 9.07) - open/close 2672 (time 1787467050.96 total 228.33 last 10.82) - open/close 2782 (time 1787467064.00 total 241.38 last 8.43) - open/close 2918 (time 1787467074.88 total 252.26 last 12.50) - open/close 3043 (time 1787467085.41 total 262.79 last 11.87) - open/close 3145 (time 1787467095.50 total 272.88 last 10.11) - open/close 3239 (time 1787467106.25 total 283.63 last 8.74) - open/close 3458 (time 1787467117.14 total 294.52 last 20.11) - open/close 3600 (time 1787467127.70 total 305.08 last 13.44) - open/close 3774 (time 1787467138.71 total 316.09 last 15.81) - open/close 3878 (time 1787467149.00 total 326.37 last 10.11) - open/close 4017 (time 1787467159.01 total 336.38 last 13.89) - open/close 4142 (time 1787467170.01 total 347.39 last 11.36) - open/close 4307 (time 1787467180.01 total 357.39 last 16.50) - open/close 4389 (time 1787467190.66 total 368.04 last 7.70) - open/close 4457 (time 1787467200.67 total 378.05 last 6.79) - open/close 4682 (time 1787467210.70 total 388.07 last 22.45) - open/close 4821 (time 1787467221.20 total 398.58 last 13.23) - open/close 4944 (time 1787467231.95 total 409.33 last 11.45) - open/close 5046 (time 1787467241.96 total 419.34 last 10.19) - open/close 5119 (time 1787467252.06 total 429.43 last 7.23) - open/close 5284 (time 1787467262.07 total 439.45 last 16.47) - open/close 5346 (time 1787467272.08 total 449.45 last 6.20) - open/close 5409 (time 1787467283.10 total 460.48 last 5.71) - open/close 5491 (time 1787467293.67 total 471.05 last 7.76) - open/close 5625 (time 1787467304.09 total 481.47 last 12.86) - open/close 5729 (time 1787467315.03 total 492.40 last 9.51) - open/close 5801 (time 1787467325.39 total 502.77 last 6.95) - open/close 5876 (time 1787467335.54 total 512.92 last 7.39) - open/close 5994 (time 1787467345.55 total 522.93 last 11.79) - open/close 6112 (time 1787467355.58 total 532.96 last 11.76) - open/close 6206 (time 1787467365.59 total 542.97 last 9.39) - open/close 6292 (time 1787467376.29 total 553.67 last 8.04) - open/close 6373 (time 1787467386.83 total 564.21 last 7.69) - open/close 6460 (time 1787467396.84 total 574.22 last 8.69) - open/close 6537 (time 1787467407.17 total 584.55 last 7.45) - open/close 6642 (time 1787467418.20 total 595.58 last 9.52) - open/close 6744 (time 1787467428.40 total 605.77 last 10.01) - open/close 6856 (time 1787467438.81 total 616.18 last 10.76) - open/close 6984 (time 1787467449.49 total 626.87 last 11.98) - open/close 7168 (time 1787467459.50 total 636.88 last 18.39) - open/close 7354 (time 1787467469.51 total 646.89 last 18.57) - open/close 7528 (time 1787467479.53 total 656.91 last 17.37) - open/close 7614 (time 1787467490.26 total 667.63 last 8.02) - open/close 7732 (time 1787467500.81 total 678.19 last 11.18) - open/close 7821 (time 1787467511.51 total 688.88 last 8.32) - open/close 7933 (time 1787467522.02 total 699.39 last 10.66) - open/close 8004 (time 1787467533.32 total 710.70 last 6.28) - open/close 8058 (time 1787467544.34 total 721.71 last 4.90) - open/close 8150 (time 1787467554.34 total 731.72 last 9.20) - open/close 8246 (time 1787467565.20 total 742.58 last 8.83) - open/close 8328 (time 1787467575.63 total 753.01 last 7.86) - open/close 8444 (time 1787467585.86 total 763.24 last 11.34) - open/close 8506 (time 1787467595.99 total 773.37 last 6.12) - open/close 8600 (time 1787467606.74 total 784.12 last 8.74) - open/close 8682 (time 1787467617.57 total 794.94 last 7.58) - open/close 8738 (time 1787467627.59 total 804.96 last 5.59) - open/close 8831 (time 1787467638.23 total 815.61 last 8.74) - open/close 8958 (time 1787467648.67 total 826.05 last 12.16) - open/close 9060 (time 1787467659.80 total 837.17 last 9.17) - open/close 9211 (time 1787467669.81 total 847.18 last 15.08) - open/close 9344 (time 1787467679.83 total 857.20 last 13.27) - open/close 9524 (time 1787467690.45 total 867.82 last 16.95) - open/close 9603 (time 1787467700.75 total 878.12 last 7.67) - open/close 9679 (time 1787467711.24 total 888.61 last 7.25) - open/close 9747 (time 1787467721.30 total 898.68 last 6.76) - open/close 9805 (time 1787467731.31 total 908.68 last 5.80) - open/close 9887 (time 1787467741.31 total 918.69 last 8.20) - open/close 10000 (time 1787467750.65 total 928.03 last 12.10) - open/close 10119 (time 1787467760.73 total 938.10 last 11.81) - open/close 10279 (time 1787467771.04 total 948.41 last 15.52) - open/close 10411 (time 1787467781.06 total 958.44 last 13.17) - open/close 10509 (time 1787467791.07 total 968.45 last 9.79) - open/close 10626 (time 1787467801.98 total 979.36 last 10.72) - open/close 10832 (time 1787467811.99 total 989.37 last 20.58) - open/close 11051 (time 1787467822.01 total 999.38 last 21.87) - open/close 11199 (time 1787467832.01 total 1009.38 last 14.80) - open/close 11295 (time 1787467842.66 total 1020.03 last 9.01) - open/close 11387 (time 1787467853.39 total 1030.77 last 8.57) - open/close 11467 (time 1787467863.56 total 1040.93 last 7.87) - open/close 11664 (time 1787467874.53 total 1051.91 last 17.95) - open/close 11787 (time 1787467884.54 total 1061.92 last 12.29) - open/close 11920 (time 1787467895.07 total 1072.45 last 12.63) - open/close 11999 (time 1787467905.50 total 1082.88 last 7.58) - open/close 12118 (time 1787467915.73 total 1093.11 last 11.64) - open/close 12189 (time 1787467926.61 total 1103.99 last 6.52) - open/close 12566 (time 1787467937.02 total 1114.40 last 36.20) - open/close 13064 (time 1787467947.53 total 1124.91 last 47.40) - open/close 13251 (time 1787467957.76 total 1135.14 last 18.28) - open/close 13461 (time 1787467968.41 total 1145.79 last 19.72) - open/close 13615 (time 1787467978.64 total 1156.01 last 15.06) - open/close 13769 (time 1787467989.54 total 1166.92 last 14.12) - open/close 13885 (time 1787468000.26 total 1177.63 last 10.83) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13937) error: No space left on device total: 13937 open/close in 1181.43 seconds: 11.80 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (114918) - No such process - unlinked 0 (time 1787468019 ; total 0 ; last 0) - unlinked 10000 (time 1787468158 ; total 139 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-13937) error: No such file or directory total: 13937 unlinks in 168 seconds: 82.958336 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2