== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:59:18 (1788274758) start mds service on oleg201-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg201-server: detected 4 online CPUs by sysfs oleg201-server: MODOPTS_LIBCFS= oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg201-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1788274853.01 total 10.01 last 18.28) - open/close 396 (time 1788274864.01 total 21.01 last 19.37) - open/close 551 (time 1788274875.07 total 32.07 last 14.01) - open/close 739 (time 1788274885.76 total 42.76 last 17.60) - open/close 815 (time 1788274896.58 total 53.57 last 7.03) - open/close 930 (time 1788274906.63 total 63.62 last 11.44) - open/close 1176 (time 1788274917.64 total 74.64 last 22.33) - open/close 1306 (time 1788274928.55 total 85.55 last 11.92) - open/close 1441 (time 1788274938.56 total 95.56 last 13.49) - open/close 1558 (time 1788274948.75 total 105.75 last 11.48) - open/close 1625 (time 1788274959.15 total 116.15 last 6.44) - open/close 1756 (time 1788274970.16 total 127.16 last 11.89) - open/close 1874 (time 1788274981.18 total 138.18 last 10.71) - open/close 1980 (time 1788274991.84 total 148.84 last 9.94) - open/close 2204 (time 1788275002.59 total 159.58 last 20.85) - open/close 2350 (time 1788275013.12 total 170.12 last 13.86) - open/close 2508 (time 1788275023.12 total 180.12 last 15.79) - open/close 2661 (time 1788275033.59 total 190.59 last 14.61) - open/close 2815 (time 1788275043.60 total 200.60 last 15.39) - open/close 2949 (time 1788275054.27 total 211.26 last 12.56) - open/close 3106 (time 1788275064.95 total 221.95 last 14.69) - open/close 3230 (time 1788275075.46 total 232.46 last 11.80) - open/close 3385 (time 1788275085.88 total 242.88 last 14.88) - open/close 3463 (time 1788275098.24 total 255.24 last 6.31) - open/close 3674 (time 1788275108.29 total 265.29 last 20.99) - open/close 3787 (time 1788275118.29 total 275.29 last 11.30) - open/close 3937 (time 1788275129.97 total 286.97 last 12.84) - open/close 4175 (time 1788275139.99 total 296.99 last 23.76) - open/close 4352 (time 1788275150.01 total 307.00 last 17.67) - open/close 4514 (time 1788275160.64 total 317.64 last 15.23) - open/close 4637 (time 1788275170.65 total 327.65 last 12.28) - open/close 4753 (time 1788275180.67 total 337.67 last 11.58) - open/close 4877 (time 1788275191.47 total 348.47 last 11.49) - open/close 5044 (time 1788275201.99 total 358.99 last 15.88) - open/close 5172 (time 1788275212.36 total 369.35 last 12.35) - open/close 5270 (time 1788275223.35 total 380.35 last 8.91) - open/close 5423 (time 1788275233.38 total 390.38 last 15.25) - open/close 5548 (time 1788275243.83 total 400.83 last 11.96) - open/close 5755 (time 1788275253.89 total 410.89 last 20.58) - open/close 5872 (time 1788275264.84 total 421.84 last 10.69) - open/close 5972 (time 1788275274.85 total 431.85 last 9.99) - open/close 6071 (time 1788275284.96 total 441.96 last 9.79) - open/close 6161 (time 1788275297.47 total 454.47 last 7.20) - open/close 6301 (time 1788275308.08 total 465.08 last 13.19) - open/close 6421 (time 1788275318.10 total 475.09 last 11.99) - open/close 6609 (time 1788275328.33 total 485.33 last 18.37) - open/close 6706 (time 1788275338.68 total 495.68 last 9.37) - open/close 6960 (time 1788275349.31 total 506.31 last 23.90) - open/close 7090 (time 1788275359.45 total 516.45 last 12.82) - open/close 7317 (time 1788275369.45 total 526.45 last 22.68) - open/close 7368 (time 1788275380.48 total 537.48 last 4.62) - open/close 7566 (time 1788275390.92 total 547.91 last 18.98) - open/close 7708 (time 1788275400.92 total 557.92 last 14.20) - open/close 7901 (time 1788275411.96 total 568.96 last 17.48) - open/close 8013 (time 1788275421.98 total 578.98 last 11.18) - open/close 8150 (time 1788275432.92 total 589.92 last 12.53) - open/close 8258 (time 1788275442.92 total 599.92 last 10.80) - open/close 8442 (time 1788275453.39 total 610.39 last 17.57) - open/close 8559 (time 1788275463.88 total 620.88 last 11.16) - open/close 8697 (time 1788275474.04 total 631.04 last 13.58) - open/close 8897 (time 1788275484.88 total 641.88 last 18.45) - open/close 9108 (time 1788275494.90 total 651.89 last 21.07) - open/close 9263 (time 1788275505.56 total 662.56 last 14.53) - open/close 9341 (time 1788275515.58 total 672.57 last 7.79) - open/close 9500 (time 1788275525.58 total 682.58 last 15.89) - open/close 9635 (time 1788275536.42 total 693.41 last 12.46) - open/close 9805 (time 1788275547.06 total 704.06 last 15.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9905) error: No space left on device total: 9905 open/close in 708.33 seconds: 13.98 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76083) - No such process - unlinked 0 (time 1788275563 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9905) error: No such file or directory total: 9905 unlinks in 129 seconds: 76.782944 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2