== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:10:16 (1781820616) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg257-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1781820693.65 total 10.16 last 24.41) - open/close 573 (time 1781820704.67 total 21.17 last 29.51) - open/close 901 (time 1781820715.62 total 32.13 last 29.95) - open/close 1111 (time 1781820726.55 total 43.06 last 19.20) - open/close 1359 (time 1781820736.88 total 53.39 last 24.01) - open/close 1617 (time 1781820747.29 total 63.79 last 24.80) - open/close 1876 (time 1781820757.59 total 74.10 last 25.12) - open/close 2178 (time 1781820767.59 total 84.10 last 30.20) - open/close 2292 (time 1781820778.01 total 94.52 last 10.95) - open/close 2469 (time 1781820788.02 total 104.52 last 17.69) - open/close 2634 (time 1781820799.90 total 116.41 last 13.88) - open/close 2899 (time 1781820810.53 total 127.04 last 24.93) - open/close 3198 (time 1781820820.53 total 137.04 last 29.90) - open/close 3447 (time 1781820830.53 total 147.04 last 24.90) - open/close 3786 (time 1781820841.56 total 158.07 last 30.74) - open/close 3960 (time 1781820853.97 total 170.48 last 14.02) - open/close 4242 (time 1781820866.59 total 183.09 last 22.35) - open/close 4479 (time 1781820877.53 total 194.04 last 21.65) - open/close 4780 (time 1781820888.02 total 204.53 last 28.70) - open/close 4999 (time 1781820898.77 total 215.28 last 20.38) - open/close 5327 (time 1781820908.83 total 225.34 last 32.60) - open/close 5580 (time 1781820919.76 total 236.27 last 23.15) - open/close 5993 (time 1781820930.14 total 246.65 last 39.78) - open/close 6194 (time 1781820940.76 total 257.26 last 18.94) - open/close 6474 (time 1781820951.01 total 267.51 last 27.32) - open/close 6828 (time 1781820961.01 total 277.52 last 35.37) - open/close 7176 (time 1781820971.99 total 288.50 last 31.70) - open/close 7557 (time 1781820982.31 total 298.82 last 36.92) - open/close 7801 (time 1781820992.75 total 309.26 last 23.37) - open/close 8064 (time 1781821003.81 total 320.32 last 23.79) - open/close 8425 (time 1781821014.05 total 330.56 last 35.25) - open/close 8754 (time 1781821024.60 total 341.10 last 31.19) - open/close 9037 (time 1781821034.60 total 351.10 last 28.30) - open/close 9323 (time 1781821045.53 total 362.03 last 26.17) - open/close 9473 (time 1781821055.83 total 372.34 last 14.56) - open/close 9740 (time 1781821065.84 total 382.35 last 26.66) - open/close 9923 (time 1781821075.85 total 392.36 last 18.29) - open/close 10244 (time 1781821085.85 total 402.36 last 32.10) - open/close 10513 (time 1781821096.02 total 412.53 last 26.45) - open/close 10737 (time 1781821106.96 total 423.47 last 20.47) - open/close 10993 (time 1781821116.97 total 433.48 last 25.57) - open/close 11298 (time 1781821127.77 total 444.28 last 28.25) - open/close 11607 (time 1781821138.20 total 454.71 last 29.63) - open/close 12507 (time 1781821148.20 total 464.71 last 89.97) - open/close 13977 (time 1781821158.21 total 474.72 last 146.92) - open/close 15255 (time 1781821168.21 total 484.72 last 127.75) - open/close 16495 (time 1781821178.22 total 494.73 last 123.93) - open/close 17784 (time 1781821188.22 total 504.73 last 128.88) - open/close 19126 (time 1781821198.22 total 514.73 last 134.20) - open/close 20000 (time 1781821205.26 total 521.77 last 124.18) - open/close 21397 (time 1781821215.26 total 531.77 last 139.64) - open/close 22631 (time 1781821225.27 total 541.77 last 123.35) - open/close 23864 (time 1781821235.27 total 551.78 last 123.29) - open/close 25072 (time 1781821245.27 total 561.78 last 120.77) - open/close 26480 (time 1781821255.28 total 571.79 last 140.70) - open/close 27814 (time 1781821265.28 total 581.79 last 133.31) - open/close 28897 (time 1781821275.28 total 591.79 last 108.30) - open/close 30000 (time 1781821284.61 total 601.11 last 118.32) - open/close 31451 (time 1781821294.61 total 611.12 last 145.03) - open/close 32757 (time 1781821304.62 total 621.13 last 130.50) - open/close 33889 (time 1781821314.64 total 631.14 last 113.02) - open/close 34994 (time 1781821324.64 total 641.14 last 110.48) - open/close 36226 (time 1781821334.64 total 651.15 last 123.16) - open/close 37373 (time 1781821344.65 total 661.15 last 114.63) - open/close 38496 (time 1781821354.65 total 671.16 last 112.28) - open/close 39708 (time 1781821364.65 total 681.16 last 121.13) - open/close 40000 (time 1781821366.83 total 683.34 last 134.03) - open/close 41303 (time 1781821376.83 total 693.34 last 130.27) - open/close 42635 (time 1781821386.84 total 703.34 last 133.17) - open/close 44066 (time 1781821396.85 total 713.35 last 142.94) - open/close 45496 (time 1781821406.85 total 723.36 last 142.98) - open/close 46839 (time 1781821416.85 total 733.36 last 134.23) - open/close 48142 (time 1781821426.86 total 743.37 last 130.25) - open/close 49448 (time 1781821436.86 total 753.37 last 130.59) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 754.70 seconds: 65.76 ops/second - unlinked 0 (time 1781821440 ; total 0 ; last 0) - unlinked 10000 (time 1781821494 ; total 54 ; last 54) - unlinked 20000 (time 1781821531 ; total 91 ; last 37) - unlinked 30000 (time 1781821570 ; total 130 ; last 39) - unlinked 40000 (time 1781821611 ; total 171 ; last 41) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 212 seconds: 234.108490 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2