== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:55:31 (1788000931) start mds service on oleg445-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg445-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 291 (time 1788001013.48 total 10.99 last 26.47) - open/close 427 (time 1788001024.23 total 21.75 last 12.65) - open/close 579 (time 1788001035.25 total 32.77 last 13.79) - open/close 746 (time 1788001045.42 total 42.93 last 16.43) - open/close 817 (time 1788001055.59 total 53.11 last 6.98) - open/close 1041 (time 1788001066.41 total 63.92 last 20.72) - open/close 1126 (time 1788001076.65 total 74.17 last 8.30) - open/close 1345 (time 1788001087.66 total 85.17 last 19.90) - open/close 1528 (time 1788001098.15 total 95.67 last 17.44) - open/close 1674 (time 1788001109.04 total 106.55 last 13.41) - open/close 1770 (time 1788001119.53 total 117.04 last 9.15) - open/close 1920 (time 1788001130.47 total 127.98 last 13.71) - open/close 2078 (time 1788001140.47 total 137.98 last 15.80) - open/close 2365 (time 1788001152.49 total 150.01 last 23.87) - open/close 2441 (time 1788001163.50 total 161.02 last 6.90) - open/close 2669 (time 1788001173.80 total 171.31 last 22.15) - open/close 2912 (time 1788001184.17 total 181.69 last 23.41) - open/close 3155 (time 1788001194.53 total 192.05 last 23.46) - open/close 3511 (time 1788001209.58 total 207.09 last 23.67) - open/close 3758 (time 1788001222.26 total 219.77 last 19.48) - open/close 3941 (time 1788001232.62 total 230.14 last 17.66) - open/close 4114 (time 1788001242.62 total 240.14 last 17.30) - open/close 4371 (time 1788001253.30 total 250.81 last 24.07) - open/close 4436 (time 1788001263.47 total 260.98 last 6.39) - open/close 4586 (time 1788001279.34 total 276.86 last 9.45) - open/close 4777 (time 1788001290.42 total 287.93 last 17.25) - open/close 5059 (time 1788001300.42 total 297.93 last 28.20) - open/close 5232 (time 1788001311.34 total 308.85 last 15.84) - open/close 5516 (time 1788001321.34 total 318.86 last 28.39) - open/close 5783 (time 1788001331.88 total 329.39 last 25.34) - open/close 6098 (time 1788001342.89 total 340.41 last 28.60) - open/close 6276 (time 1788001353.00 total 350.51 last 17.61) - open/close 6587 (time 1788001363.01 total 360.52 last 31.07) - open/close 6824 (time 1788001373.35 total 370.86 last 22.92) - open/close 7045 (time 1788001383.72 total 381.23 last 21.32) - open/close 7302 (time 1788001394.30 total 391.81 last 24.29) - open/close 7571 (time 1788001404.97 total 402.49 last 25.20) - open/close 7899 (time 1788001414.98 total 412.50 last 32.77) - open/close 8076 (time 1788001424.99 total 422.50 last 17.69) - open/close 8280 (time 1788001434.99 total 432.50 last 20.40) - open/close 8495 (time 1788001445.30 total 442.81 last 20.86) - open/close 8785 (time 1788001455.30 total 452.81 last 28.99) - open/close 8968 (time 1788001465.39 total 462.90 last 18.14) - open/close 9261 (time 1788001475.39 total 472.91 last 29.27) - open/close 9536 (time 1788001486.00 total 483.52 last 25.92) - open/close 9823 (time 1788001496.04 total 493.56 last 28.59) - open/close 10000 (time 1788001500.57 total 498.09 last 39.07) - open/close 10329 (time 1788001510.58 total 508.09 last 32.90) - open/close 10563 (time 1788001520.74 total 518.25 last 23.02) - open/close 11378 (time 1788001530.75 total 528.26 last 81.45) - open/close 12488 (time 1788001540.75 total 538.26 last 110.96) - open/close 13610 (time 1788001550.75 total 548.27 last 112.14) - open/close 14742 (time 1788001560.76 total 558.27 last 113.17) - open/close 15790 (time 1788001570.76 total 568.28 last 104.72) - open/close 16859 (time 1788001580.77 total 578.29 last 106.81) - open/close 17919 (time 1788001590.78 total 588.29 last 105.96) - open/close 19015 (time 1788001600.78 total 598.29 last 109.57) - open/close 20000 (time 1788001609.64 total 607.15 last 111.20) - open/close 21161 (time 1788001619.64 total 617.16 last 116.04) - open/close 22268 (time 1788001629.66 total 627.17 last 110.56) - open/close 23409 (time 1788001639.66 total 637.17 last 114.07) - open/close 24550 (time 1788001649.66 total 647.18 last 114.04) - open/close 25629 (time 1788001659.66 total 657.18 last 107.90) - open/close 26911 (time 1788001669.66 total 667.18 last 128.20) - open/close 28100 (time 1788001679.66 total 677.18 last 118.90) - open/close 29350 (time 1788001689.67 total 687.19 last 124.91) - open/close 30000 (time 1788001694.10 total 691.62 last 146.68) - open/close 31826 (time 1788001704.11 total 701.63 last 182.45) - open/close 33196 (time 1788001714.11 total 711.63 last 137.00) - open/close 34442 (time 1788001724.11 total 721.63 last 124.56) - open/close 35711 (time 1788001734.12 total 731.63 last 126.85) - open/close 37067 (time 1788001744.12 total 741.63 last 135.57) - open/close 38329 (time 1788001754.13 total 751.65 last 126.01) - open/close 39456 (time 1788001764.14 total 761.66 last 112.63) - open/close 40000 (time 1788001768.76 total 766.28 last 117.68) - open/close 41183 (time 1788001778.77 total 776.28 last 118.27) - open/close 42502 (time 1788001788.77 total 786.29 last 131.81) - open/close 44199 (time 1788001798.77 total 796.29 last 169.67) - open/close 45460 (time 1788001808.79 total 806.30 last 125.94) - open/close 46868 (time 1788001818.79 total 816.30 last 140.77) - open/close 48224 (time 1788001828.79 total 826.30 last 135.60) - open/close 49554 (time 1788001838.79 total 836.31 last 132.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 836.85 seconds: 59.31 ops/second - unlinked 0 (time 1788001841 ; total 0 ; last 0) - unlinked 10000 (time 1788001890 ; total 49 ; last 49) - unlinked 20000 (time 1788001927 ; total 86 ; last 37) - unlinked 30000 (time 1788001958 ; total 117 ; last 31) - unlinked 40000 (time 1788001995 ; total 154 ; last 37) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 182 seconds: 272.697815 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: modules unloaded. pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2