== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:12:06 (1780326726) start mds service on oleg212-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing load_modules_local oleg212-server: Loading modules from /home/green/git/lustre-release/lustre oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg212-server: detected 4 online CPUs by sysfs oleg212-server: MODOPTS_LIBCFS= oleg212-server: Force libcfs to create 2 CPU partitions oleg212-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg212-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg212-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg212-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg212-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 282 (time 1780326792.86 total 10.70 last 26.35) - open/close 541 (time 1780326802.86 total 20.71 last 25.88) - open/close 862 (time 1780326812.87 total 30.71 last 32.08) - open/close 1026 (time 1780326822.88 total 40.73 last 16.38) - open/close 1212 (time 1780326832.89 total 50.73 last 18.59) - open/close 1308 (time 1780326843.17 total 61.01 last 9.34) - open/close 1542 (time 1780326854.01 total 71.85 last 21.58) - open/close 1786 (time 1780326864.02 total 81.86 last 24.38) - open/close 2019 (time 1780326874.03 total 91.87 last 23.28) - open/close 2320 (time 1780326884.83 total 102.68 last 27.85) - open/close 2534 (time 1780326895.15 total 112.99 last 20.74) - open/close 2802 (time 1780326906.14 total 123.98 last 24.39) - open/close 3086 (time 1780326917.02 total 134.87 last 26.09) - open/close 3423 (time 1780326927.03 total 144.87 last 33.69) - open/close 3643 (time 1780326937.20 total 155.04 last 21.63) - open/close 3880 (time 1780326947.20 total 165.05 last 23.68) - open/close 4104 (time 1780326957.73 total 175.57 last 21.29) - open/close 4307 (time 1780326967.84 total 185.68 last 20.07) - open/close 4577 (time 1780326977.85 total 195.69 last 26.98) - open/close 4801 (time 1780326988.45 total 206.30 last 21.12) - open/close 5024 (time 1780326998.88 total 216.72 last 21.39) - open/close 5236 (time 1780327009.39 total 227.23 last 20.17) - open/close 5467 (time 1780327019.80 total 237.64 last 22.18) - open/close 5781 (time 1780327030.75 total 248.59 last 28.69) - open/close 6055 (time 1780327040.75 total 258.59 last 27.39) - open/close 6276 (time 1780327051.82 total 269.66 last 19.97) - open/close 6525 (time 1780327062.63 total 280.47 last 23.02) - open/close 6787 (time 1780327073.56 total 291.40 last 23.97) - open/close 7037 (time 1780327083.57 total 301.41 last 24.97) - open/close 7267 (time 1780327094.01 total 311.85 last 22.03) - open/close 7514 (time 1780327105.00 total 322.84 last 22.48) - open/close 7773 (time 1780327115.01 total 332.85 last 25.88) - open/close 7988 (time 1780327125.28 total 343.12 last 20.93) - open/close 8242 (time 1780327135.28 total 353.13 last 25.39) - open/close 8446 (time 1780327145.29 total 363.13 last 20.38) - open/close 8673 (time 1780327155.51 total 373.35 last 22.22) - open/close 8871 (time 1780327165.53 total 383.37 last 19.76) - open/close 9107 (time 1780327175.55 total 393.39 last 23.56) - open/close 9332 (time 1780327186.08 total 403.92 last 21.36) - open/close 9599 (time 1780327196.08 total 413.93 last 26.69) - open/close 9823 (time 1780327206.94 total 424.78 last 20.64) - open/close 10000 (time 1780327213.17 total 431.01 last 28.40) - open/close 10265 (time 1780327223.45 total 441.29 last 25.78) - open/close 10473 (time 1780327233.64 total 451.49 last 20.40) - open/close 10779 (time 1780327244.01 total 461.85 last 29.53) - open/close 10996 (time 1780327254.42 total 472.27 last 20.83) - open/close 11261 (time 1780327264.44 total 482.28 last 26.46) - open/close 11520 (time 1780327274.45 total 492.29 last 25.87) - open/close 11751 (time 1780327284.57 total 502.41 last 22.82) - open/close 11969 (time 1780327295.33 total 513.18 last 20.25) - open/close 12187 (time 1780327305.36 total 523.20 last 21.75) - open/close 12398 (time 1780327316.34 total 534.18 last 19.22) - open/close 12606 (time 1780327326.34 total 544.18 last 20.79) - open/close 12830 (time 1780327336.54 total 554.38 last 21.96) - open/close 13033 (time 1780327347.55 total 565.40 last 18.43) - open/close 13257 (time 1780327357.78 total 575.63 last 21.90) - open/close 13493 (time 1780327367.78 total 585.63 last 23.60) - open/close 13740 (time 1780327378.14 total 595.99 last 23.84) - open/close 13987 (time 1780327388.15 total 605.99 last 24.68) - open/close 14214 (time 1780327398.49 total 616.34 last 21.95) - open/close 14403 (time 1780327408.50 total 626.34 last 18.89) - open/close 14673 (time 1780327418.51 total 636.35 last 26.97) - open/close 14868 (time 1780327429.48 total 647.32 last 17.78) - open/close 15045 (time 1780327440.30 total 658.14 last 16.37) - open/close 15228 (time 1780327451.04 total 668.89 last 17.03) - open/close 15418 (time 1780327461.04 total 678.89 last 19.00) - open/close 15677 (time 1780327471.84 total 689.69 last 23.98) - open/close 15876 (time 1780327482.46 total 700.30 last 18.75) - open/close 16009 (time 1780327492.47 total 710.31 last 13.29) - open/close 16123 (time 1780327502.47 total 720.31 last 11.40) - open/close 16292 (time 1780327512.54 total 730.38 last 16.77) - open/close 16403 (time 1780327523.23 total 741.07 last 10.39) - open/close 16619 (time 1780327533.91 total 751.76 last 20.21) - open/close 16755 (time 1780327543.92 total 761.76 last 13.59) - open/close 17871 (time 1780327553.92 total 771.76 last 111.60) - open/close 19158 (time 1780327563.93 total 781.77 last 128.62) - open/close 20000 (time 1780327570.17 total 788.01 last 134.95) - open/close 21303 (time 1780327580.17 total 798.01 last 130.21) - open/close 22439 (time 1780327590.18 total 808.02 last 113.54) - open/close 23565 (time 1780327600.18 total 818.02 last 112.56) - open/close 24793 (time 1780327610.18 total 828.02 last 122.79) - open/close 26005 (time 1780327620.19 total 838.03 last 121.14) - open/close 27257 (time 1780327630.19 total 848.03 last 125.19) - open/close 28599 (time 1780327640.19 total 858.04 last 134.11) - open/close 29702 (time 1780327650.20 total 868.04 last 110.26) - open/close 30000 (time 1780327652.54 total 870.38 last 127.21) - open/close 31266 (time 1780327662.55 total 880.39 last 126.55) - open/close 32629 (time 1780327672.55 total 890.39 last 136.24) - open/close 33926 (time 1780327682.55 total 900.39 last 129.68) - open/close 35094 (time 1780327692.56 total 910.40 last 116.71) - open/close 36322 (time 1780327702.57 total 920.41 last 122.72) - open/close 37755 (time 1780327712.57 total 930.41 last 143.27) - open/close 39203 (time 1780327722.57 total 940.41 last 144.74) - open/close 40000 (time 1780327728.68 total 946.52 last 130.44) - open/close 41290 (time 1780327738.68 total 956.53 last 128.97) - open/close 42602 (time 1780327748.69 total 966.53 last 131.15) - open/close 43835 (time 1780327758.69 total 976.54 last 123.22) - open/close 45099 (time 1780327768.69 total 986.54 last 126.40) - open/close 46576 (time 1780327778.70 total 996.54 last 147.62) - open/close 47892 (time 1780327788.70 total 1006.54 last 131.59) - open/close 49091 (time 1780327798.71 total 1016.55 last 119.83) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1020.91 seconds: 48.61 ops/second - unlinked 0 (time 1780327805 ; total 0 ; last 0) - unlinked 10000 (time 1780327920 ; total 115 ; last 115) - unlinked 20000 (time 1780327995 ; total 190 ; last 75) - unlinked 30000 (time 1780328054 ; total 249 ; last 59) - unlinked 40000 (time 1780328109 ; total 304 ; last 55) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 356 seconds: 139.412918 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg212-server unloading modules via unload_modules_local on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing unload_modules_local oleg212-server: modules unloaded. pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2