== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:19:18 (1783142358) start mds service on oleg230-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing load_modules_local oleg230-server: Loading modules from /home/green/git/lustre-release/lustre oleg230-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg230-server: detected 4 online CPUs by sysfs oleg230-server: MODOPTS_LIBCFS= oleg230-server: Force libcfs to create 2 CPU partitions oleg230-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg230-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg230-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg230-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg230-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug -1 all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1783142468.04 total 10.89 last 19.01) - open/close 451 (time 1783142478.04 total 20.90 last 24.39) - open/close 689 (time 1783142488.07 total 30.93 last 23.73) - open/close 887 (time 1783142498.09 total 40.94 last 19.77) - open/close 1123 (time 1783142508.34 total 51.20 last 23.01) - open/close 1462 (time 1783142519.42 total 62.27 last 30.62) - open/close 1696 (time 1783142529.92 total 72.77 last 22.28) - open/close 1995 (time 1783142540.22 total 83.07 last 29.04) - open/close 2177 (time 1783142550.89 total 93.75 last 17.04) - open/close 2411 (time 1783142561.15 total 104.01 last 22.81) - open/close 2639 (time 1783142571.71 total 114.56 last 21.60) - open/close 2879 (time 1783142581.72 total 124.57 last 23.97) - open/close 3066 (time 1783142592.57 total 135.43 last 17.23) - open/close 3269 (time 1783142603.46 total 146.31 last 18.64) - open/close 3482 (time 1783142613.74 total 156.60 last 20.71) - open/close 3749 (time 1783142623.76 total 166.61 last 26.65) - open/close 3967 (time 1783142633.99 total 176.85 last 21.30) - open/close 4255 (time 1783142644.01 total 186.87 last 28.75) - open/close 4559 (time 1783142654.26 total 197.11 last 29.66) - open/close 4841 (time 1783142664.70 total 207.55 last 27.01) - open/close 5096 (time 1783142675.64 total 218.50 last 23.30) - open/close 5356 (time 1783142686.72 total 229.57 last 23.48) - open/close 5622 (time 1783142697.33 total 240.18 last 25.06) - open/close 5874 (time 1783142707.39 total 250.24 last 25.06) - open/close 6150 (time 1783142718.03 total 260.88 last 25.94) - open/close 6427 (time 1783142728.03 total 270.89 last 27.68) - open/close 6641 (time 1783142738.41 total 281.27 last 20.62) - open/close 6889 (time 1783142749.43 total 292.28 last 22.52) - open/close 7193 (time 1783142759.81 total 302.66 last 29.29) - open/close 7415 (time 1783142769.84 total 312.70 last 22.12) - open/close 7660 (time 1783142780.34 total 323.19 last 23.34) - open/close 7906 (time 1783142791.03 total 333.88 last 23.02) - open/close 8198 (time 1783142801.39 total 344.24 last 28.18) - open/close 8475 (time 1783142811.57 total 354.42 last 27.21) - open/close 8702 (time 1783142821.96 total 364.81 last 21.85) - open/close 8910 (time 1783142831.97 total 374.83 last 20.77) - open/close 9134 (time 1783142842.20 total 385.05 last 21.90) - open/close 9414 (time 1783142853.23 total 396.09 last 25.38) - open/close 9624 (time 1783142863.26 total 406.12 last 20.94) - open/close 9831 (time 1783142873.66 total 416.51 last 19.91) - open/close 9979 (time 1783142883.89 total 426.74 last 14.47) - open/close 10192 (time 1783142894.90 total 437.75 last 19.35) - open/close 10465 (time 1783142905.02 total 447.87 last 26.97) - open/close 10685 (time 1783142915.05 total 457.91 last 21.92) - open/close 10891 (time 1783142925.18 total 468.03 last 20.35) - open/close 11151 (time 1783142935.19 total 478.04 last 25.96) - open/close 11354 (time 1783142945.92 total 488.77 last 18.92) - open/close 11536 (time 1783142955.92 total 498.78 last 18.19) - open/close 11701 (time 1783142966.39 total 509.25 last 15.76) - open/close 11912 (time 1783142977.27 total 520.12 last 19.40) - open/close 12151 (time 1783142988.29 total 531.15 last 21.68) - open/close 12402 (time 1783142998.30 total 541.15 last 25.08) - open/close 12707 (time 1783143008.30 total 551.16 last 30.49) - open/close 12924 (time 1783143019.00 total 561.86 last 20.28) - open/close 13138 (time 1783143033.08 total 575.94 last 15.20) - open/close 13354 (time 1783143043.32 total 586.17 last 21.11) - open/close 13605 (time 1783143054.06 total 596.92 last 23.36) - open/close 13782 (time 1783143064.50 total 607.36 last 16.96) - open/close 13983 (time 1783143075.44 total 618.30 last 18.37) - open/close 14190 (time 1783143085.76 total 628.61 last 20.07) - open/close 14369 (time 1783143096.18 total 639.03 last 17.18) - open/close 14495 (time 1783143106.19 total 649.05 last 12.58) - open/close 14687 (time 1783143117.17 total 660.03 last 17.48) - open/close 14833 (time 1783143127.30 total 670.15 last 14.42) - open/close 15032 (time 1783143138.19 total 681.05 last 18.26) - open/close 15228 (time 1783143149.04 total 691.90 last 18.06) - open/close 15398 (time 1783143159.53 total 702.39 last 16.21) - open/close 15526 (time 1783143169.53 total 712.39 last 12.80) - open/close 15642 (time 1783143179.54 total 722.39 last 11.60) - open/close 15795 (time 1783143189.54 total 732.39 last 15.30) - open/close 16840 (time 1783143199.54 total 742.40 last 104.43) - open/close 17865 (time 1783143209.55 total 752.41 last 102.38) - open/close 19037 (time 1783143219.56 total 762.41 last 117.18) - open/close 20000 (time 1783143228.72 total 771.58 last 105.08) - open/close 21042 (time 1783143238.73 total 781.58 last 104.13) - open/close 22076 (time 1783143248.73 total 791.59 last 103.36) - open/close 23073 (time 1783143258.74 total 801.59 last 99.65) - open/close 24196 (time 1783143268.74 total 811.60 last 112.26) - open/close 25281 (time 1783143278.74 total 821.60 last 108.46) - open/close 26205 (time 1783143288.75 total 831.60 last 92.38) - open/close 27203 (time 1783143298.76 total 841.61 last 99.71) - open/close 28280 (time 1783143308.76 total 851.62 last 107.62) - open/close 29347 (time 1783143318.77 total 861.62 last 106.66) - open/close 30000 (time 1783143325.57 total 868.43 last 95.95) - open/close 30983 (time 1783143335.58 total 878.43 last 98.25) - open/close 32050 (time 1783143345.58 total 888.44 last 106.65) - open/close 33190 (time 1783143355.59 total 898.45 last 113.87) - open/close 34188 (time 1783143365.60 total 908.45 last 99.75) - open/close 35225 (time 1783143375.61 total 918.46 last 103.62) - open/close 36314 (time 1783143385.61 total 928.46 last 108.87) - open/close 37560 (time 1783143395.61 total 938.47 last 124.52) - open/close 38850 (time 1783143405.62 total 948.48 last 128.90) - open/close 40000 (time 1783143415.01 total 957.86 last 122.50) - open/close 41108 (time 1783143425.02 total 967.87 last 110.74) - open/close 42369 (time 1783143435.02 total 977.87 last 126.05) - open/close 43586 (time 1783143445.02 total 987.87 last 121.70) - open/close 44930 (time 1783143455.02 total 997.88 last 134.34) - open/close 46328 (time 1783143465.03 total 1007.89 last 139.68) - open/close 47671 (time 1783143475.04 total 1017.89 last 134.24) - open/close 49064 (time 1783143485.05 total 1027.90 last 139.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1031.88 seconds: 48.10 ops/second - unlinked 0 (time 1783143490 ; total 0 ; last 0) - unlinked 10000 (time 1783143562 ; total 72 ; last 72) - unlinked 20000 (time 1783143605 ; total 115 ; last 43) - unlinked 30000 (time 1783143642 ; total 152 ; last 37) - unlinked 40000 (time 1783143680 ; total 190 ; last 38) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 229 seconds: 216.729263 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg230-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg230-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg230-server stop mds service on oleg230-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg230-server unloading modules via unload_modules_local on: 'oleg230-server' oleg230-server: oleg230-server.virtnet: executing unload_modules_local oleg230-server: modules unloaded. pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-client: ssh exited with exit code 2 pdsh@oleg230-client: oleg230-server: ssh exited with exit code 2