== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:10:10 (1781237410) start mds service on oleg455-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg455-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg455-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 209 (time 1781237488.68 total 10.01 last 20.87) - open/close 419 (time 1781237499.70 total 21.04 last 19.04) - open/close 659 (time 1781237510.08 total 31.42 last 23.12) - open/close 913 (time 1781237520.49 total 41.83 last 24.41) - open/close 1148 (time 1781237530.50 total 51.84 last 23.48) - open/close 1439 (time 1781237540.50 total 61.84 last 29.10) - open/close 1622 (time 1781237550.97 total 72.30 last 17.48) - open/close 1836 (time 1781237561.66 total 83.00 last 20.02) - open/close 2068 (time 1781237571.89 total 93.23 last 22.67) - open/close 2320 (time 1781237582.46 total 103.80 last 23.85) - open/close 2582 (time 1781237592.47 total 113.81 last 26.16) - open/close 2778 (time 1781237603.00 total 124.34 last 18.62) - open/close 3030 (time 1781237613.42 total 134.76 last 24.19) - open/close 3308 (time 1781237623.74 total 145.08 last 26.94) - open/close 3554 (time 1781237634.17 total 155.50 last 23.60) - open/close 3672 (time 1781237644.79 total 166.12 last 11.11) - open/close 3793 (time 1781237655.49 total 176.82 last 11.31) - open/close 3974 (time 1781237665.49 total 186.83 last 18.09) - open/close 4180 (time 1781237676.54 total 197.88 last 18.64) - open/close 4355 (time 1781237687.23 total 208.56 last 16.38) - open/close 4595 (time 1781237698.03 total 219.37 last 22.21) - open/close 4829 (time 1781237708.79 total 230.12 last 21.76) - open/close 4951 (time 1781237718.95 total 240.28 last 12.01) - open/close 5085 (time 1781237729.70 total 251.04 last 12.46) - open/close 5362 (time 1781237739.72 total 261.06 last 27.64) - open/close 5571 (time 1781237749.88 total 271.22 last 20.57) - open/close 5827 (time 1781237759.89 total 281.23 last 25.58) - open/close 6053 (time 1781237770.31 total 291.65 last 21.68) - open/close 6196 (time 1781237781.43 total 302.77 last 12.86) - open/close 6427 (time 1781237791.43 total 312.77 last 23.09) - open/close 6676 (time 1781237801.91 total 323.24 last 23.77) - open/close 6935 (time 1781237812.22 total 333.56 last 25.11) - open/close 7223 (time 1781237822.23 total 343.56 last 28.79) - open/close 7463 (time 1781237832.24 total 353.58 last 23.96) - open/close 7670 (time 1781237843.30 total 364.63 last 18.73) - open/close 7809 (time 1781237853.94 total 375.27 last 13.06) - open/close 8063 (time 1781237866.96 total 388.30 last 19.50) - open/close 8248 (time 1781237876.99 total 398.32 last 18.45) - open/close 8377 (time 1781237887.94 total 409.27 last 11.78) - open/close 8687 (time 1781237897.94 total 419.28 last 30.98) - open/close 8923 (time 1781237908.40 total 429.74 last 22.56) - open/close 9112 (time 1781237918.85 total 440.18 last 18.10) - open/close 9336 (time 1781237928.86 total 450.19 last 22.38) - open/close 9531 (time 1781237938.88 total 460.22 last 19.45) - open/close 9723 (time 1781237949.81 total 471.15 last 17.56) - open/close 9946 (time 1781237959.99 total 481.33 last 21.91) - open/close 10000 (time 1781237963.98 total 485.31 last 13.56) - open/close 10128 (time 1781237973.98 total 495.32 last 12.80) - open/close 10215 (time 1781237986.41 total 507.75 last 7.00) - open/close 10321 (time 1781237996.71 total 518.05 last 10.29) - open/close 10405 (time 1781238006.72 total 528.05 last 8.39) - open/close 10587 (time 1781238016.75 total 538.09 last 18.14) - open/close 10827 (time 1781238026.78 total 548.11 last 23.94) - open/close 10990 (time 1781238041.53 total 562.87 last 11.05) - open/close 11191 (time 1781238051.72 total 573.06 last 19.73) - open/close 11404 (time 1781238062.07 total 583.41 last 20.57) - open/close 11655 (time 1781238072.25 total 593.59 last 24.65) - open/close 11852 (time 1781238082.54 total 603.88 last 19.15) - open/close 12105 (time 1781238092.55 total 613.89 last 25.27) - open/close 12233 (time 1781238102.58 total 623.92 last 12.76) - open/close 12320 (time 1781238112.63 total 633.97 last 8.66) - open/close 12521 (time 1781238122.82 total 644.16 last 19.74) - open/close 12681 (time 1781238133.00 total 654.33 last 15.72) - open/close 12887 (time 1781238143.01 total 664.35 last 20.57) - open/close 13099 (time 1781238153.39 total 674.73 last 20.42) - open/close 13304 (time 1781238163.59 total 684.93 last 20.10) - open/close 13616 (time 1781238174.62 total 695.96 last 28.30) - open/close 13770 (time 1781238184.83 total 706.17 last 15.08) - open/close 13966 (time 1781238196.85 total 718.18 last 16.31) - open/close 14130 (time 1781238207.36 total 728.70 last 15.60) - open/close 14278 (time 1781238218.44 total 739.77 last 13.36) - open/close 14493 (time 1781238228.46 total 749.79 last 21.46) - open/close 14699 (time 1781238238.48 total 759.81 last 20.56) - open/close 14867 (time 1781238249.03 total 770.37 last 15.91) - open/close 15060 (time 1781238259.27 total 780.61 last 18.85) - open/close 15306 (time 1781238269.28 total 790.61 last 24.59) - open/close 15544 (time 1781238279.29 total 800.63 last 23.77) - open/close 15656 (time 1781238289.91 total 811.24 last 10.55) - open/close 15879 (time 1781238300.74 total 822.08 last 20.59) - open/close 15942 (time 1781238311.57 total 832.90 last 5.82) - open/close 16084 (time 1781238321.57 total 842.91 last 14.19) - open/close 16170 (time 1781238332.50 total 853.83 last 7.87) - open/close 16285 (time 1781238343.22 total 864.56 last 10.72) - open/close 16440 (time 1781238353.33 total 874.67 last 15.33) - open/close 16629 (time 1781238364.41 total 885.75 last 17.06) - open/close 16869 (time 1781238374.53 total 895.86 last 23.73) - open/close 17147 (time 1781238384.53 total 905.86 last 27.80) - open/close 17398 (time 1781238395.00 total 916.33 last 23.97) - open/close 17700 (time 1781238407.92 total 929.26 last 23.37) - open/close 17936 (time 1781238418.42 total 939.76 last 22.48) - open/close 18192 (time 1781238428.43 total 949.77 last 25.57) - open/close 18408 (time 1781238438.44 total 959.77 last 21.59) - open/close 19002 (time 1781238448.45 total 969.79 last 59.32) - open/close 19749 (time 1781238458.45 total 979.79 last 74.70) - open/close 20000 (time 1781238461.73 total 983.07 last 76.44) - open/close 20687 (time 1781238471.74 total 993.08 last 68.64) - open/close 21331 (time 1781238481.75 total 1003.08 last 64.36) - open/close 21931 (time 1781238491.76 total 1013.10 last 59.93) - open/close 22499 (time 1781238501.77 total 1023.11 last 56.72) - open/close 23103 (time 1781238511.80 total 1033.14 last 60.24) - open/close 23733 (time 1781238521.81 total 1043.14 last 62.95) - open/close 24393 (time 1781238531.82 total 1053.16 last 65.91) - open/close 25116 (time 1781238541.83 total 1063.16 last 72.27) - open/close 25908 (time 1781238551.84 total 1073.17 last 79.10) - open/close 26638 (time 1781238561.84 total 1083.18 last 72.98) - open/close 27449 (time 1781238571.85 total 1093.18 last 81.05) - open/close 28184 (time 1781238581.85 total 1103.19 last 73.44) - open/close 28874 (time 1781238591.87 total 1113.21 last 68.90) - open/close 29611 (time 1781238601.87 total 1123.21 last 73.70) - open/close 30000 (time 1781238607.12 total 1128.46 last 74.09) - open/close 30727 (time 1781238617.13 total 1138.47 last 72.60) - open/close 31484 (time 1781238627.14 total 1148.47 last 75.67) - open/close 32220 (time 1781238637.16 total 1158.50 last 73.42) - open/close 32962 (time 1781238647.17 total 1168.51 last 74.13) - open/close 33720 (time 1781238657.18 total 1178.52 last 75.74) - open/close 34476 (time 1781238667.20 total 1188.53 last 75.47) - open/close 35248 (time 1781238677.20 total 1198.54 last 77.13) - open/close 36014 (time 1781238687.21 total 1208.54 last 76.58) - open/close 36738 (time 1781238697.22 total 1218.56 last 72.29) - open/close 37498 (time 1781238707.25 total 1228.59 last 75.80) - open/close 38252 (time 1781238717.25 total 1238.59 last 75.37) - open/close 38961 (time 1781238727.28 total 1248.62 last 70.70) - open/close 39697 (time 1781238737.31 total 1258.64 last 73.41) - open/close 40000 (time 1781238741.18 total 1262.52 last 78.20) - open/close 40711 (time 1781238751.19 total 1272.53 last 71.02) - open/close 41361 (time 1781238761.21 total 1282.55 last 64.86) - open/close 42057 (time 1781238771.22 total 1292.56 last 69.57) - open/close 42767 (time 1781238781.23 total 1302.57 last 70.91) - open/close 43423 (time 1781238791.26 total 1312.59 last 65.43) - open/close 44162 (time 1781238801.27 total 1322.61 last 73.81) - open/close 44925 (time 1781238811.27 total 1332.61 last 76.28) - open/close 45633 (time 1781238821.28 total 1342.61 last 70.75) - open/close 46333 (time 1781238831.29 total 1352.63 last 69.88) - open/close 47084 (time 1781238841.31 total 1362.64 last 75.02) - open/close 47813 (time 1781238851.31 total 1372.64 last 72.88) - open/close 48592 (time 1781238861.32 total 1382.65 last 77.84) - open/close 49311 (time 1781238871.33 total 1392.66 last 71.83) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No space left on device total: 49729 open/close in 1398.69 seconds: 35.55 ops/second - unlinked 0 (time 1781238882 ; total 0 ; last 0) - unlinked 10000 (time 1781239017 ; total 135 ; last 135) - unlinked 20000 (time 1781239158 ; total 276 ; last 141) - unlinked 30000 (time 1781239295 ; total 413 ; last 137) - unlinked 40000 (time 1781239432 ; total 550 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No such file or directory total: 49729 unlinks in 684 seconds: 72.703217 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2