== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:30:26 (1782484226) start mds service on oleg408-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg408-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1782484295.18 total 10.73 last 13.23) - open/close 493 (time 1782484305.42 total 20.97 last 34.29) - open/close 749 (time 1782484315.72 total 31.27 last 24.87) - open/close 1066 (time 1782484326.72 total 42.28 last 28.80) - open/close 1382 (time 1782484336.97 total 52.53 last 30.83) - open/close 1742 (time 1782484347.02 total 62.57 last 35.84) - open/close 1973 (time 1782484357.03 total 72.58 last 23.08) - open/close 2121 (time 1782484367.03 total 82.58 last 14.80) - open/close 2235 (time 1782484378.09 total 93.65 last 10.30) - open/close 2469 (time 1782484388.10 total 103.66 last 23.38) - open/close 2641 (time 1782484398.11 total 113.66 last 17.19) - open/close 2707 (time 1782484408.14 total 123.69 last 6.58) - open/close 2874 (time 1782484418.45 total 134.00 last 16.20) - open/close 3106 (time 1782484428.45 total 144.01 last 23.19) - open/close 3234 (time 1782484439.08 total 154.63 last 12.05) - open/close 3468 (time 1782484449.10 total 164.65 last 23.36) - open/close 3648 (time 1782484459.11 total 174.67 last 17.96) - open/close 3889 (time 1782484469.12 total 184.67 last 24.08) - open/close 4121 (time 1782484479.76 total 195.31 last 21.81) - open/close 4338 (time 1782484490.26 total 205.81 last 20.66) - open/close 4559 (time 1782484500.27 total 215.83 last 22.07) - open/close 4827 (time 1782484510.28 total 225.83 last 26.78) - open/close 5009 (time 1782484520.34 total 235.89 last 18.10) - open/close 5200 (time 1782484530.99 total 246.54 last 17.94) - open/close 5453 (time 1782484541.47 total 257.02 last 24.14) - open/close 5609 (time 1782484551.48 total 267.03 last 15.58) - open/close 5856 (time 1782484562.53 total 278.08 last 22.36) - open/close 6093 (time 1782484572.55 total 288.10 last 23.66) - open/close 6333 (time 1782484583.06 total 298.61 last 22.83) - open/close 6548 (time 1782484593.22 total 308.78 last 21.15) - open/close 6762 (time 1782484603.23 total 318.78 last 21.38) - open/close 6979 (time 1782484613.86 total 329.41 last 20.42) - open/close 7200 (time 1782484624.28 total 339.83 last 21.20) - open/close 7419 (time 1782484634.95 total 350.50 last 20.52) - open/close 7589 (time 1782484645.77 total 361.32 last 15.72) - open/close 7789 (time 1782484655.89 total 371.44 last 19.75) - open/close 8010 (time 1782484666.32 total 381.87 last 21.20) - open/close 8232 (time 1782484677.14 total 392.69 last 20.52) - open/close 8477 (time 1782484687.25 total 402.80 last 24.23) - open/close 8733 (time 1782484697.27 total 412.82 last 25.54) - open/close 8989 (time 1782484708.00 total 423.55 last 23.86) - open/close 9194 (time 1782484718.98 total 434.54 last 18.66) - open/close 9297 (time 1782484729.48 total 445.03 last 9.81) - open/close 9481 (time 1782484739.49 total 455.04 last 18.40) - open/close 9695 (time 1782484749.96 total 465.51 last 20.43) - open/close 9975 (time 1782484760.13 total 475.69 last 27.52) - open/close 10266 (time 1782484772.93 total 488.48 last 22.74) - open/close 10414 (time 1782484782.98 total 498.54 last 14.72) - open/close 10629 (time 1782484793.16 total 508.71 last 21.13) - open/close 10883 (time 1782484803.73 total 519.28 last 24.03) - open/close 11081 (time 1782484814.09 total 529.64 last 19.12) - open/close 11241 (time 1782484824.15 total 539.70 last 15.91) - open/close 11470 (time 1782484834.37 total 549.93 last 22.39) - open/close 11768 (time 1782484844.67 total 560.23 last 28.93) - open/close 12007 (time 1782484855.69 total 571.24 last 21.70) - open/close 12285 (time 1782484866.05 total 581.61 last 26.82) - open/close 12623 (time 1782484878.30 total 593.85 last 27.61) - open/close 12937 (time 1782484888.30 total 603.85 last 31.38) - open/close 13134 (time 1782484898.43 total 613.98 last 19.45) - open/close 13369 (time 1782484908.63 total 624.18 last 23.04) - open/close 13598 (time 1782484919.25 total 634.81 last 21.55) - open/close 13856 (time 1782484930.22 total 645.77 last 23.54) - open/close 14050 (time 1782484940.22 total 655.78 last 19.39) - open/close 14187 (time 1782484950.22 total 665.78 last 13.70) - open/close 14300 (time 1782484960.23 total 675.78 last 11.30) - open/close 14443 (time 1782484970.23 total 685.78 last 14.29) - open/close 14645 (time 1782484980.82 total 696.37 last 19.07) - open/close 15561 (time 1782484990.83 total 706.38 last 91.55) - open/close 16458 (time 1782485000.85 total 716.40 last 89.51) - open/close 17386 (time 1782485010.86 total 726.41 last 92.74) - open/close 18437 (time 1782485020.86 total 736.41 last 105.05) - open/close 19375 (time 1782485030.87 total 746.42 last 93.72) - open/close 20000 (time 1782485037.44 total 752.99 last 95.09) - open/close 20970 (time 1782485047.45 total 763.00 last 96.97) - open/close 21903 (time 1782485057.45 total 773.00 last 93.27) - open/close 22910 (time 1782485067.45 total 783.00 last 100.70) - open/close 23940 (time 1782485077.45 total 793.01 last 102.95) - open/close 24936 (time 1782485087.47 total 803.02 last 99.47) - open/close 25872 (time 1782485097.47 total 813.02 last 93.58) - open/close 26893 (time 1782485107.48 total 823.03 last 102.03) - open/close 27867 (time 1782485117.48 total 833.03 last 97.33) - open/close 28835 (time 1782485127.49 total 843.04 last 96.75) - open/close 29796 (time 1782485137.49 total 853.04 last 96.09) - open/close 30000 (time 1782485139.51 total 855.06 last 101.06) - open/close 30973 (time 1782485149.51 total 865.06 last 97.26) - open/close 32007 (time 1782485159.52 total 875.07 last 103.31) - open/close 33106 (time 1782485169.52 total 885.07 last 109.89) - open/close 34214 (time 1782485179.53 total 895.08 last 110.77) - open/close 35341 (time 1782485189.53 total 905.08 last 112.63) - open/close 36360 (time 1782485199.55 total 915.10 last 101.69) - open/close 37349 (time 1782485209.55 total 925.11 last 98.89) - open/close 38355 (time 1782485219.56 total 935.11 last 100.58) - open/close 39247 (time 1782485229.56 total 945.11 last 89.15) - open/close 40000 (time 1782485237.76 total 953.31 last 91.87) - open/close 40883 (time 1782485247.77 total 963.32 last 88.24) - open/close 41845 (time 1782485257.77 total 973.32 last 96.12) - open/close 42929 (time 1782485267.77 total 983.33 last 108.38) - open/close 43984 (time 1782485277.78 total 993.33 last 105.48) - open/close 45097 (time 1782485287.79 total 1003.34 last 111.12) - open/close 46021 (time 1782485297.81 total 1013.36 last 92.28) - open/close 47038 (time 1782485307.81 total 1023.36 last 101.65) - open/close 48062 (time 1782485317.81 total 1033.37 last 102.36) - open/close 49075 (time 1782485327.82 total 1043.38 last 101.21) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1048.68 seconds: 47.33 ops/second - unlinked 0 (time 1782485335 ; total 0 ; last 0) - unlinked 10000 (time 1782485421 ; total 86 ; last 86) - unlinked 20000 (time 1782485471 ; total 136 ; last 50) - unlinked 30000 (time 1782485516 ; total 181 ; last 45) - unlinked 40000 (time 1782485559 ; total 224 ; last 43) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 268 seconds: 185.190292 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: modules unloaded. pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2