== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:22:09 (1787390529) start mds service on oleg334-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg334-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg334-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 245 (time 1787390610.34 total 10.53 last 23.27) - open/close 493 (time 1787390620.55 total 20.75 last 24.28) - open/close 711 (time 1787390630.72 total 30.91 last 21.44) - open/close 994 (time 1787390640.73 total 40.92 last 28.29) - open/close 1208 (time 1787390650.97 total 51.16 last 20.89) - open/close 1354 (time 1787390661.65 total 61.84 last 13.67) - open/close 1651 (time 1787390672.27 total 72.46 last 27.97) - open/close 1880 (time 1787390682.27 total 82.46 last 22.90) - open/close 2116 (time 1787390693.10 total 93.29 last 21.80) - open/close 2214 (time 1787390706.45 total 106.64 last 7.34) - open/close 2435 (time 1787390716.64 total 116.83 last 21.70) - open/close 2717 (time 1787390726.64 total 126.83 last 28.20) - open/close 2945 (time 1787390736.65 total 136.84 last 22.77) - open/close 3195 (time 1787390746.67 total 146.86 last 24.95) - open/close 3420 (time 1787390757.40 total 157.59 last 20.97) - open/close 3640 (time 1787390767.41 total 167.60 last 21.99) - open/close 3935 (time 1787390777.41 total 177.60 last 29.47) - open/close 4157 (time 1787390788.03 total 188.22 last 20.91) - open/close 4346 (time 1787390798.92 total 199.11 last 17.36) - open/close 4606 (time 1787390809.87 total 210.06 last 23.74) - open/close 4854 (time 1787390820.49 total 220.68 last 23.35) - open/close 5152 (time 1787390830.50 total 230.69 last 29.79) - open/close 5375 (time 1787390840.75 total 240.94 last 21.76) - open/close 5621 (time 1787390850.76 total 250.95 last 24.55) - open/close 5828 (time 1787390860.87 total 261.06 last 20.48) - open/close 6098 (time 1787390871.82 total 272.01 last 24.66) - open/close 6328 (time 1787390881.83 total 282.02 last 22.98) - open/close 6619 (time 1787390892.78 total 292.97 last 26.58) - open/close 6882 (time 1787390903.30 total 303.49 last 24.99) - open/close 7146 (time 1787390914.12 total 314.31 last 24.40) - open/close 7417 (time 1787390924.16 total 324.35 last 27.00) - open/close 7613 (time 1787390936.08 total 336.27 last 16.44) - open/close 7829 (time 1787390946.08 total 346.27 last 21.59) - open/close 8051 (time 1787390956.24 total 356.43 last 21.86) - open/close 8326 (time 1787390966.24 total 366.43 last 27.49) - open/close 8542 (time 1787390977.04 total 377.23 last 20.01) - open/close 8782 (time 1787390987.27 total 387.46 last 23.46) - open/close 9029 (time 1787390997.28 total 397.47 last 24.68) - open/close 9243 (time 1787391007.69 total 407.88 last 20.55) - open/close 9465 (time 1787391018.00 total 418.19 last 21.53) - open/close 9742 (time 1787391028.02 total 428.21 last 27.66) - open/close 9955 (time 1787391038.10 total 438.29 last 21.12) - open/close 10165 (time 1787391048.68 total 448.87 last 19.86) - open/close 10414 (time 1787391059.40 total 459.59 last 23.23) - open/close 10653 (time 1787391070.41 total 470.60 last 21.71) - open/close 10796 (time 1787391080.91 total 481.10 last 13.62) - open/close 10894 (time 1787391091.80 total 491.99 last 9.00) - open/close 10956 (time 1787391102.80 total 502.99 last 5.63) - open/close 11124 (time 1787391112.83 total 513.02 last 16.76) - open/close 11296 (time 1787391123.46 total 523.65 last 16.18) - open/close 11561 (time 1787391134.09 total 534.28 last 24.93) - open/close 11796 (time 1787391144.26 total 544.45 last 23.09) - open/close 12024 (time 1787391154.45 total 554.64 last 22.37) - open/close 12243 (time 1787391164.58 total 564.77 last 21.62) - open/close 12512 (time 1787391174.59 total 574.78 last 26.88) - open/close 12746 (time 1787391184.91 total 585.10 last 22.67) - open/close 12975 (time 1787391195.02 total 595.21 last 22.66) - open/close 13236 (time 1787391205.57 total 605.76 last 24.73) - open/close 13444 (time 1787391215.60 total 615.79 last 20.74) - open/close 13605 (time 1787391225.61 total 625.80 last 16.09) - open/close 13747 (time 1787391235.63 total 635.82 last 14.17) - open/close 13863 (time 1787391246.16 total 646.35 last 11.02) - open/close 14054 (time 1787391256.33 total 656.52 last 18.78) - open/close 14359 (time 1787391267.10 total 667.29 last 28.31) - open/close 14614 (time 1787391277.11 total 677.30 last 25.49) - open/close 14848 (time 1787391288.08 total 688.27 last 21.32) - open/close 15132 (time 1787391298.08 total 698.27 last 28.39) - open/close 15354 (time 1787391308.12 total 708.31 last 22.12) - open/close 15683 (time 1787391318.13 total 718.32 last 32.87) - open/close 15921 (time 1787391329.10 total 729.29 last 21.69) - open/close 16162 (time 1787391339.12 total 739.31 last 24.06) - open/close 16409 (time 1787391349.13 total 749.32 last 24.66) - open/close 16641 (time 1787391359.13 total 759.32 last 23.20) - open/close 16898 (time 1787391369.14 total 769.33 last 25.69) - open/close 17126 (time 1787391379.92 total 780.11 last 21.14) - open/close 17360 (time 1787391389.93 total 790.12 last 23.39) - open/close 17683 (time 1787391400.60 total 800.79 last 30.25) - open/close 17900 (time 1787391410.77 total 810.96 last 21.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-18097) error: No space left on device total: 18097 open/close in 817.49 seconds: 22.14 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (75299) - No such process - unlinked 0 (time 1787391427 ; total 0 ; last 0) - unlinked 10000 (time 1787391546 ; total 119 ; last 119) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-18097) error: No such file or directory total: 18097 unlinks in 184 seconds: 98.353264 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local oleg334-server: modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2