== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:07:30 (1786568850) start mds service on oleg437-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg437-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1786568920.32 total 10.82 last 22.46) - open/close 468 (time 1786568930.32 total 20.82 last 22.50) - open/close 690 (time 1786568941.10 total 31.60 last 20.60) - open/close 937 (time 1786568951.73 total 42.23 last 23.24) - open/close 1138 (time 1786568961.74 total 52.23 last 20.09) - open/close 1332 (time 1786568972.73 total 63.23 last 17.65) - open/close 1576 (time 1786568983.60 total 74.10 last 22.44) - open/close 1786 (time 1786568993.98 total 84.48 last 20.23) - open/close 1985 (time 1786569004.41 total 94.91 last 19.08) - open/close 2238 (time 1786569014.43 total 104.93 last 25.26) - open/close 2460 (time 1786569024.82 total 115.32 last 21.36) - open/close 2666 (time 1786569035.20 total 125.70 last 19.85) - open/close 2923 (time 1786569045.49 total 135.99 last 24.97) - open/close 3129 (time 1786569055.87 total 146.37 last 19.86) - open/close 3360 (time 1786569066.43 total 156.93 last 21.87) - open/close 3611 (time 1786569076.43 total 166.93 last 25.09) - open/close 3843 (time 1786569087.36 total 177.86 last 21.23) - open/close 4093 (time 1786569097.37 total 187.87 last 24.97) - open/close 4297 (time 1786569107.89 total 198.39 last 19.40) - open/close 4506 (time 1786569118.46 total 208.96 last 19.77) - open/close 4696 (time 1786569128.62 total 219.12 last 18.69) - open/close 4899 (time 1786569138.64 total 229.14 last 20.26) - open/close 5129 (time 1786569148.66 total 239.16 last 22.96) - open/close 5294 (time 1786569159.22 total 249.72 last 15.62) - open/close 5461 (time 1786569169.65 total 260.15 last 16.01) - open/close 5603 (time 1786569179.76 total 270.26 last 14.05) - open/close 5750 (time 1786569189.78 total 280.28 last 14.67) - open/close 5932 (time 1786569200.37 total 290.87 last 17.19) - open/close 6112 (time 1786569210.82 total 301.32 last 17.22) - open/close 6307 (time 1786569220.82 total 311.32 last 19.49) - open/close 6443 (time 1786569231.03 total 321.52 last 13.33) - open/close 6638 (time 1786569241.77 total 332.27 last 18.14) - open/close 6781 (time 1786569252.03 total 342.52 last 13.95) - open/close 7046 (time 1786569262.45 total 352.95 last 25.41) - open/close 7205 (time 1786569272.70 total 363.20 last 15.52) - open/close 7388 (time 1786569283.44 total 373.94 last 17.03) - open/close 7518 (time 1786569294.39 total 384.88 last 11.88) - open/close 7648 (time 1786569304.56 total 395.06 last 12.77) - open/close 7801 (time 1786569314.62 total 405.11 last 15.22) - open/close 7884 (time 1786569324.91 total 415.41 last 8.06) - open/close 8067 (time 1786569335.48 total 425.98 last 17.32) - open/close 8231 (time 1786569345.78 total 436.28 last 15.92) - open/close 8347 (time 1786569356.21 total 446.71 last 11.12) - open/close 8628 (time 1786569366.22 total 456.72 last 28.07) - open/close 8824 (time 1786569376.44 total 466.94 last 19.18) - open/close 8952 (time 1786569387.19 total 477.69 last 11.91) - open/close 9113 (time 1786569400.04 total 490.54 last 12.53) - open/close 9260 (time 1786569410.80 total 501.30 last 13.67) - open/close 9430 (time 1786569420.81 total 511.31 last 16.98) - open/close 9504 (time 1786569431.03 total 521.53 last 7.24) - open/close 9562 (time 1786569443.05 total 533.55 last 4.82) - open/close 9791 (time 1786569453.61 total 544.11 last 21.68) - open/close 10000 (time 1786569460.41 total 550.91 last 30.76) - open/close 10355 (time 1786569470.71 total 561.21 last 34.46) - open/close 10431 (time 1786569480.71 total 571.21 last 7.60) - open/close 10580 (time 1786569491.45 total 581.95 last 13.88) - open/close 10820 (time 1786569501.83 total 592.33 last 23.12) - open/close 11048 (time 1786569513.28 total 603.78 last 19.92) - open/close 11339 (time 1786569524.08 total 614.58 last 26.93) - open/close 11682 (time 1786569534.08 total 624.58 last 34.29) - open/close 12761 (time 1786569544.08 total 634.58 last 107.89) - open/close 14070 (time 1786569554.09 total 644.59 last 130.82) - open/close 15691 (time 1786569564.09 total 654.59 last 162.07) - open/close 17240 (time 1786569574.10 total 664.60 last 154.76) - open/close 19044 (time 1786569584.11 total 674.61 last 180.32) - open/close 20000 (time 1786569590.53 total 681.03 last 148.90) - open/close 21394 (time 1786569600.53 total 691.03 last 139.39) - open/close 22682 (time 1786569610.53 total 701.03 last 128.78) - open/close 24026 (time 1786569620.53 total 711.03 last 134.37) - open/close 25465 (time 1786569630.54 total 721.03 last 143.84) - open/close 26980 (time 1786569640.54 total 731.04 last 151.38) - open/close 28193 (time 1786569650.55 total 741.05 last 121.26) - open/close 29327 (time 1786569660.55 total 751.05 last 113.32) - open/close 30000 (time 1786569666.03 total 756.53 last 122.96) - open/close 31434 (time 1786569676.03 total 766.53 last 143.40) - open/close 32658 (time 1786569686.03 total 776.53 last 122.35) - open/close 33816 (time 1786569696.03 total 786.53 last 115.76) - open/close 34997 (time 1786569706.04 total 796.53 last 118.10) - open/close 36416 (time 1786569716.04 total 806.53 last 141.89) - open/close 37836 (time 1786569726.04 total 816.54 last 141.95) - open/close 39200 (time 1786569736.04 total 826.54 last 136.37) - open/close 40000 (time 1786569742.49 total 832.99 last 124.08) - open/close 41330 (time 1786569752.49 total 842.99 last 133.00) - open/close 42655 (time 1786569762.49 total 852.99 last 132.47) - open/close 43891 (time 1786569772.50 total 863.00 last 123.49) - open/close 44974 (time 1786569782.51 total 873.01 last 108.18) - open/close 46134 (time 1786569792.51 total 883.01 last 115.98) - open/close 47322 (time 1786569802.51 total 893.01 last 118.78) - open/close 48444 (time 1786569812.52 total 903.02 last 112.15) - open/close 49539 (time 1786569822.52 total 913.02 last 109.50) total: 50000 open/close in 917.44 seconds: 54.50 ops/second - unlinked 0 (time 1786569829 ; total 0 ; last 0) - unlinked 10000 (time 1786569918 ; total 89 ; last 89) - unlinked 20000 (time 1786569967 ; total 138 ; last 49) - unlinked 30000 (time 1786570006 ; total 177 ; last 39) - unlinked 40000 (time 1786570051 ; total 222 ; last 45) total: 50000 unlinks in 269 seconds: 185.873611 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local oleg437-server: modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2