== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:28:05 (1786465685) start mds service on oleg302-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg302-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg302-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 277 (time 1786465738.53 total 10.00 last 27.70) - open/close 593 (time 1786465749.17 total 20.64 last 29.71) - open/close 973 (time 1786465759.17 total 30.64 last 37.98) - open/close 1299 (time 1786465769.77 total 41.23 last 30.78) - open/close 1644 (time 1786465780.34 total 51.80 last 32.64) - open/close 1875 (time 1786465790.35 total 61.82 last 23.06) - open/close 2099 (time 1786465800.42 total 71.89 last 22.24) - open/close 2320 (time 1786465810.50 total 81.96 last 21.94) - open/close 2540 (time 1786465820.54 total 92.01 last 21.91) - open/close 2618 (time 1786465832.11 total 103.58 last 6.74) - open/close 2818 (time 1786465842.23 total 113.70 last 19.77) - open/close 3022 (time 1786465852.23 total 123.70 last 20.40) - open/close 3229 (time 1786465862.53 total 134.00 last 20.10) - open/close 3458 (time 1786465873.26 total 144.73 last 21.34) - open/close 3624 (time 1786465883.38 total 154.85 last 16.39) - open/close 3783 (time 1786465893.62 total 165.09 last 15.53) - open/close 3886 (time 1786465904.11 total 175.58 last 9.82) - open/close 3977 (time 1786465914.43 total 185.90 last 8.82) - open/close 4145 (time 1786465925.24 total 196.71 last 15.55) - open/close 4360 (time 1786465935.24 total 206.71 last 21.49) - open/close 4497 (time 1786465945.47 total 216.94 last 13.39) - open/close 4648 (time 1786465955.82 total 227.29 last 14.59) - open/close 4803 (time 1786465969.07 total 240.54 last 11.70) - open/close 4943 (time 1786465979.07 total 250.54 last 14.00) - open/close 5115 (time 1786465989.63 total 261.10 last 16.29) - open/close 5336 (time 1786466000.43 total 271.90 last 20.46) - open/close 5501 (time 1786466010.80 total 282.26 last 15.92) - open/close 5641 (time 1786466021.80 total 293.27 last 12.72) - open/close 5799 (time 1786466031.81 total 303.28 last 15.79) - open/close 5928 (time 1786466042.29 total 313.76 last 12.31) - open/close 6060 (time 1786466052.30 total 323.77 last 13.18) - open/close 6230 (time 1786466062.69 total 334.16 last 16.36) - open/close 6391 (time 1786466072.88 total 344.35 last 15.80) - open/close 6632 (time 1786466083.54 total 355.01 last 22.61) - open/close 6737 (time 1786466094.25 total 365.72 last 9.80) - open/close 6869 (time 1786466104.28 total 375.74 last 13.17) - open/close 7067 (time 1786466114.88 total 386.35 last 18.66) - open/close 7169 (time 1786466125.44 total 396.91 last 9.66) - open/close 7326 (time 1786466135.44 total 406.91 last 15.69) - open/close 7460 (time 1786466145.45 total 416.92 last 13.39) - open/close 7672 (time 1786466156.33 total 427.80 last 19.49) - open/close 7786 (time 1786466166.34 total 437.80 last 11.39) - open/close 8021 (time 1786466176.50 total 447.96 last 23.13) - open/close 8089 (time 1786466186.67 total 458.13 last 6.69) - open/close 8151 (time 1786466196.67 total 468.14 last 6.20) - open/close 8310 (time 1786466206.67 total 478.14 last 15.90) - open/close 8474 (time 1786466216.68 total 488.15 last 16.38) - open/close 8769 (time 1786466227.44 total 498.91 last 27.42) - open/close 9109 (time 1786466237.50 total 508.97 last 33.78) - open/close 9264 (time 1786466248.37 total 519.84 last 14.26) - open/close 9422 (time 1786466258.61 total 530.08 last 15.44) - open/close 9599 (time 1786466268.61 total 540.08 last 17.70) - open/close 9843 (time 1786466278.61 total 550.08 last 24.39) - open/close 10000 (time 1786466284.47 total 555.93 last 26.82) - open/close 10200 (time 1786466294.48 total 565.95 last 19.97) - open/close 10272 (time 1786466304.63 total 576.10 last 7.09) - open/close 10480 (time 1786466315.18 total 586.65 last 19.72) - open/close 11487 (time 1786466325.18 total 596.65 last 100.68) - open/close 12865 (time 1786466335.19 total 606.65 last 137.75) - open/close 14063 (time 1786466345.19 total 616.65 last 119.80) - open/close 15214 (time 1786466355.21 total 626.68 last 114.80) - open/close 16378 (time 1786466365.22 total 636.69 last 116.29) - open/close 17716 (time 1786466375.22 total 646.69 last 133.79) - open/close 19172 (time 1786466385.22 total 656.69 last 145.58) - open/close 20000 (time 1786466391.37 total 662.84 last 134.62) - open/close 21342 (time 1786466401.38 total 672.85 last 134.14) - open/close 22768 (time 1786466411.38 total 682.85 last 142.52) - open/close 24097 (time 1786466421.39 total 692.86 last 132.86) - open/close 25377 (time 1786466431.40 total 702.87 last 127.84) - open/close 26636 (time 1786466441.41 total 712.87 last 125.83) - open/close 27791 (time 1786466451.41 total 722.88 last 115.40) - open/close 28906 (time 1786466461.42 total 732.89 last 111.41) - open/close 30000 (time 1786466470.54 total 742.01 last 119.98) - open/close 31111 (time 1786466480.54 total 752.01 last 111.07) - open/close 32229 (time 1786466490.55 total 762.01 last 111.77) - open/close 33383 (time 1786466500.56 total 772.02 last 115.31) - open/close 34582 (time 1786466510.56 total 782.02 last 119.88) - open/close 35753 (time 1786466520.56 total 792.03 last 117.02) - open/close 36895 (time 1786466530.57 total 802.04 last 114.12) - open/close 38181 (time 1786466540.58 total 812.04 last 128.52) - open/close 39465 (time 1786466550.58 total 822.05 last 128.30) - open/close 40000 (time 1786466554.65 total 826.12 last 131.55) - open/close 41249 (time 1786466564.66 total 836.13 last 124.81) - open/close 42619 (time 1786466574.66 total 846.13 last 136.99) - open/close 43902 (time 1786466584.67 total 856.14 last 128.18) - open/close 45211 (time 1786466594.67 total 866.14 last 130.82) - open/close 46621 (time 1786466604.68 total 876.15 last 140.93) - open/close 48170 (time 1786466614.68 total 886.15 last 154.84) - open/close 49762 (time 1786466624.69 total 896.15 last 159.15) total: 50000 open/close in 897.65 seconds: 55.70 ops/second - unlinked 0 (time 1786466628 ; total 0 ; last 0) - unlinked 10000 (time 1786466696 ; total 68 ; last 68) - unlinked 20000 (time 1786466736 ; total 108 ; last 40) - unlinked 30000 (time 1786466769 ; total 141 ; last 33) - unlinked 40000 (time 1786466812 ; total 184 ; last 43) total: 50000 unlinks in 223 seconds: 224.215240 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: modules unloaded. pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2