== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:03:26 (1774890206) start mds service on oleg306-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg306-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 134 (time 1774890258.40 total 10.62 last 12.61) - open/close 275 (time 1774890269.35 total 21.58 last 12.87) - open/close 445 (time 1774890279.71 total 31.93 last 16.42) - open/close 653 (time 1774890292.06 total 44.29 last 16.84) - open/close 904 (time 1774890302.81 total 55.04 last 23.35) - open/close 1113 (time 1774890313.11 total 65.34 last 20.29) - open/close 1437 (time 1774890323.11 total 75.34 last 32.39) - open/close 1681 (time 1774890333.53 total 85.76 last 23.41) - open/close 1900 (time 1774890343.96 total 96.19 last 21.00) - open/close 2163 (time 1774890354.91 total 107.14 last 24.03) - open/close 2329 (time 1774890365.08 total 117.31 last 16.32) - open/close 2485 (time 1774890375.97 total 128.20 last 14.33) - open/close 2599 (time 1774890386.71 total 138.94 last 10.61) - open/close 2818 (time 1774890396.76 total 148.99 last 21.80) - open/close 2957 (time 1774890406.77 total 159.00 last 13.89) - open/close 3209 (time 1774890416.78 total 169.01 last 25.18) - open/close 3275 (time 1774890426.90 total 179.13 last 6.52) - open/close 3364 (time 1774890437.15 total 189.38 last 8.68) - open/close 3626 (time 1774890447.16 total 199.39 last 26.18) - open/close 3894 (time 1774890457.62 total 209.85 last 25.63) - open/close 4223 (time 1774890468.63 total 220.86 last 29.87) - open/close 4385 (time 1774890479.58 total 231.81 last 14.80) - open/close 4594 (time 1774890490.13 total 242.36 last 19.81) - open/close 4903 (time 1774890500.37 total 252.60 last 30.18) - open/close 5232 (time 1774890510.68 total 262.91 last 31.92) - open/close 5427 (time 1774890520.87 total 273.10 last 19.13) - open/close 5763 (time 1774890531.03 total 283.26 last 33.06) - open/close 6024 (time 1774890541.14 total 293.37 last 25.81) - open/close 6347 (time 1774890551.96 total 304.19 last 29.87) - open/close 6668 (time 1774890562.02 total 314.25 last 31.91) - open/close 6989 (time 1774890573.01 total 325.24 last 29.19) - open/close 7303 (time 1774890583.02 total 335.25 last 31.39) - open/close 7611 (time 1774890593.68 total 345.91 last 28.87) - open/close 7880 (time 1774890604.06 total 356.29 last 25.92) - open/close 8254 (time 1774890614.43 total 366.66 last 36.08) - open/close 8551 (time 1774890624.44 total 376.67 last 29.67) - open/close 8870 (time 1774890634.58 total 386.81 last 31.45) - open/close 9175 (time 1774890645.27 total 397.50 last 28.53) - open/close 9546 (time 1774890655.71 total 407.93 last 35.55) - open/close 9875 (time 1774890665.71 total 417.94 last 32.90) - open/close 9991 (time 1774890676.19 total 428.42 last 11.06) - open/close 10323 (time 1774890686.62 total 438.85 last 31.83) - open/close 10577 (time 1774890697.44 total 449.67 last 23.49) - open/close 10928 (time 1774890707.80 total 460.03 last 33.87) - open/close 11273 (time 1774890717.81 total 470.04 last 34.47) - open/close 11617 (time 1774890727.81 total 480.04 last 34.39) - open/close 11917 (time 1774890738.27 total 490.50 last 28.69) - open/close 12450 (time 1774890748.27 total 500.50 last 53.27) - open/close 13769 (time 1774890758.28 total 510.51 last 131.85) - open/close 15259 (time 1774890768.28 total 520.51 last 148.93) - open/close 16817 (time 1774890778.28 total 530.51 last 155.78) - open/close 18481 (time 1774890788.29 total 540.51 last 166.35) - open/close 19756 (time 1774890798.29 total 550.52 last 127.48) - open/close 21344 (time 1774890808.29 total 560.52 last 158.79) - open/close 22884 (time 1774890818.29 total 570.52 last 153.94) - open/close 24626 (time 1774890828.29 total 580.52 last 174.17) - open/close 26124 (time 1774890838.30 total 590.53 last 149.69) - open/close 27826 (time 1774890848.30 total 600.53 last 170.15) - open/close 30000 (time 1774890857.21 total 609.43 last 244.23) - open/close 32528 (time 1774890867.21 total 619.44 last 252.60) - open/close 34931 (time 1774890877.22 total 629.44 last 240.27) - open/close 36884 (time 1774890887.22 total 639.44 last 195.28) - open/close 38643 (time 1774890897.22 total 649.45 last 175.85) - open/close 39959 (time 1774890907.22 total 659.45 last 131.52) - open/close 41561 (time 1774890917.23 total 669.46 last 160.15) - open/close 43063 (time 1774890927.23 total 679.46 last 150.19) - open/close 44632 (time 1774890937.23 total 689.46 last 156.89) - open/close 46434 (time 1774890947.23 total 699.46 last 180.17) - open/close 48111 (time 1774890957.23 total 709.46 last 167.65) - open/close 49544 (time 1774890967.24 total 719.46 last 143.27) total: 50000 open/close in 722.29 seconds: 69.22 ops/second - unlinked 0 (time 1774890971 ; total 0 ; last 0) - unlinked 10000 (time 1774891011 ; total 40 ; last 40) - unlinked 20000 (time 1774891043 ; total 72 ; last 32) - unlinked 30000 (time 1774891079 ; total 108 ; last 36) - unlinked 40000 (time 1774891106 ; total 135 ; last 27) total: 50000 unlinks in 163 seconds: 306.748474 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local oleg306-server: modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2