== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:10:01 (1776921001) start mds service on oleg411-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg411-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 753 (time 1776921041.99 total 10.45 last 72.06) - open/close 1512 (time 1776921052.48 total 20.95 last 72.31) - open/close 2101 (time 1776921063.50 total 31.96 last 53.47) - open/close 2275 (time 1776921073.50 total 41.97 last 17.39) - open/close 2390 (time 1776921084.09 total 52.55 last 10.86) - open/close 2572 (time 1776921094.48 total 62.94 last 17.52) - open/close 2694 (time 1776921104.49 total 72.95 last 12.18) - open/close 2802 (time 1776921115.09 total 83.55 last 10.19) - open/close 2933 (time 1776921125.09 total 93.55 last 13.10) - open/close 3142 (time 1776921135.51 total 103.97 last 20.07) - open/close 3333 (time 1776921145.51 total 113.97 last 19.10) - open/close 3864 (time 1776921156.36 total 124.82 last 48.95) - open/close 4120 (time 1776921167.37 total 135.84 last 23.23) - open/close 4327 (time 1776921179.98 total 148.44 last 16.42) - open/close 4391 (time 1776921192.08 total 160.54 last 5.29) - open/close 4442 (time 1776921202.13 total 170.59 last 5.08) - open/close 4596 (time 1776921212.14 total 180.60 last 15.39) - open/close 4732 (time 1776921222.14 total 190.60 last 13.59) - open/close 4877 (time 1776921232.66 total 201.13 last 13.78) - open/close 5117 (time 1776921242.82 total 211.28 last 23.63) - open/close 5261 (time 1776921253.70 total 222.17 last 13.23) - open/close 5506 (time 1776921264.20 total 232.66 last 23.35) - open/close 5752 (time 1776921275.01 total 243.47 last 22.75) - open/close 5910 (time 1776921286.03 total 254.49 last 14.34) - open/close 6022 (time 1776921296.91 total 265.37 last 10.29) - open/close 6181 (time 1776921306.91 total 275.37 last 15.90) - open/close 6301 (time 1776921317.85 total 286.31 last 10.97) - open/close 6394 (time 1776921328.53 total 296.99 last 8.71) - open/close 6530 (time 1776921338.55 total 307.01 last 13.58) - open/close 6667 (time 1776921348.76 total 317.22 last 13.41) - open/close 6765 (time 1776921359.31 total 327.77 last 9.29) - open/close 6999 (time 1776921369.32 total 337.78 last 23.39) - open/close 7174 (time 1776921380.18 total 348.64 last 16.12) - open/close 7321 (time 1776921390.49 total 358.95 last 14.25) - open/close 7512 (time 1776921401.93 total 370.39 last 16.70) - open/close 7633 (time 1776921412.62 total 381.08 last 11.32) - open/close 7762 (time 1776921422.62 total 391.08 last 12.90) - open/close 7870 (time 1776921433.18 total 401.64 last 10.23) - open/close 7956 (time 1776921443.66 total 412.12 last 8.21) - open/close 8088 (time 1776921454.67 total 423.13 last 11.99) - open/close 8251 (time 1776921465.17 total 433.63 last 15.51) - open/close 8369 (time 1776921476.17 total 444.63 last 10.73) - open/close 8523 (time 1776921486.35 total 454.81 last 15.13) - open/close 8635 (time 1776921496.98 total 465.44 last 10.53) - open/close 8781 (time 1776921507.27 total 475.73 last 14.19) - open/close 8911 (time 1776921518.10 total 486.56 last 12.01) - open/close 9050 (time 1776921528.20 total 496.66 last 13.75) - open/close 9184 (time 1776921538.32 total 506.78 last 13.24) - open/close 9317 (time 1776921548.34 total 516.80 last 13.27) - open/close 9499 (time 1776921559.25 total 527.71 last 16.69) - open/close 9616 (time 1776921570.06 total 538.52 last 10.82) - open/close 9757 (time 1776921580.06 total 548.53 last 14.09) - open/close 9941 (time 1776921590.09 total 558.55 last 18.35) - open/close 10251 (time 1776921600.10 total 568.56 last 30.99) - open/close 10549 (time 1776921610.10 total 578.56 last 29.78) - open/close 11812 (time 1776921620.10 total 588.57 last 126.27) - open/close 13027 (time 1776921630.10 total 598.57 last 121.49) - open/close 14298 (time 1776921640.11 total 608.57 last 127.08) - open/close 15666 (time 1776921650.11 total 618.57 last 136.71) - open/close 17098 (time 1776921660.12 total 628.58 last 143.12) - open/close 18752 (time 1776921670.12 total 638.58 last 165.37) - open/close 20000 (time 1776921679.73 total 648.20 last 129.79) - open/close 21347 (time 1776921689.74 total 658.20 last 134.67) - open/close 22592 (time 1776921699.74 total 668.20 last 124.46) - open/close 23955 (time 1776921709.75 total 678.21 last 136.18) - open/close 25461 (time 1776921719.75 total 688.21 last 150.55) - open/close 26996 (time 1776921729.76 total 698.22 last 153.41) - open/close 28590 (time 1776921739.76 total 708.23 last 159.33) - open/close 29904 (time 1776921749.77 total 718.23 last 131.35) - open/close 31322 (time 1776921759.78 total 728.24 last 141.67) - open/close 32716 (time 1776921769.78 total 738.24 last 139.30) - open/close 34099 (time 1776921779.78 total 748.25 last 138.27) - open/close 35508 (time 1776921789.79 total 758.26 last 140.78) - open/close 37141 (time 1776921799.80 total 768.26 last 163.23) - open/close 38774 (time 1776921809.80 total 778.26 last 163.30) - open/close 40000 (time 1776921818.25 total 786.71 last 145.03) - open/close 41603 (time 1776921828.26 total 796.72 last 160.22) - open/close 43026 (time 1776921838.26 total 806.72 last 142.26) - open/close 44618 (time 1776921848.26 total 816.72 last 159.20) - open/close 46142 (time 1776921858.26 total 826.72 last 152.35) - open/close 47808 (time 1776921868.26 total 836.73 last 166.55) - open/close 49410 (time 1776921878.27 total 846.73 last 160.15) total: 50000 open/close in 849.91 seconds: 58.83 ops/second - unlinked 0 (time 1776921883 ; total 0 ; last 0) - unlinked 10000 (time 1776921934 ; total 51 ; last 51) - unlinked 20000 (time 1776921969 ; total 86 ; last 35) - unlinked 30000 (time 1776922002 ; total 119 ; last 33) - unlinked 40000 (time 1776922028 ; total 145 ; last 26) total: 50000 unlinks in 176 seconds: 284.090912 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2