== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:31:17 (1783351877) start mds service on oleg125-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg125-server: detected 4 online CPUs by sysfs oleg125-server: MODOPTS_LIBCFS= oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg125-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 329 (time 1783351928.74 total 13.10 last 25.12) - open/close 568 (time 1783351938.74 total 23.10 last 23.89) - open/close 827 (time 1783351948.74 total 33.10 last 25.90) - open/close 1166 (time 1783351959.65 total 44.02 last 31.06) - open/close 1500 (time 1783351969.66 total 54.02 last 33.38) - open/close 1761 (time 1783351979.81 total 64.17 last 25.72) - open/close 2004 (time 1783351989.92 total 74.28 last 24.04) - open/close 2277 (time 1783352000.02 total 84.39 last 27.00) - open/close 2614 (time 1783352010.85 total 95.21 last 31.12) - open/close 2800 (time 1783352021.66 total 106.02 last 17.21) - open/close 3052 (time 1783352032.28 total 116.64 last 23.73) - open/close 3335 (time 1783352043.23 total 127.59 last 25.85) - open/close 3596 (time 1783352053.24 total 137.60 last 26.08) - open/close 3671 (time 1783352063.33 total 147.69 last 7.43) - open/close 3805 (time 1783352074.27 total 158.64 last 12.24) - open/close 4009 (time 1783352084.90 total 169.27 last 19.19) - open/close 4219 (time 1783352095.46 total 179.82 last 19.89) - open/close 4412 (time 1783352107.55 total 191.92 last 15.96) - open/close 4735 (time 1783352117.86 total 202.22 last 31.35) - open/close 5009 (time 1783352128.87 total 213.23 last 24.89) - open/close 5259 (time 1783352139.30 total 223.66 last 23.96) - open/close 5341 (time 1783352149.34 total 233.70 last 8.17) - open/close 5526 (time 1783352159.34 total 243.71 last 18.49) - open/close 5837 (time 1783352169.50 total 253.87 last 30.61) - open/close 6070 (time 1783352180.32 total 264.69 last 21.54) - open/close 6351 (time 1783352191.34 total 275.70 last 25.50) - open/close 6735 (time 1783352202.21 total 286.57 last 35.35) - open/close 7009 (time 1783352212.21 total 296.57 last 27.40) - open/close 7333 (time 1783352222.42 total 306.79 last 31.71) - open/close 7625 (time 1783352233.11 total 317.47 last 27.32) - open/close 7851 (time 1783352243.30 total 327.67 last 22.17) - open/close 8071 (time 1783352253.34 total 337.70 last 21.92) - open/close 8442 (time 1783352264.16 total 348.52 last 34.31) - open/close 8655 (time 1783352274.46 total 358.82 last 20.67) - open/close 9014 (time 1783352284.83 total 369.19 last 34.64) - open/close 9361 (time 1783352294.83 total 379.19 last 34.68) - open/close 9694 (time 1783352304.83 total 389.20 last 33.29) - open/close 10000 (time 1783352314.24 total 398.60 last 32.53) - open/close 10236 (time 1783352324.37 total 408.74 last 23.28) - open/close 10418 (time 1783352334.82 total 419.18 last 17.42) - open/close 10639 (time 1783352345.43 total 429.80 last 20.82) - open/close 10866 (time 1783352356.25 total 440.61 last 20.98) - open/close 11041 (time 1783352366.62 total 450.98 last 16.88) - open/close 11343 (time 1783352376.80 total 461.16 last 29.66) - open/close 11738 (time 1783352386.80 total 471.16 last 39.50) - open/close 13442 (time 1783352396.81 total 481.17 last 170.30) - open/close 15041 (time 1783352406.81 total 491.18 last 159.82) - open/close 16713 (time 1783352416.81 total 501.18 last 167.18) - open/close 18393 (time 1783352426.82 total 511.18 last 167.94) - open/close 20000 (time 1783352436.51 total 520.87 last 165.86) - open/close 22580 (time 1783352446.51 total 530.87 last 258.00) - open/close 25038 (time 1783352456.51 total 540.88 last 245.63) - open/close 27860 (time 1783352466.51 total 550.88 last 282.19) - open/close 30000 (time 1783352476.00 total 560.36 last 225.55) - open/close 31977 (time 1783352486.01 total 570.37 last 197.64) - open/close 33812 (time 1783352496.01 total 580.37 last 183.42) - open/close 35741 (time 1783352506.01 total 590.38 last 192.81) - open/close 37547 (time 1783352516.02 total 600.38 last 180.53) - open/close 39402 (time 1783352526.02 total 610.38 last 185.47) - open/close 40000 (time 1783352528.89 total 613.25 last 208.68) - open/close 41717 (time 1783352538.89 total 623.25 last 171.68) - open/close 43330 (time 1783352548.89 total 633.25 last 161.29) - open/close 45109 (time 1783352558.90 total 643.26 last 177.74) - open/close 46940 (time 1783352568.90 total 653.26 last 183.08) - open/close 48912 (time 1783352578.90 total 663.27 last 197.08) total: 50000 open/close in 669.77 seconds: 74.65 ops/second - unlinked 0 (time 1783352587 ; total 0 ; last 0) - unlinked 10000 (time 1783352621 ; total 34 ; last 34) - unlinked 20000 (time 1783352645 ; total 58 ; last 24) - unlinked 30000 (time 1783352665 ; total 78 ; last 20) - unlinked 40000 (time 1783352689 ; total 102 ; last 24) total: 50000 unlinks in 133 seconds: 375.939850 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server unloading modules via unload_modules_local on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local oleg125-server: modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2