== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:23:23 (1788362603) start mds service on oleg327-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg327-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 254 (time 1788362646.00 total 10.01 last 25.37) - open/close 505 (time 1788362656.00 total 20.01 last 25.09) - open/close 783 (time 1788362666.60 total 30.61 last 26.23) - open/close 1116 (time 1788362676.84 total 40.85 last 32.52) - open/close 1374 (time 1788362687.46 total 51.48 last 24.28) - open/close 1572 (time 1788362697.47 total 61.48 last 19.79) - open/close 1791 (time 1788362707.47 total 71.49 last 21.89) - open/close 1957 (time 1788362717.49 total 81.51 last 16.57) - open/close 2208 (time 1788362727.50 total 91.52 last 25.07) - open/close 2363 (time 1788362738.55 total 102.56 last 14.03) - open/close 2563 (time 1788362748.56 total 112.58 last 19.97) - open/close 2783 (time 1788362759.48 total 123.49 last 20.15) - open/close 3067 (time 1788362769.67 total 133.69 last 27.86) - open/close 3380 (time 1788362780.20 total 144.22 last 29.73) - open/close 3559 (time 1788362790.89 total 154.90 last 16.75) - open/close 3744 (time 1788362801.38 total 165.40 last 17.63) - open/close 3895 (time 1788362811.38 total 175.40 last 15.10) - open/close 4005 (time 1788362822.00 total 186.01 last 10.36) - open/close 4198 (time 1788362832.30 total 196.31 last 18.74) - open/close 4343 (time 1788362842.34 total 206.36 last 14.43) - open/close 4442 (time 1788362852.65 total 216.66 last 9.61) - open/close 4638 (time 1788362863.60 total 227.61 last 17.90) - open/close 4812 (time 1788362874.09 total 238.11 last 16.58) - open/close 4980 (time 1788362884.71 total 248.72 last 15.83) - open/close 5116 (time 1788362895.02 total 259.03 last 13.20) - open/close 5305 (time 1788362905.78 total 269.79 last 17.56) - open/close 5506 (time 1788362915.83 total 279.84 last 20.01) - open/close 5711 (time 1788362925.84 total 289.85 last 20.47) - open/close 5790 (time 1788362936.42 total 300.44 last 7.46) - open/close 5976 (time 1788362946.86 total 310.88 last 17.81) - open/close 6082 (time 1788362957.54 total 321.56 last 9.92) - open/close 6234 (time 1788362967.55 total 331.56 last 15.20) - open/close 6341 (time 1788362978.47 total 342.48 last 9.80) - open/close 6448 (time 1788362989.17 total 353.19 last 10.00) - open/close 6634 (time 1788362999.46 total 363.48 last 18.07) - open/close 6709 (time 1788363009.65 total 373.66 last 7.36) - open/close 6849 (time 1788363020.58 total 384.60 last 12.81) - open/close 6967 (time 1788363033.26 total 397.27 last 9.31) - open/close 7139 (time 1788363043.56 total 407.57 last 16.70) - open/close 7440 (time 1788363054.51 total 418.52 last 27.50) - open/close 7782 (time 1788363065.38 total 429.39 last 31.46) - open/close 8129 (time 1788363075.56 total 439.57 last 34.09) - open/close 8294 (time 1788363085.56 total 449.58 last 16.49) - open/close 8555 (time 1788363096.30 total 460.31 last 24.31) - open/close 8679 (time 1788363107.05 total 471.07 last 11.53) - open/close 8793 (time 1788363119.59 total 483.61 last 9.09) - open/close 9039 (time 1788363130.61 total 494.62 last 22.33) - open/close 9209 (time 1788363141.54 total 505.56 last 15.55) - open/close 9388 (time 1788363153.66 total 517.67 last 14.77) - open/close 9546 (time 1788363163.66 total 527.68 last 15.80) - open/close 9763 (time 1788363173.93 total 537.94 last 21.13) - open/close 9803 (time 1788363184.67 total 548.69 last 3.72) - open/close 10000 (time 1788363190.84 total 554.85 last 31.97) - open/close 10176 (time 1788363200.84 total 564.85 last 17.59) - open/close 10486 (time 1788363210.84 total 574.86 last 30.99) - open/close 10806 (time 1788363220.85 total 584.86 last 31.98) - open/close 11988 (time 1788363230.86 total 594.87 last 118.12) - open/close 13091 (time 1788363240.86 total 604.88 last 110.25) - open/close 14366 (time 1788363250.87 total 614.88 last 127.42) - open/close 15572 (time 1788363260.87 total 624.89 last 120.54) - open/close 16846 (time 1788363270.88 total 634.89 last 127.33) - open/close 18079 (time 1788363280.89 total 644.90 last 123.20) - open/close 19396 (time 1788363290.89 total 654.90 last 131.64) - open/close 20000 (time 1788363295.59 total 659.60 last 128.53) - open/close 21228 (time 1788363305.59 total 669.61 last 122.77) - open/close 22339 (time 1788363315.60 total 679.61 last 111.06) - open/close 23438 (time 1788363325.60 total 689.61 last 109.86) - open/close 24602 (time 1788363335.60 total 699.61 last 116.39) - open/close 25924 (time 1788363345.61 total 709.62 last 132.12) - open/close 27200 (time 1788363355.61 total 719.62 last 127.57) - open/close 28548 (time 1788363365.61 total 729.63 last 134.74) - open/close 29820 (time 1788363375.61 total 739.63 last 127.19) - open/close 31289 (time 1788363385.62 total 749.63 last 146.86) - open/close 32591 (time 1788363395.62 total 759.63 last 130.19) - open/close 33869 (time 1788363405.62 total 769.63 last 127.77) - open/close 35410 (time 1788363415.62 total 779.64 last 154.07) - open/close 36786 (time 1788363425.63 total 789.64 last 137.56) - open/close 38335 (time 1788363435.63 total 799.64 last 154.87) - open/close 39685 (time 1788363445.63 total 809.65 last 134.93) - open/close 40000 (time 1788363447.95 total 811.96 last 136.05) - open/close 41366 (time 1788363457.95 total 821.97 last 136.51) - open/close 42796 (time 1788363467.96 total 831.97 last 142.93) - open/close 43982 (time 1788363477.96 total 841.98 last 118.58) - open/close 45142 (time 1788363487.97 total 851.98 last 115.95) - open/close 46229 (time 1788363497.97 total 861.98 last 108.67) - open/close 47468 (time 1788363507.97 total 871.99 last 123.81) - open/close 48650 (time 1788363517.98 total 881.99 last 118.16) - open/close 49783 (time 1788363527.98 total 892.00 last 113.26) total: 50000 open/close in 894.17 seconds: 55.92 ops/second - unlinked 0 (time 1788363532 ; total 0 ; last 0) - unlinked 10000 (time 1788363596 ; total 64 ; last 64) - unlinked 20000 (time 1788363637 ; total 105 ; last 41) - unlinked 30000 (time 1788363674 ; total 142 ; last 37) - unlinked 40000 (time 1788363707 ; total 175 ; last 33) total: 50000 unlinks in 208 seconds: 240.384613 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2