== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:18:45 (1781219925) start mds service on oleg658-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing load_modules_local oleg658-server: Loading modules from /home/green/git/lustre-release/lustre oleg658-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg658-server: detected 4 online CPUs by sysfs oleg658-server: MODOPTS_LIBCFS= oleg658-server: Force libcfs to create 2 CPU partitions oleg658-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg658-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg658-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg658-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-OST0000 oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg658-client.virtnet: -o user_xattr,flock 192.168.206.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 301 (time 1781219967.37 total 10.66 last 28.24) - open/close 535 (time 1781219978.31 total 21.60 last 21.39) - open/close 727 (time 1781219988.94 total 32.23 last 18.06) - open/close 968 (time 1781219998.94 total 42.23 last 24.09) - open/close 1204 (time 1781220009.03 total 52.32 last 23.40) - open/close 1410 (time 1781220019.91 total 63.20 last 18.93) - open/close 1644 (time 1781220029.91 total 73.20 last 23.39) - open/close 1829 (time 1781220040.51 total 83.81 last 17.45) - open/close 2074 (time 1781220051.52 total 94.81 last 22.26) - open/close 2384 (time 1781220062.02 total 105.32 last 29.52) - open/close 2703 (time 1781220072.03 total 115.32 last 31.89) - open/close 2942 (time 1781220082.03 total 125.32 last 23.89) - open/close 3181 (time 1781220092.16 total 135.46 last 23.58) - open/close 3522 (time 1781220103.18 total 146.47 last 30.96) - open/close 3718 (time 1781220113.86 total 157.15 last 18.35) - open/close 3931 (time 1781220124.62 total 167.91 last 19.80) - open/close 4169 (time 1781220134.62 total 177.91 last 23.79) - open/close 4517 (time 1781220145.61 total 188.90 last 31.68) - open/close 4759 (time 1781220155.84 total 199.13 last 23.65) - open/close 4984 (time 1781220166.47 total 209.76 last 21.18) - open/close 5161 (time 1781220176.96 total 220.25 last 16.86) - open/close 5408 (time 1781220186.97 total 230.26 last 24.69) - open/close 5637 (time 1781220197.95 total 241.25 last 20.84) - open/close 5919 (time 1781220208.20 total 251.49 last 27.52) - open/close 6251 (time 1781220218.21 total 261.50 last 33.18) - open/close 6513 (time 1781220228.56 total 271.85 last 25.30) - open/close 6901 (time 1781220239.43 total 282.72 last 35.70) - open/close 7139 (time 1781220249.43 total 292.72 last 23.80) - open/close 7395 (time 1781220260.29 total 303.58 last 23.57) - open/close 7544 (time 1781220270.60 total 313.89 last 14.46) - open/close 7775 (time 1781220281.60 total 324.89 last 20.99) - open/close 8003 (time 1781220292.36 total 335.65 last 21.20) - open/close 8175 (time 1781220302.91 total 346.20 last 16.30) - open/close 8398 (time 1781220313.54 total 356.83 last 20.98) - open/close 8558 (time 1781220323.97 total 367.27 last 15.33) - open/close 8771 (time 1781220334.79 total 378.08 last 19.69) - open/close 9046 (time 1781220345.28 total 388.58 last 26.21) - open/close 9356 (time 1781220355.29 total 398.58 last 31.00) - open/close 9638 (time 1781220365.58 total 408.87 last 27.39) - open/close 10000 (time 1781220372.31 total 415.60 last 53.79) - open/close 11391 (time 1781220382.32 total 425.61 last 138.99) - open/close 12754 (time 1781220392.33 total 435.62 last 136.22) - open/close 14436 (time 1781220402.33 total 445.62 last 168.12) - open/close 15974 (time 1781220412.34 total 455.63 last 153.74) - open/close 17692 (time 1781220422.34 total 465.63 last 171.75) - open/close 19355 (time 1781220432.34 total 475.63 last 166.22) - open/close 20000 (time 1781220437.84 total 481.13 last 117.37) - open/close 21404 (time 1781220447.84 total 491.14 last 140.32) - open/close 22694 (time 1781220457.85 total 501.14 last 128.95) - open/close 24030 (time 1781220467.85 total 511.14 last 133.54) - open/close 25337 (time 1781220477.85 total 521.15 last 130.67) - open/close 26732 (time 1781220487.86 total 531.15 last 139.47) - open/close 27951 (time 1781220497.86 total 541.15 last 121.89) - open/close 29105 (time 1781220507.86 total 551.15 last 115.37) - open/close 30000 (time 1781220513.90 total 557.19 last 148.15) - open/close 31587 (time 1781220523.91 total 567.20 last 158.59) - open/close 32804 (time 1781220533.91 total 577.21 last 121.63) - open/close 33998 (time 1781220543.91 total 587.21 last 119.39) - open/close 35176 (time 1781220553.92 total 597.21 last 117.78) - open/close 36266 (time 1781220563.93 total 607.22 last 108.88) - open/close 37473 (time 1781220573.93 total 617.22 last 120.69) - open/close 38747 (time 1781220583.93 total 627.22 last 127.35) - open/close 40000 (time 1781220593.34 total 636.63 last 133.21) - open/close 41240 (time 1781220603.34 total 646.63 last 123.95) - open/close 42642 (time 1781220613.34 total 656.64 last 140.16) - open/close 44101 (time 1781220623.35 total 666.64 last 145.85) - open/close 45777 (time 1781220633.35 total 676.65 last 167.49) - open/close 47214 (time 1781220643.36 total 686.65 last 143.68) - open/close 48473 (time 1781220653.36 total 696.66 last 125.80) - open/close 49452 (time 1781220663.38 total 706.67 last 97.79) total: 50000 open/close in 712.00 seconds: 70.23 ops/second - unlinked 0 (time 1781220671 ; total 0 ; last 0) - unlinked 10000 (time 1781220749 ; total 78 ; last 78) - unlinked 20000 (time 1781220802 ; total 131 ; last 53) - unlinked 30000 (time 1781220853 ; total 182 ; last 51) - unlinked 40000 (time 1781220901 ; total 230 ; last 48) total: 50000 unlinks in 276 seconds: 181.159424 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg658-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server unloading modules via unload_modules_local on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing unload_modules_local oleg658-server: modules unloaded. pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2