== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:00:06 (1788361206) start mds service on oleg324-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: Force libcfs to create 2 CPU partitions oleg324-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg324-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg324-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg324-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 169 (time 1788361275.49 total 10.81 last 15.63) - open/close 291 (time 1788361286.43 total 21.76 last 11.15) - open/close 424 (time 1788361296.61 total 31.93 last 13.07) - open/close 604 (time 1788361307.48 total 42.81 last 16.55) - open/close 733 (time 1788361317.60 total 52.93 last 12.75) - open/close 857 (time 1788361328.61 total 63.94 last 11.26) - open/close 976 (time 1788361339.49 total 74.82 last 10.94) - open/close 1102 (time 1788361349.54 total 84.87 last 12.54) - open/close 1268 (time 1788361360.17 total 95.50 last 15.61) - open/close 1389 (time 1788361371.04 total 106.37 last 11.13) - open/close 1531 (time 1788361382.11 total 117.44 last 12.83) - open/close 1576 (time 1788361394.02 total 129.34 last 3.78) - open/close 1676 (time 1788361404.64 total 139.96 last 9.42) - open/close 1812 (time 1788361415.58 total 150.91 last 12.43) - open/close 1907 (time 1788361426.08 total 161.41 last 9.05) - open/close 1992 (time 1788361436.45 total 171.77 last 8.20) - open/close 2176 (time 1788361447.02 total 182.35 last 17.40) - open/close 2344 (time 1788361457.19 total 192.51 last 16.52) - open/close 2439 (time 1788361467.62 total 202.94 last 9.11) - open/close 2592 (time 1788361478.51 total 213.84 last 14.04) - open/close 2696 (time 1788361489.31 total 224.63 last 9.63) - open/close 2840 (time 1788361499.32 total 234.64 last 14.39) - open/close 2930 (time 1788361509.92 total 245.24 last 8.49) - open/close 3076 (time 1788361519.96 total 255.29 last 14.53) - open/close 3248 (time 1788361530.41 total 265.74 last 16.46) - open/close 3411 (time 1788361540.41 total 275.74 last 16.29) - open/close 3508 (time 1788361551.97 total 287.30 last 8.39) - open/close 3627 (time 1788361562.65 total 297.98 last 11.14) - open/close 3749 (time 1788361573.53 total 308.86 last 11.22) - open/close 3953 (time 1788361585.64 total 320.96 last 16.85) - open/close 4126 (time 1788361595.64 total 330.97 last 17.29) - open/close 4273 (time 1788361606.51 total 341.84 last 13.53) - open/close 4469 (time 1788361617.70 total 353.02 last 17.52) - open/close 4714 (time 1788361628.19 total 363.52 last 23.35) - open/close 4983 (time 1788361638.81 total 374.13 last 25.33) - open/close 5292 (time 1788361649.57 total 384.90 last 28.70) - open/close 5631 (time 1788361660.64 total 395.96 last 30.64) - open/close 5922 (time 1788361670.64 total 405.97 last 29.09) - open/close 6179 (time 1788361680.86 total 416.19 last 25.14) - open/close 6417 (time 1788361691.80 total 427.13 last 21.76) - open/close 6701 (time 1788361702.69 total 438.02 last 26.07) - open/close 7012 (time 1788361713.69 total 449.02 last 28.27) - open/close 7284 (time 1788361724.06 total 459.38 last 26.25) - open/close 7582 (time 1788361734.56 total 469.89 last 28.37) - open/close 7894 (time 1788361745.38 total 480.70 last 28.84) - open/close 8190 (time 1788361756.19 total 491.52 last 27.38) - open/close 8451 (time 1788361766.43 total 501.76 last 25.48) - open/close 8677 (time 1788361776.44 total 511.76 last 22.59) - open/close 8948 (time 1788361786.97 total 522.30 last 25.72) - open/close 9969 (time 1788361796.98 total 532.31 last 102.05) - open/close 11560 (time 1788361806.99 total 542.31 last 158.99) - open/close 12863 (time 1788361816.99 total 552.31 last 130.28) - open/close 14081 (time 1788361826.99 total 562.32 last 121.73) - open/close 15346 (time 1788361837.00 total 572.33 last 126.39) - open/close 16502 (time 1788361847.00 total 582.33 last 115.59) - open/close 17698 (time 1788361857.00 total 592.33 last 119.59) - open/close 19091 (time 1788361867.01 total 602.33 last 139.25) - open/close 20000 (time 1788361873.26 total 608.59 last 145.29) - open/close 21732 (time 1788361883.27 total 618.59 last 173.17) - open/close 23337 (time 1788361893.27 total 628.59 last 160.47) - open/close 24678 (time 1788361903.28 total 638.60 last 133.99) - open/close 26067 (time 1788361913.28 total 648.60 last 138.90) - open/close 27519 (time 1788361923.28 total 658.61 last 145.15) - open/close 28991 (time 1788361933.29 total 668.61 last 147.12) - open/close 30000 (time 1788361940.02 total 675.35 last 149.73) - open/close 31459 (time 1788361950.04 total 685.37 last 145.66) - open/close 32994 (time 1788361960.04 total 695.37 last 153.46) - open/close 34452 (time 1788361970.05 total 705.38 last 145.69) - open/close 35599 (time 1788361980.06 total 715.38 last 114.63) - open/close 36912 (time 1788361990.06 total 725.39 last 131.26) - open/close 38523 (time 1788362000.07 total 735.39 last 160.94) - open/close 39859 (time 1788362010.07 total 745.40 last 133.57) - open/close 41120 (time 1788362020.07 total 755.40 last 126.08) - open/close 42356 (time 1788362030.07 total 765.40 last 123.59) - open/close 43562 (time 1788362040.08 total 775.40 last 120.54) - open/close 44846 (time 1788362050.08 total 785.41 last 128.38) - open/close 46188 (time 1788362060.09 total 795.41 last 134.10) - open/close 47556 (time 1788362070.09 total 805.42 last 136.78) - open/close 48951 (time 1788362080.09 total 815.42 last 139.48) total: 50000 open/close in 822.57 seconds: 60.78 ops/second - unlinked 0 (time 1788362089 ; total 0 ; last 0) - unlinked 10000 (time 1788362139 ; total 50 ; last 50) - unlinked 20000 (time 1788362177 ; total 88 ; last 38) - unlinked 30000 (time 1788362214 ; total 125 ; last 37) - unlinked 40000 (time 1788362255 ; total 166 ; last 41) total: 50000 unlinks in 208 seconds: 240.384613 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2