== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:46:07 (1788360367) start mds service on oleg426-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing load_modules_local oleg426-server: Loading modules from /home/green/git/lustre-release/lustre oleg426-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg426-server: detected 4 online CPUs by sysfs oleg426-server: MODOPTS_LIBCFS= oleg426-server: Force libcfs to create 2 CPU partitions oleg426-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg426-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg426-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 273 (time 1788360416.18 total 10.21 last 26.74) - open/close 571 (time 1788360426.88 total 20.91 last 27.85) - open/close 789 (time 1788360437.69 total 31.71 last 20.18) - open/close 1047 (time 1788360448.37 total 42.40 last 24.14) - open/close 1232 (time 1788360458.56 total 52.59 last 18.16) - open/close 1515 (time 1788360468.66 total 62.69 last 28.01) - open/close 2022 (time 1788360479.61 total 73.63 last 46.33) - open/close 2529 (time 1788360491.77 total 85.79 last 41.70) - open/close 2736 (time 1788360502.07 total 96.10 last 20.08) - open/close 3005 (time 1788360512.82 total 106.85 last 25.03) - open/close 3329 (time 1788360523.57 total 117.60 last 30.13) - open/close 3848 (time 1788360533.88 total 127.90 last 50.37) - open/close 4270 (time 1788360544.89 total 138.92 last 38.31) - open/close 4871 (time 1788360555.26 total 149.28 last 57.99) - open/close 5500 (time 1788360565.56 total 159.59 last 61.06) - open/close 6145 (time 1788360575.67 total 169.70 last 63.79) - open/close 6389 (time 1788360586.49 total 180.51 last 22.56) - open/close 6668 (time 1788360597.21 total 191.24 last 26.01) - open/close 6989 (time 1788360608.02 total 202.05 last 29.71) - open/close 7266 (time 1788360618.77 total 212.80 last 25.76) - open/close 7427 (time 1788360628.99 total 223.02 last 15.76) - open/close 7508 (time 1788360639.00 total 233.02 last 8.09) - open/close 7740 (time 1788360649.00 total 243.03 last 23.19) - open/close 7920 (time 1788360659.03 total 253.06 last 17.94) - open/close 8149 (time 1788360669.05 total 263.07 last 22.86) - open/close 8411 (time 1788360679.06 total 273.09 last 26.16) - open/close 8603 (time 1788360689.47 total 283.50 last 18.45) - open/close 8754 (time 1788360699.78 total 293.81 last 14.64) - open/close 8817 (time 1788360709.79 total 303.81 last 6.30) - open/close 8923 (time 1788360720.44 total 314.47 last 9.95) - open/close 9111 (time 1788360731.39 total 325.41 last 17.18) - open/close 9259 (time 1788360741.63 total 335.66 last 14.45) - open/close 9478 (time 1788360751.63 total 345.66 last 21.89) - open/close 9621 (time 1788360761.64 total 355.67 last 14.29) - open/close 9849 (time 1788360771.90 total 365.92 last 22.23) - open/close 9954 (time 1788360782.40 total 376.43 last 9.99) - open/close 10000 (time 1788360785.04 total 379.07 last 17.46) - open/close 10235 (time 1788360795.78 total 389.81 last 21.87) - open/close 10379 (time 1788360806.04 total 400.06 last 14.04) - open/close 10526 (time 1788360816.45 total 410.48 last 14.12) - open/close 10716 (time 1788360826.46 total 420.48 last 18.99) - open/close 10882 (time 1788360836.87 total 430.90 last 15.94) - open/close 11012 (time 1788360847.74 total 441.76 last 11.96) - open/close 11146 (time 1788360858.69 total 452.71 last 12.24) - open/close 11312 (time 1788360869.13 total 463.16 last 15.90) - open/close 11462 (time 1788360879.55 total 473.57 last 14.40) - open/close 11663 (time 1788360889.56 total 483.59 last 20.08) - open/close 11904 (time 1788360900.10 total 494.12 last 22.87) - open/close 12091 (time 1788360910.80 total 504.83 last 17.47) - open/close 12304 (time 1788360921.28 total 515.31 last 20.32) - open/close 12531 (time 1788360931.29 total 525.32 last 22.68) - open/close 13026 (time 1788360941.30 total 535.32 last 49.48) - open/close 13842 (time 1788360951.30 total 545.33 last 81.54) - open/close 14696 (time 1788360961.31 total 555.34 last 85.35) - open/close 15578 (time 1788360971.31 total 565.34 last 88.18) - open/close 16450 (time 1788360981.31 total 575.34 last 87.19) - open/close 17309 (time 1788360991.32 total 585.35 last 85.82) - open/close 18132 (time 1788361001.33 total 595.36 last 82.24) - open/close 19022 (time 1788361011.33 total 605.36 last 88.98) - open/close 19889 (time 1788361021.34 total 615.36 last 86.65) - open/close 20738 (time 1788361031.35 total 625.37 last 84.83) - open/close 21558 (time 1788361041.35 total 635.38 last 81.96) - open/close 22420 (time 1788361051.36 total 645.39 last 86.13) - open/close 23316 (time 1788361061.36 total 655.39 last 89.59) - open/close 24181 (time 1788361071.37 total 665.39 last 86.44) - open/close 25059 (time 1788361081.38 total 675.41 last 87.66) - open/close 25871 (time 1788361091.39 total 685.41 last 81.19) - open/close 26731 (time 1788361101.40 total 695.42 last 85.91) - open/close 27601 (time 1788361111.40 total 705.43 last 86.94) - open/close 28456 (time 1788361121.41 total 715.43 last 85.46) - open/close 29318 (time 1788361131.41 total 725.44 last 86.18) - open/close 30000 (time 1788361139.75 total 733.77 last 81.81) - open/close 30872 (time 1788361149.76 total 743.79 last 87.05) - open/close 31813 (time 1788361159.78 total 753.81 last 93.95) - open/close 32717 (time 1788361169.79 total 763.82 last 90.30) - open/close 33782 (time 1788361179.80 total 773.83 last 106.41) - open/close 34764 (time 1788361189.80 total 783.83 last 98.16) - open/close 35860 (time 1788361199.81 total 793.83 last 109.57) - open/close 37060 (time 1788361209.81 total 803.84 last 119.94) - open/close 38220 (time 1788361219.82 total 813.84 last 115.96) - open/close 39510 (time 1788361229.82 total 823.84 last 128.98) - open/close 40000 (time 1788361233.85 total 827.88 last 121.50) - open/close 41257 (time 1788361243.85 total 837.88 last 125.66) - open/close 42515 (time 1788361253.86 total 847.88 last 125.75) - open/close 43580 (time 1788361263.86 total 857.88 last 106.49) - open/close 44837 (time 1788361273.86 total 867.89 last 125.63) - open/close 46109 (time 1788361283.86 total 877.89 last 127.19) - open/close 47302 (time 1788361293.87 total 887.90 last 119.21) - open/close 48529 (time 1788361303.87 total 897.90 last 122.66) - open/close 49869 (time 1788361313.88 total 907.91 last 133.94) total: 50000 open/close in 908.77 seconds: 55.02 ops/second - unlinked 0 (time 1788361316 ; total 0 ; last 0) - unlinked 10000 (time 1788361394 ; total 78 ; last 78) - unlinked 20000 (time 1788361448 ; total 132 ; last 54) - unlinked 30000 (time 1788361496 ; total 180 ; last 48) - unlinked 40000 (time 1788361541 ; total 225 ; last 45) total: 50000 unlinks in 272 seconds: 183.823532 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg426-server unloading modules via unload_modules_local on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing unload_modules_local oleg426-server: modules unloaded. pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2