== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:51:03 (1784094663) start mds service on oleg403-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg403-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1784094730.53 total 10.02 last 23.96) - open/close 450 (time 1784094740.54 total 20.03 last 20.97) - open/close 681 (time 1784094750.54 total 30.03 last 23.10) - open/close 956 (time 1784094761.22 total 40.70 last 25.76) - open/close 1163 (time 1784094771.26 total 50.75 last 20.61) - open/close 1388 (time 1784094781.28 total 60.76 last 22.47) - open/close 1638 (time 1784094792.08 total 71.57 last 23.14) - open/close 1775 (time 1784094802.12 total 81.61 last 13.64) - open/close 1901 (time 1784094812.83 total 92.32 last 11.76) - open/close 2135 (time 1784094822.86 total 102.35 last 23.33) - open/close 2348 (time 1784094832.87 total 112.36 last 21.28) - open/close 2514 (time 1784094842.89 total 122.38 last 16.58) - open/close 2718 (time 1784094853.65 total 133.14 last 18.96) - open/close 2931 (time 1784094863.67 total 143.16 last 21.26) - open/close 3159 (time 1784094873.93 total 153.42 last 22.21) - open/close 3417 (time 1784094883.94 total 163.42 last 25.79) - open/close 3611 (time 1784094893.95 total 173.44 last 19.37) - open/close 3784 (time 1784094904.47 total 183.96 last 16.45) - open/close 3969 (time 1784094914.48 total 193.97 last 18.48) - open/close 4098 (time 1784094924.48 total 203.97 last 12.90) - open/close 4183 (time 1784094934.84 total 214.33 last 8.20) - open/close 4327 (time 1784094945.09 total 224.58 last 14.06) - open/close 4445 (time 1784094956.03 total 235.52 last 10.78) - open/close 4578 (time 1784094966.15 total 245.64 last 13.14) - open/close 4758 (time 1784094976.16 total 255.65 last 17.99) - open/close 4894 (time 1784094987.22 total 266.71 last 12.29) - open/close 5056 (time 1784094997.89 total 277.38 last 15.19) - open/close 5189 (time 1784095008.19 total 287.68 last 12.91) - open/close 5285 (time 1784095018.71 total 298.20 last 9.13) - open/close 5541 (time 1784095028.81 total 308.30 last 25.34) - open/close 5713 (time 1784095038.85 total 318.34 last 17.13) - open/close 5919 (time 1784095049.34 total 328.83 last 19.64) - open/close 6036 (time 1784095060.28 total 339.77 last 10.70) - open/close 6200 (time 1784095070.97 total 350.46 last 15.34) - open/close 6370 (time 1784095081.66 total 361.15 last 15.90) - open/close 6516 (time 1784095091.85 total 371.33 last 14.34) - open/close 6585 (time 1784095101.96 total 381.45 last 6.82) - open/close 6737 (time 1784095112.25 total 391.74 last 14.77) - open/close 6941 (time 1784095123.27 total 402.76 last 18.51) - open/close 7172 (time 1784095134.08 total 413.57 last 21.37) - open/close 7326 (time 1784095144.26 total 423.75 last 15.13) - open/close 7471 (time 1784095154.26 total 433.75 last 14.49) - open/close 7604 (time 1784095164.29 total 443.77 last 13.27) - open/close 7738 (time 1784095175.29 total 454.78 last 12.17) - open/close 7871 (time 1784095186.32 total 465.81 last 12.07) - open/close 7934 (time 1784095197.32 total 476.81 last 5.73) - open/close 8110 (time 1784095207.67 total 487.16 last 17.00) - open/close 8236 (time 1784095219.53 total 499.02 last 10.63) - open/close 8340 (time 1784095230.27 total 509.76 last 9.68) - open/close 8524 (time 1784095241.15 total 520.64 last 16.91) - open/close 8639 (time 1784095252.03 total 531.52 last 10.57) - open/close 8733 (time 1784095262.40 total 541.88 last 9.07) - open/close 8897 (time 1784095272.77 total 552.26 last 15.81) - open/close 9072 (time 1784095282.78 total 562.27 last 17.48) - open/close 9226 (time 1784095293.18 total 572.67 last 14.80) - open/close 9344 (time 1784095304.00 total 583.49 last 10.91) - open/close 9461 (time 1784095314.81 total 594.30 last 10.82) - open/close 9768 (time 1784095324.82 total 604.31 last 30.68) - open/close 9842 (time 1784095334.91 total 614.40 last 7.33) - open/close 9983 (time 1784095346.88 total 626.37 last 11.78) - open/close 10178 (time 1784095357.69 total 637.18 last 18.04) - open/close 10407 (time 1784095367.70 total 647.19 last 22.89) - open/close 10812 (time 1784095377.70 total 657.19 last 40.49) - open/close 11576 (time 1784095387.70 total 667.19 last 76.39) - open/close 12425 (time 1784095397.71 total 677.19 last 84.87) - open/close 13423 (time 1784095407.71 total 687.20 last 99.75) - open/close 14352 (time 1784095417.71 total 697.20 last 92.89) - open/close 15231 (time 1784095427.72 total 707.21 last 87.83) - open/close 16220 (time 1784095437.72 total 717.21 last 98.88) - open/close 17204 (time 1784095447.72 total 727.21 last 98.38) - open/close 18227 (time 1784095457.73 total 737.22 last 102.23) - open/close 19196 (time 1784095467.74 total 747.23 last 96.80) - open/close 20000 (time 1784095475.52 total 755.01 last 103.31) - open/close 20936 (time 1784095485.52 total 765.01 last 93.58) - open/close 21776 (time 1784095495.53 total 775.02 last 83.97) - open/close 22737 (time 1784095505.53 total 785.02 last 96.08) - open/close 23703 (time 1784095515.53 total 795.02 last 96.58) - open/close 24635 (time 1784095525.54 total 805.03 last 93.12) - open/close 25679 (time 1784095535.55 total 815.03 last 104.34) - open/close 26835 (time 1784095545.55 total 825.04 last 115.57) - open/close 27923 (time 1784095555.56 total 835.05 last 108.70) - open/close 28868 (time 1784095565.56 total 845.05 last 94.45) - open/close 29813 (time 1784095575.56 total 855.05 last 94.50) - open/close 30000 (time 1784095577.77 total 857.26 last 84.66) - open/close 30998 (time 1784095587.78 total 867.27 last 99.72) - open/close 31923 (time 1784095597.78 total 877.27 last 92.46) - open/close 32946 (time 1784095607.80 total 887.28 last 102.19) - open/close 33970 (time 1784095617.80 total 897.29 last 102.34) - open/close 35003 (time 1784095627.82 total 907.31 last 103.10) - open/close 36045 (time 1784095637.82 total 917.31 last 104.15) - open/close 37058 (time 1784095647.83 total 927.32 last 101.26) - open/close 38070 (time 1784095657.83 total 937.32 last 101.14) - open/close 39308 (time 1784095667.84 total 947.32 last 123.77) - open/close 40000 (time 1784095672.51 total 952.00 last 148.13) - open/close 41311 (time 1784095682.51 total 962.00 last 131.04) - open/close 42667 (time 1784095692.52 total 972.01 last 135.48) - open/close 43978 (time 1784095702.53 total 982.01 last 131.05) - open/close 45252 (time 1784095712.53 total 992.01 last 127.39) - open/close 46512 (time 1784095722.53 total 1002.02 last 125.95) - open/close 47703 (time 1784095732.54 total 1012.03 last 119.01) - open/close 49045 (time 1784095742.54 total 1022.03 last 134.17) total: 50000 open/close in 1030.39 seconds: 48.53 ops/second - unlinked 0 (time 1784095753 ; total 0 ; last 0) - unlinked 10000 (time 1784095833 ; total 80 ; last 80) - unlinked 20000 (time 1784095882 ; total 129 ; last 49) - unlinked 30000 (time 1784095926 ; total 173 ; last 44) - unlinked 40000 (time 1784095967 ; total 214 ; last 41) total: 50000 unlinks in 253 seconds: 197.628464 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: modules unloaded. pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2