== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:31:02 (1782225062) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 126 (time 1782225172.22 total 10.14 last 12.42) - open/close 339 (time 1782225184.40 total 22.32 last 17.49) - open/close 479 (time 1782225194.57 total 32.49 last 13.77) - open/close 580 (time 1782225204.58 total 42.50 last 10.08) - open/close 801 (time 1782225214.78 total 52.70 last 21.67) - open/close 1050 (time 1782225224.78 total 62.70 last 24.89) - open/close 1330 (time 1782225235.78 total 73.70 last 25.46) - open/close 1594 (time 1782225245.80 total 83.72 last 26.35) - open/close 1875 (time 1782225256.70 total 94.62 last 25.77) - open/close 2137 (time 1782225266.73 total 104.65 last 26.14) - open/close 2396 (time 1782225276.92 total 114.85 last 25.40) - open/close 2676 (time 1782225287.95 total 125.87 last 25.40) - open/close 2924 (time 1782225298.48 total 136.40 last 23.54) - open/close 3146 (time 1782225309.20 total 147.13 last 20.71) - open/close 3380 (time 1782225319.21 total 157.13 last 23.39) - open/close 3663 (time 1782225329.21 total 167.13 last 28.29) - open/close 3929 (time 1782225339.97 total 177.89 last 24.72) - open/close 4164 (time 1782225350.98 total 188.90 last 21.35) - open/close 4404 (time 1782225361.75 total 199.67 last 22.29) - open/close 4576 (time 1782225372.16 total 210.08 last 16.51) - open/close 4772 (time 1782225382.16 total 220.09 last 19.60) - open/close 4972 (time 1782225392.80 total 230.72 last 18.81) - open/close 5240 (time 1782225403.45 total 241.37 last 25.15) - open/close 5521 (time 1782225413.47 total 251.39 last 28.06) - open/close 5802 (time 1782225423.47 total 261.39 last 28.10) - open/close 6066 (time 1782225433.99 total 271.91 last 25.10) - open/close 6343 (time 1782225444.60 total 282.52 last 26.10) - open/close 6621 (time 1782225454.60 total 292.52 last 27.80) - open/close 6796 (time 1782225464.61 total 302.53 last 17.48) - open/close 7010 (time 1782225474.63 total 312.55 last 21.36) - open/close 7259 (time 1782225484.74 total 322.66 last 24.64) - open/close 7527 (time 1782225495.29 total 333.21 last 25.39) - open/close 7745 (time 1782225506.31 total 344.23 last 19.80) - open/close 7979 (time 1782225516.31 total 354.23 last 23.38) - open/close 8211 (time 1782225526.32 total 364.24 last 23.19) - open/close 8449 (time 1782225536.38 total 374.30 last 23.66) - open/close 8671 (time 1782225546.55 total 384.47 last 21.83) - open/close 8921 (time 1782225556.64 total 394.56 last 24.76) - open/close 9124 (time 1782225566.65 total 404.57 last 20.30) - open/close 9396 (time 1782225577.34 total 415.26 last 25.43) - open/close 9678 (time 1782225587.36 total 425.29 last 28.13) - open/close 9962 (time 1782225598.21 total 436.13 last 26.20) - open/close 10244 (time 1782225608.21 total 446.13 last 28.19) - open/close 10536 (time 1782225618.21 total 456.13 last 29.20) - open/close 10787 (time 1782225628.22 total 466.14 last 25.09) - open/close 11044 (time 1782225638.22 total 476.14 last 25.70) - open/close 11322 (time 1782225648.22 total 486.14 last 27.80) - open/close 11509 (time 1782225658.62 total 496.54 last 17.97) - open/close 11757 (time 1782225668.63 total 506.55 last 24.78) - open/close 12006 (time 1782225679.35 total 517.27 last 23.23) - open/close 12296 (time 1782225689.36 total 527.28 last 28.99) - open/close 12507 (time 1782225699.37 total 537.29 last 21.07) - open/close 12589 (time 1782225709.37 total 547.29 last 8.20) - open/close 12694 (time 1782225719.92 total 557.85 last 9.95) - open/close 12840 (time 1782225730.24 total 568.16 last 14.15) - open/close 13089 (time 1782225740.25 total 578.17 last 24.87) - open/close 13265 (time 1782225750.37 total 588.29 last 17.40) - open/close 13501 (time 1782225760.75 total 598.67 last 22.72) - open/close 13744 (time 1782225771.45 total 609.37 last 22.71) - open/close 13947 (time 1782225781.47 total 619.39 last 20.25) - open/close 14120 (time 1782225791.82 total 629.74 last 16.72) - open/close 14357 (time 1782225802.11 total 640.03 last 23.04) - open/close 14665 (time 1782225812.13 total 650.05 last 30.73) - open/close 14911 (time 1782225822.41 total 660.33 last 23.92) - open/close 15155 (time 1782225832.75 total 670.67 last 23.60) - open/close 15437 (time 1782225843.45 total 681.37 last 26.37) - open/close 15642 (time 1782225853.45 total 691.37 last 20.48) - open/close 15927 (time 1782225863.47 total 701.39 last 28.45) - open/close 16196 (time 1782225873.78 total 711.70 last 26.09) - open/close 16478 (time 1782225883.80 total 721.72 last 28.15) - open/close 16769 (time 1782225893.81 total 731.74 last 29.05) - open/close 16927 (time 1782225904.50 total 742.42 last 14.79) - open/close 17200 (time 1782225914.51 total 752.43 last 27.27) - open/close 17416 (time 1782225924.61 total 762.53 last 21.39) - open/close 17652 (time 1782225935.17 total 773.09 last 22.35) - open/close 17947 (time 1782225945.18 total 783.10 last 29.47) - open/close 18235 (time 1782225955.19 total 793.11 last 28.76) - open/close 18470 (time 1782225965.20 total 803.12 last 23.48) - open/close 18705 (time 1782225976.19 total 814.11 last 21.38) - open/close 18915 (time 1782225986.21 total 824.13 last 20.96) - open/close 19225 (time 1782225996.99 total 834.91 last 28.77) - open/close 19553 (time 1782226007.45 total 845.37 last 31.36) - open/close 19821 (time 1782226017.80 total 855.72 last 25.89) - open/close 20000 (time 1782226026.04 total 863.96 last 21.71) - open/close 20291 (time 1782226036.06 total 873.98 last 29.06) - open/close 20545 (time 1782226046.66 total 884.58 last 23.95) - open/close 20769 (time 1782226056.67 total 894.59 last 22.38) - open/close 20983 (time 1782226066.68 total 904.60 last 21.39) - open/close 21121 (time 1782226079.70 total 917.62 last 10.60) - open/close 21358 (time 1782226089.71 total 927.63 last 23.67) - open/close 22175 (time 1782226099.72 total 937.64 last 81.65) - open/close 23030 (time 1782226109.72 total 947.65 last 85.46) - open/close 23781 (time 1782226119.73 total 957.65 last 75.08) - open/close 24459 (time 1782226129.75 total 967.67 last 67.66) - open/close 25205 (time 1782226139.76 total 977.68 last 74.54) - open/close 26028 (time 1782226149.76 total 987.69 last 82.23) - open/close 26900 (time 1782226159.77 total 997.69 last 87.16) - open/close 27738 (time 1782226169.77 total 1007.69 last 83.79) - open/close 28584 (time 1782226179.78 total 1017.70 last 84.55) - open/close 29397 (time 1782226189.78 total 1027.70 last 81.27) - open/close 30000 (time 1782226196.71 total 1034.63 last 87.02) - open/close 30855 (time 1782226206.71 total 1044.63 last 85.47) - open/close 31677 (time 1782226216.72 total 1054.64 last 82.18) - open/close 32455 (time 1782226226.72 total 1064.64 last 77.79) - open/close 33303 (time 1782226236.73 total 1074.65 last 84.72) - open/close 34155 (time 1782226246.73 total 1084.66 last 85.14) - open/close 34854 (time 1782226256.76 total 1094.68 last 69.70) - open/close 35550 (time 1782226266.77 total 1104.69 last 69.56) - open/close 36297 (time 1782226276.79 total 1114.71 last 74.52) - open/close 36967 (time 1782226286.80 total 1124.72 last 66.94) - open/close 37709 (time 1782226296.80 total 1134.72 last 74.19) - open/close 38457 (time 1782226306.81 total 1144.73 last 74.74) - open/close 39334 (time 1782226316.81 total 1154.73 last 87.67) - open/close 40000 (time 1782226324.66 total 1162.58 last 84.85) - open/close 40886 (time 1782226334.67 total 1172.59 last 88.55) - open/close 41577 (time 1782226344.67 total 1182.59 last 69.09) - open/close 42256 (time 1782226354.67 total 1192.59 last 67.88) - open/close 43066 (time 1782226364.68 total 1202.60 last 80.97) - open/close 43855 (time 1782226374.69 total 1212.61 last 78.80) - open/close 44652 (time 1782226384.69 total 1222.61 last 79.66) - open/close 45411 (time 1782226394.70 total 1232.62 last 75.87) - open/close 46309 (time 1782226404.70 total 1242.62 last 89.75) - open/close 47117 (time 1782226414.71 total 1252.63 last 80.73) - open/close 47845 (time 1782226424.71 total 1262.63 last 72.79) - open/close 48716 (time 1782226434.72 total 1272.64 last 87.04) - open/close 49511 (time 1782226444.75 total 1282.67 last 79.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1284.11 seconds: 38.65 ops/second - unlinked 0 (time 1782226451 ; total 0 ; last 0) - unlinked 10000 (time 1782226581 ; total 130 ; last 130) - unlinked 20000 (time 1782226712 ; total 261 ; last 131) - unlinked 30000 (time 1782226845 ; total 394 ; last 133) - unlinked 40000 (time 1782226972 ; total 521 ; last 127) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 641 seconds: 77.427460 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2