== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:57:56 (1786341476) start mds service on oleg224-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 166 (time 1786341565.86 total 10.00 last 16.60) - open/close 428 (time 1786341576.60 total 20.75 last 24.39) - open/close 642 (time 1786341586.63 total 30.78 last 21.33) - open/close 885 (time 1786341599.20 total 43.34 last 19.35) - open/close 1146 (time 1786341609.83 total 53.97 last 24.55) - open/close 1330 (time 1786341620.18 total 64.33 last 17.77) - open/close 1460 (time 1786341631.19 total 75.33 last 11.81) - open/close 1651 (time 1786341641.55 total 85.70 last 18.43) - open/close 1876 (time 1786341651.57 total 95.71 last 22.46) - open/close 2091 (time 1786341661.58 total 105.72 last 21.48) - open/close 2355 (time 1786341672.02 total 116.16 last 25.29) - open/close 2617 (time 1786341682.88 total 127.02 last 24.12) - open/close 2868 (time 1786341692.88 total 137.03 last 25.09) - open/close 3095 (time 1786341703.69 total 147.84 last 21.00) - open/close 3318 (time 1786341714.38 total 158.53 last 20.86) - open/close 3550 (time 1786341724.39 total 168.54 last 23.17) - open/close 3787 (time 1786341734.40 total 178.55 last 23.68) - open/close 4026 (time 1786341744.91 total 189.06 last 22.74) - open/close 4327 (time 1786341754.92 total 199.06 last 30.08) - open/close 4587 (time 1786341765.09 total 209.23 last 25.56) - open/close 4833 (time 1786341775.70 total 219.85 last 23.18) - open/close 5073 (time 1786341785.95 total 230.09 last 23.43) - open/close 5341 (time 1786341796.31 total 240.45 last 25.87) - open/close 5597 (time 1786341806.97 total 251.11 last 24.01) - open/close 5863 (time 1786341816.99 total 261.13 last 26.55) - open/close 6081 (time 1786341827.31 total 271.46 last 21.11) - open/close 6207 (time 1786341838.58 total 282.73 last 11.18) - open/close 6418 (time 1786341848.60 total 292.74 last 21.08) - open/close 6636 (time 1786341858.77 total 302.92 last 21.42) - open/close 6863 (time 1786341868.78 total 312.93 last 22.68) - open/close 7097 (time 1786341879.25 total 323.39 last 22.36) - open/close 7331 (time 1786341889.25 total 333.40 last 23.40) - open/close 7509 (time 1786341899.54 total 343.68 last 17.31) - open/close 7761 (time 1786341909.79 total 353.93 last 24.59) - open/close 7997 (time 1786341919.95 total 364.10 last 23.21) - open/close 8235 (time 1786341930.70 total 374.84 last 22.15) - open/close 8452 (time 1786341941.08 total 385.22 last 20.90) - open/close 8675 (time 1786341951.12 total 395.27 last 22.20) - open/close 8920 (time 1786341961.56 total 405.70 last 23.47) - open/close 9127 (time 1786341971.56 total 415.70 last 20.70) - open/close 9342 (time 1786341981.74 total 425.88 last 21.12) - open/close 9536 (time 1786341992.28 total 436.42 last 18.41) - open/close 9766 (time 1786342003.16 total 447.30 last 21.14) - open/close 9977 (time 1786342013.65 total 457.80 last 20.11) - open/close 10174 (time 1786342023.66 total 467.80 last 19.69) - open/close 10425 (time 1786342034.60 total 478.75 last 22.93) - open/close 10662 (time 1786342044.75 total 488.89 last 23.36) - open/close 10960 (time 1786342054.76 total 498.91 last 29.75) - open/close 11126 (time 1786342065.29 total 509.44 last 15.76) - open/close 11288 (time 1786342075.34 total 519.49 last 16.12) - open/close 11470 (time 1786342085.93 total 530.07 last 17.19) - open/close 11688 (time 1786342095.95 total 540.10 last 21.75) - open/close 11886 (time 1786342106.14 total 550.29 last 19.43) - open/close 12129 (time 1786342116.14 total 560.29 last 24.30) - open/close 12361 (time 1786342126.17 total 570.32 last 23.13) - open/close 12593 (time 1786342136.18 total 580.33 last 23.18) - open/close 12787 (time 1786342146.51 total 590.66 last 18.78) - open/close 13009 (time 1786342156.52 total 600.67 last 22.18) - open/close 13246 (time 1786342166.54 total 610.69 last 23.66) - open/close 13487 (time 1786342176.66 total 620.81 last 23.81) - open/close 13721 (time 1786342186.86 total 631.00 last 22.95) - open/close 13984 (time 1786342196.87 total 641.01 last 26.28) - open/close 14217 (time 1786342207.83 total 651.98 last 21.25) - open/close 14421 (time 1786342218.33 total 662.47 last 19.44) - open/close 14580 (time 1786342228.63 total 672.78 last 15.43) - open/close 14793 (time 1786342241.44 total 685.58 last 16.64) - open/close 15017 (time 1786342251.45 total 695.59 last 22.37) - open/close 15285 (time 1786342261.45 total 705.60 last 26.78) - open/close 15492 (time 1786342271.70 total 715.84 last 20.21) - open/close 15722 (time 1786342281.70 total 725.84 last 23.00) - open/close 15911 (time 1786342291.87 total 736.01 last 18.59) - open/close 16139 (time 1786342301.87 total 746.01 last 22.80) - open/close 16321 (time 1786342312.73 total 756.88 last 16.76) - open/close 16571 (time 1786342323.74 total 767.88 last 22.71) - open/close 16732 (time 1786342333.76 total 777.90 last 16.07) - open/close 16975 (time 1786342343.78 total 787.93 last 24.23) - open/close 17157 (time 1786342354.28 total 798.42 last 17.34) - open/close 17442 (time 1786342364.95 total 809.09 last 26.72) - open/close 17675 (time 1786342374.96 total 819.10 last 23.27) - open/close 18382 (time 1786342384.97 total 829.12 last 70.61) - open/close 19399 (time 1786342394.97 total 839.12 last 101.69) - open/close 20000 (time 1786342401.59 total 845.74 last 90.79) - open/close 21047 (time 1786342411.60 total 855.74 last 104.64) - open/close 22080 (time 1786342421.60 total 865.75 last 103.29) - open/close 23261 (time 1786342431.60 total 875.75 last 118.10) - open/close 24459 (time 1786342441.60 total 885.75 last 119.78) - open/close 25465 (time 1786342451.61 total 895.76 last 100.52) - open/close 26509 (time 1786342461.62 total 905.76 last 104.35) - open/close 27628 (time 1786342471.62 total 915.76 last 111.89) - open/close 28755 (time 1786342481.62 total 925.77 last 112.61) - open/close 29907 (time 1786342491.63 total 935.78 last 115.11) - open/close 30943 (time 1786342501.64 total 945.78 last 103.56) - open/close 31984 (time 1786342511.65 total 955.80 last 103.91) - open/close 33074 (time 1786342521.66 total 965.80 last 108.94) - open/close 34519 (time 1786342531.66 total 975.81 last 144.47) - open/close 35646 (time 1786342541.66 total 985.81 last 112.67) - open/close 36750 (time 1786342551.67 total 995.82 last 110.30) - open/close 37944 (time 1786342561.68 total 1005.82 last 119.38) - open/close 39033 (time 1786342571.68 total 1015.82 last 108.86) - open/close 40000 (time 1786342580.70 total 1024.84 last 107.22) - open/close 41183 (time 1786342590.71 total 1034.85 last 118.22) - open/close 42291 (time 1786342600.71 total 1044.86 last 110.72) - open/close 43403 (time 1786342610.71 total 1054.86 last 111.19) - open/close 44576 (time 1786342620.72 total 1064.86 last 117.26) - open/close 45773 (time 1786342630.73 total 1074.88 last 119.51) - open/close 46909 (time 1786342640.73 total 1084.88 last 113.59) - open/close 48087 (time 1786342650.74 total 1094.88 last 117.77) - open/close 49200 (time 1786342660.75 total 1104.90 last 111.14) total: 50000 open/close in 1111.68 seconds: 44.98 ops/second - unlinked 0 (time 1786342670 ; total 0 ; last 0) - unlinked 10000 (time 1786342756 ; total 86 ; last 86) - unlinked 20000 (time 1786342822 ; total 152 ; last 66) - unlinked 30000 (time 1786342872 ; total 202 ; last 50) - unlinked 40000 (time 1786342925 ; total 255 ; last 53) total: 50000 unlinks in 303 seconds: 165.016495 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2