== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:24:33 (1763342673) start mds service on oleg310-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1763342748.43 total 10.01 last 20.68) - open/close 403 (time 1763342758.98 total 20.56 last 18.58) - open/close 605 (time 1763342769.93 total 31.50 last 18.45) - open/close 805 (time 1763342779.93 total 41.51 last 19.99) - open/close 1065 (time 1763342789.95 total 51.52 last 25.97) - open/close 1287 (time 1763342800.64 total 62.22 last 20.75) - open/close 1473 (time 1763342810.66 total 72.23 last 18.57) - open/close 1696 (time 1763342820.87 total 82.45 last 21.84) - open/close 1940 (time 1763342831.64 total 93.21 last 22.67) - open/close 2196 (time 1763342841.65 total 103.22 last 25.57) - open/close 2436 (time 1763342851.74 total 113.32 last 23.77) - open/close 2666 (time 1763342862.80 total 124.38 last 20.80) - open/close 2831 (time 1763342872.81 total 134.38 last 16.49) - open/close 3045 (time 1763342882.81 total 144.38 last 21.40) - open/close 3252 (time 1763342892.81 total 154.39 last 20.70) - open/close 3436 (time 1763342903.17 total 164.75 last 17.76) - open/close 3726 (time 1763342914.12 total 175.70 last 26.48) - open/close 3905 (time 1763342924.14 total 185.71 last 17.87) - open/close 4083 (time 1763342934.41 total 195.98 last 17.33) - open/close 4305 (time 1763342944.41 total 205.98 last 22.19) - open/close 4501 (time 1763342955.02 total 216.59 last 18.48) - open/close 4713 (time 1763342965.70 total 227.27 last 19.85) - open/close 4931 (time 1763342976.66 total 238.23 last 19.89) - open/close 5164 (time 1763342987.35 total 248.92 last 21.79) - open/close 5374 (time 1763342997.37 total 258.94 last 20.96) - open/close 5626 (time 1763343007.38 total 268.96 last 25.16) - open/close 5827 (time 1763343017.40 total 278.97 last 20.07) - open/close 6035 (time 1763343027.60 total 289.17 last 20.40) - open/close 6298 (time 1763343038.42 total 299.99 last 24.30) - open/close 6562 (time 1763343049.04 total 310.61 last 24.87) - open/close 6767 (time 1763343060.05 total 321.62 last 18.62) - open/close 6974 (time 1763343070.05 total 331.62 last 20.70) - open/close 7169 (time 1763343080.07 total 341.64 last 19.46) - open/close 7382 (time 1763343090.57 total 352.14 last 20.28) - open/close 7569 (time 1763343100.75 total 362.32 last 18.38) - open/close 7784 (time 1763343111.56 total 373.13 last 19.89) - open/close 7995 (time 1763343121.56 total 383.13 last 21.10) - open/close 8211 (time 1763343131.56 total 393.13 last 21.60) - open/close 8425 (time 1763343142.12 total 403.69 last 20.28) - open/close 8500 (time 1763343152.27 total 413.85 last 7.38) - open/close 8706 (time 1763343162.51 total 424.09 last 20.12) - open/close 8953 (time 1763343172.68 total 434.26 last 24.29) - open/close 9161 (time 1763343183.70 total 445.27 last 18.88) - open/close 9390 (time 1763343194.57 total 456.14 last 21.06) - open/close 9612 (time 1763343204.61 total 466.18 last 22.11) - open/close 9862 (time 1763343215.14 total 476.72 last 23.73) - open/close 10000 (time 1763343221.65 total 483.22 last 21.22) - open/close 10223 (time 1763343232.26 total 493.84 last 21.01) - open/close 10457 (time 1763343245.01 total 506.58 last 18.36) - open/close 10682 (time 1763343255.79 total 517.36 last 20.87) - open/close 10958 (time 1763343266.12 total 527.69 last 26.73) - open/close 11130 (time 1763343276.24 total 537.82 last 16.98) - open/close 11350 (time 1763343286.25 total 547.83 last 21.98) - open/close 11529 (time 1763343296.66 total 558.23 last 17.20) - open/close 11754 (time 1763343306.76 total 568.34 last 22.27) - open/close 11964 (time 1763343316.77 total 578.35 last 20.99) - open/close 12129 (time 1763343327.71 total 589.28 last 15.08) - open/close 12318 (time 1763343338.25 total 599.83 last 17.93) - open/close 12450 (time 1763343348.52 total 610.09 last 12.86) - open/close 12607 (time 1763343358.52 total 620.10 last 15.69) - open/close 12812 (time 1763343368.53 total 630.11 last 20.48) - open/close 12999 (time 1763343379.15 total 640.73 last 17.61) - open/close 13187 (time 1763343389.64 total 651.21 last 17.93) - open/close 13374 (time 1763343400.14 total 661.72 last 17.80) - open/close 13538 (time 1763343410.15 total 671.73 last 16.38) - open/close 13735 (time 1763343420.50 total 682.08 last 19.03) - open/close 13945 (time 1763343430.52 total 692.10 last 20.96) - open/close 14142 (time 1763343440.52 total 702.10 last 19.70) - open/close 14398 (time 1763343451.14 total 712.72 last 24.11) - open/close 14583 (time 1763343461.41 total 722.98 last 18.02) - open/close 14820 (time 1763343471.43 total 733.01 last 23.63) - open/close 15035 (time 1763343481.44 total 743.01 last 21.49) - open/close 15202 (time 1763343491.44 total 753.02 last 16.69) - open/close 15422 (time 1763343502.47 total 764.05 last 19.94) - open/close 15623 (time 1763343513.36 total 774.93 last 18.47) - open/close 15831 (time 1763343523.42 total 784.99 last 20.67) - open/close 16177 (time 1763343533.43 total 795.00 last 34.56) - open/close 16928 (time 1763343543.44 total 805.01 last 75.02) - open/close 17670 (time 1763343553.45 total 815.02 last 74.12) - open/close 18435 (time 1763343563.46 total 825.04 last 76.40) - open/close 19213 (time 1763343573.47 total 835.05 last 77.73) - open/close 20000 (time 1763343582.97 total 844.55 last 82.81) - open/close 20759 (time 1763343592.99 total 854.56 last 75.80) - open/close 21494 (time 1763343602.99 total 864.56 last 73.50) - open/close 22236 (time 1763343613.00 total 874.57 last 74.15) - open/close 22985 (time 1763343623.01 total 884.59 last 74.77) - open/close 23738 (time 1763343633.02 total 894.60 last 75.23) - open/close 24383 (time 1763343643.04 total 904.62 last 64.38) - open/close 25115 (time 1763343653.05 total 914.62 last 73.17) - open/close 25833 (time 1763343663.06 total 924.63 last 71.69) - open/close 26596 (time 1763343673.08 total 934.66 last 76.12) - open/close 27422 (time 1763343683.09 total 944.67 last 82.52) - open/close 28226 (time 1763343693.11 total 954.68 last 80.30) - open/close 29025 (time 1763343703.12 total 964.69 last 79.79) - open/close 29863 (time 1763343713.14 total 974.71 last 83.67) - open/close 30677 (time 1763343723.15 total 984.72 last 81.30) - open/close 31481 (time 1763343733.16 total 994.73 last 80.31) - open/close 32210 (time 1763343743.16 total 1004.74 last 72.88) - open/close 33038 (time 1763343753.16 total 1014.74 last 82.78) - open/close 33874 (time 1763343763.18 total 1024.75 last 83.51) - open/close 34715 (time 1763343773.19 total 1034.76 last 84.00) - open/close 35471 (time 1763343783.20 total 1044.77 last 75.52) - open/close 36225 (time 1763343793.21 total 1054.78 last 75.34) - open/close 37036 (time 1763343803.21 total 1064.79 last 81.04) - open/close 37821 (time 1763343813.21 total 1074.79 last 78.50) - open/close 38668 (time 1763343823.22 total 1084.79 last 84.66) - open/close 39441 (time 1763343833.22 total 1094.79 last 77.28) - open/close 40000 (time 1763343840.93 total 1102.50 last 72.54) - open/close 40809 (time 1763343850.94 total 1112.51 last 80.78) - open/close 41522 (time 1763343860.95 total 1122.52 last 71.23) - open/close 42247 (time 1763343870.96 total 1132.53 last 72.47) - open/close 42987 (time 1763343880.97 total 1142.54 last 73.89) - open/close 43749 (time 1763343890.97 total 1152.55 last 76.17) - open/close 44505 (time 1763343900.98 total 1162.56 last 75.53) - open/close 45321 (time 1763343910.99 total 1172.57 last 81.53) - open/close 46008 (time 1763343921.00 total 1182.57 last 68.65) - open/close 46773 (time 1763343931.01 total 1192.59 last 76.38) - open/close 47565 (time 1763343941.02 total 1202.60 last 79.12) - open/close 48305 (time 1763343951.03 total 1212.60 last 73.96) - open/close 49087 (time 1763343961.04 total 1222.61 last 78.12) - open/close 49886 (time 1763343971.04 total 1232.62 last 79.87) total: 50000 open/close in 1234.11 seconds: 40.51 ops/second - unlinked 0 (time 1763343976 ; total 0 ; last 0) - unlinked 10000 (time 1763344113 ; total 137 ; last 137) - unlinked 20000 (time 1763344249 ; total 273 ; last 136) - unlinked 30000 (time 1763344386 ; total 410 ; last 137) - unlinked 40000 (time 1763344529 ; total 553 ; last 143) total: 50000 unlinks in 691 seconds: 72.358902 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg310-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2