== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:26:26 (1761287186) start mds service on oleg306-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 241 (time 1761287250.98 total 10.00 last 24.10) - open/close 416 (time 1761287260.98 total 20.00 last 17.50) - open/close 607 (time 1761287271.02 total 30.05 last 19.02) - open/close 828 (time 1761287281.04 total 40.06 last 22.06) - open/close 968 (time 1761287291.05 total 50.08 last 13.98) - open/close 1227 (time 1761287301.06 total 60.08 last 25.89) - open/close 1403 (time 1761287311.29 total 70.31 last 17.20) - open/close 1605 (time 1761287321.77 total 80.79 last 19.27) - open/close 1831 (time 1761287331.78 total 90.80 last 22.59) - open/close 2049 (time 1761287341.79 total 100.81 last 21.78) - open/close 2282 (time 1761287351.80 total 110.82 last 23.27) - open/close 2576 (time 1761287361.82 total 120.84 last 29.35) - open/close 2788 (time 1761287371.85 total 130.87 last 21.14) - open/close 3041 (time 1761287382.28 total 141.30 last 24.24) - open/close 3201 (time 1761287392.51 total 151.53 last 15.64) - open/close 3445 (time 1761287403.06 total 162.08 last 23.14) - open/close 3677 (time 1761287413.07 total 172.09 last 23.17) - open/close 3869 (time 1761287425.78 total 184.80 last 15.11) - open/close 4091 (time 1761287436.46 total 195.48 last 20.78) - open/close 4320 (time 1761287446.47 total 205.49 last 22.88) - open/close 4509 (time 1761287456.62 total 215.64 last 18.62) - open/close 4755 (time 1761287467.20 total 226.22 last 23.25) - open/close 4988 (time 1761287477.85 total 236.88 last 21.87) - open/close 5195 (time 1761287488.36 total 247.38 last 19.71) - open/close 5371 (time 1761287499.24 total 258.26 last 16.17) - open/close 5570 (time 1761287509.79 total 268.81 last 18.87) - open/close 5803 (time 1761287519.97 total 278.99 last 22.88) - open/close 6037 (time 1761287530.86 total 289.89 last 21.48) - open/close 6264 (time 1761287541.03 total 300.05 last 22.33) - open/close 6494 (time 1761287551.03 total 310.06 last 22.99) - open/close 6689 (time 1761287561.65 total 320.67 last 18.37) - open/close 6922 (time 1761287573.73 total 332.75 last 19.29) - open/close 7136 (time 1761287584.56 total 343.59 last 19.76) - open/close 7393 (time 1761287595.36 total 354.38 last 23.80) - open/close 7611 (time 1761287605.43 total 364.45 last 21.65) - open/close 7788 (time 1761287615.74 total 374.77 last 17.16) - open/close 8011 (time 1761287625.77 total 384.79 last 22.25) - open/close 8169 (time 1761287635.83 total 394.85 last 15.71) - open/close 8421 (time 1761287646.49 total 405.51 last 23.63) - open/close 8722 (time 1761287657.03 total 416.05 last 28.56) - open/close 8947 (time 1761287667.04 total 426.06 last 22.47) - open/close 9163 (time 1761287677.42 total 436.44 last 20.81) - open/close 9370 (time 1761287688.42 total 447.44 last 18.82) - open/close 9686 (time 1761287698.42 total 457.45 last 31.58) - open/close 9926 (time 1761287708.93 total 467.95 last 22.85) - open/close 10000 (time 1761287711.08 total 470.10 last 34.43) - open/close 10222 (time 1761287722.03 total 481.05 last 20.27) - open/close 10421 (time 1761287732.40 total 491.42 last 19.19) - open/close 10597 (time 1761287744.69 total 503.71 last 14.33) - open/close 10817 (time 1761287755.13 total 514.15 last 21.07) - open/close 11044 (time 1761287765.60 total 524.62 last 21.69) - open/close 11264 (time 1761287775.65 total 534.68 last 21.87) - open/close 11491 (time 1761287785.66 total 544.69 last 22.68) - open/close 11773 (time 1761287795.67 total 554.69 last 28.19) - open/close 12006 (time 1761287805.70 total 564.73 last 23.22) - open/close 12194 (time 1761287816.06 total 575.08 last 18.16) - open/close 12440 (time 1761287826.47 total 585.49 last 23.63) - open/close 12618 (time 1761287836.58 total 595.60 last 17.61) - open/close 12843 (time 1761287846.59 total 605.61 last 22.47) - open/close 13035 (time 1761287856.99 total 616.01 last 18.46) - open/close 13291 (time 1761287867.50 total 626.52 last 24.35) - open/close 13442 (time 1761287877.51 total 636.53 last 15.09) - open/close 13629 (time 1761287887.52 total 646.54 last 18.67) - open/close 13802 (time 1761287897.54 total 656.56 last 17.28) - open/close 14024 (time 1761287907.56 total 666.58 last 22.15) - open/close 14224 (time 1761287918.25 total 677.27 last 18.71) - open/close 14472 (time 1761287928.96 total 687.98 last 23.14) - open/close 14670 (time 1761287939.13 total 698.15 last 19.47) - open/close 14930 (time 1761287949.15 total 708.17 last 25.95) - open/close 15152 (time 1761287959.36 total 718.38 last 21.75) - open/close 15396 (time 1761287970.42 total 729.44 last 22.07) - open/close 15622 (time 1761287980.43 total 739.45 last 22.57) - open/close 15837 (time 1761287991.16 total 750.18 last 20.03) - open/close 16099 (time 1761288001.71 total 760.73 last 24.85) - open/close 16295 (time 1761288011.88 total 770.90 last 19.26) - open/close 16505 (time 1761288021.88 total 780.91 last 21.00) - open/close 16706 (time 1761288032.86 total 791.88 last 18.31) - open/close 16903 (time 1761288043.19 total 802.21 last 19.07) - open/close 17077 (time 1761288053.79 total 812.81 last 16.41) - open/close 17335 (time 1761288063.80 total 822.82 last 25.78) - open/close 18331 (time 1761288073.80 total 832.82 last 99.59) - open/close 19203 (time 1761288083.80 total 842.83 last 87.16) - open/close 20000 (time 1761288091.94 total 850.96 last 98.00) - open/close 20984 (time 1761288101.95 total 860.97 last 98.27) - open/close 21900 (time 1761288111.97 total 871.00 last 91.36) - open/close 22735 (time 1761288121.99 total 881.01 last 83.36) - open/close 23695 (time 1761288132.00 total 891.02 last 95.93) - open/close 24647 (time 1761288142.00 total 901.02 last 95.20) - open/close 25525 (time 1761288152.02 total 911.04 last 87.65) - open/close 26384 (time 1761288162.03 total 921.05 last 85.79) - open/close 27281 (time 1761288172.03 total 931.06 last 89.66) - open/close 28182 (time 1761288182.04 total 941.06 last 90.07) - open/close 29161 (time 1761288192.05 total 951.07 last 97.82) - open/close 30000 (time 1761288201.65 total 960.67 last 87.35) - open/close 30989 (time 1761288211.66 total 970.68 last 98.83) - open/close 32099 (time 1761288221.66 total 980.68 last 110.98) - open/close 33157 (time 1761288231.67 total 990.69 last 105.73) - open/close 34283 (time 1761288241.67 total 1000.69 last 112.57) - open/close 35564 (time 1761288251.67 total 1010.69 last 128.10) - open/close 36916 (time 1761288261.67 total 1020.70 last 135.13) - open/close 38238 (time 1761288271.68 total 1030.70 last 132.14) - open/close 39490 (time 1761288281.69 total 1040.71 last 125.11) - open/close 40000 (time 1761288286.77 total 1045.79 last 100.37) - open/close 41076 (time 1761288296.78 total 1055.80 last 107.48) - open/close 42114 (time 1761288306.79 total 1065.81 last 103.72) - open/close 43142 (time 1761288316.81 total 1075.83 last 102.59) - open/close 44125 (time 1761288326.81 total 1085.83 last 98.29) - open/close 45128 (time 1761288336.82 total 1095.84 last 100.19) - open/close 46233 (time 1761288346.82 total 1105.84 last 110.48) - open/close 47338 (time 1761288356.83 total 1115.85 last 110.43) - open/close 48680 (time 1761288366.83 total 1125.85 last 134.19) - open/close 49762 (time 1761288376.83 total 1135.86 last 108.14) total: 50000 open/close in 1138.14 seconds: 43.93 ops/second - unlinked 0 (time 1761288381 ; total 0 ; last 0) - unlinked 10000 (time 1761288511 ; total 130 ; last 130) - unlinked 20000 (time 1761288608 ; total 227 ; last 97) - unlinked 30000 (time 1761288678 ; total 297 ; last 70) - unlinked 40000 (time 1761288748 ; total 367 ; last 70) total: 50000 unlinks in 421 seconds: 118.764847 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg306-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2