== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:53:59 (1763344439) start mds service on oleg301-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 219 (time 1763344508.07 total 10.01 last 21.88) - open/close 414 (time 1763344518.54 total 20.48 last 18.62) - open/close 655 (time 1763344528.60 total 30.55 last 23.94) - open/close 850 (time 1763344539.09 total 41.04 last 18.59) - open/close 1085 (time 1763344549.11 total 51.05 last 23.46) - open/close 1243 (time 1763344559.83 total 61.78 last 14.74) - open/close 1455 (time 1763344570.38 total 72.32 last 20.10) - open/close 1656 (time 1763344580.62 total 82.56 last 19.64) - open/close 1887 (time 1763344590.62 total 92.57 last 23.08) - open/close 2080 (time 1763344601.36 total 103.30 last 17.98) - open/close 2279 (time 1763344611.37 total 113.32 last 19.87) - open/close 2520 (time 1763344622.41 total 124.36 last 21.83) - open/close 2734 (time 1763344632.42 total 134.36 last 21.39) - open/close 2863 (time 1763344642.42 total 144.36 last 12.90) - open/close 3122 (time 1763344652.50 total 154.44 last 25.71) - open/close 3319 (time 1763344662.93 total 164.88 last 18.87) - open/close 3514 (time 1763344673.25 total 175.19 last 18.91) - open/close 3740 (time 1763344683.34 total 185.28 last 22.40) - open/close 3997 (time 1763344693.35 total 195.29 last 25.67) - open/close 4174 (time 1763344703.76 total 205.70 last 17.00) - open/close 4392 (time 1763344714.59 total 216.53 last 20.13) - open/close 4600 (time 1763344725.13 total 227.08 last 19.73) - open/close 4783 (time 1763344735.60 total 237.54 last 17.49) - open/close 5022 (time 1763344746.64 total 248.58 last 21.65) - open/close 5291 (time 1763344757.46 total 259.40 last 24.86) - open/close 5529 (time 1763344768.35 total 270.29 last 21.85) - open/close 5688 (time 1763344779.15 total 281.09 last 14.72) - open/close 5839 (time 1763344789.64 total 291.58 last 14.40) - open/close 6004 (time 1763344799.95 total 301.90 last 16.00) - open/close 6196 (time 1763344810.34 total 312.28 last 18.48) - open/close 6331 (time 1763344820.70 total 322.64 last 13.03) - open/close 6500 (time 1763344830.77 total 332.71 last 16.78) - open/close 6724 (time 1763344841.18 total 343.12 last 21.53) - open/close 6955 (time 1763344851.60 total 353.54 last 22.17) - open/close 7168 (time 1763344861.60 total 363.55 last 21.29) - open/close 7384 (time 1763344871.61 total 373.55 last 21.59) - open/close 7621 (time 1763344882.64 total 384.59 last 21.48) - open/close 7825 (time 1763344893.20 total 395.14 last 19.33) - open/close 8069 (time 1763344903.57 total 405.51 last 23.52) - open/close 8236 (time 1763344913.58 total 415.52 last 16.68) - open/close 8451 (time 1763344924.18 total 426.12 last 20.28) - open/close 8677 (time 1763344934.21 total 436.15 last 22.53) - open/close 8889 (time 1763344944.22 total 446.16 last 21.18) - open/close 9060 (time 1763344954.23 total 456.17 last 17.09) - open/close 9240 (time 1763344964.45 total 466.39 last 17.61) - open/close 9447 (time 1763344975.19 total 477.13 last 19.28) - open/close 9693 (time 1763344985.19 total 487.13 last 24.59) - open/close 9980 (time 1763344995.79 total 497.73 last 27.08) - open/close 10000 (time 1763344998.31 total 500.25 last 7.93) - open/close 10214 (time 1763345008.59 total 510.53 last 20.83) - open/close 10416 (time 1763345018.59 total 520.53 last 20.20) - open/close 10617 (time 1763345028.95 total 530.89 last 19.40) - open/close 10798 (time 1763345039.13 total 541.08 last 17.77) - open/close 11036 (time 1763345049.56 total 551.50 last 22.83) - open/close 11245 (time 1763345059.66 total 561.60 last 20.69) - open/close 11428 (time 1763345069.67 total 571.61 last 18.28) - open/close 11642 (time 1763345079.76 total 581.70 last 21.21) - open/close 11864 (time 1763345090.02 total 591.97 last 21.62) - open/close 12093 (time 1763345100.04 total 601.98 last 22.87) - open/close 12336 (time 1763345110.07 total 612.01 last 24.23) - open/close 12547 (time 1763345120.60 total 622.55 last 20.02) - open/close 12772 (time 1763345131.16 total 633.10 last 21.32) - open/close 12986 (time 1763345141.84 total 643.78 last 20.04) - open/close 13201 (time 1763345152.75 total 654.69 last 19.71) - open/close 13406 (time 1763345163.35 total 665.30 last 19.33) - open/close 13612 (time 1763345174.36 total 676.30 last 18.73) - open/close 13799 (time 1763345184.53 total 686.47 last 18.38) - open/close 13988 (time 1763345194.53 total 696.48 last 18.89) - open/close 14246 (time 1763345204.55 total 706.49 last 25.76) - open/close 14455 (time 1763345214.56 total 716.51 last 20.87) - open/close 14681 (time 1763345225.07 total 727.01 last 21.52) - open/close 14915 (time 1763345235.42 total 737.36 last 22.60) - open/close 15148 (time 1763345245.52 total 747.46 last 23.07) - open/close 15368 (time 1763345256.60 total 758.54 last 19.86) - open/close 15583 (time 1763345267.47 total 769.41 last 19.78) - open/close 15804 (time 1763345278.36 total 780.30 last 20.30) - open/close 16038 (time 1763345288.36 total 790.30 last 23.39) - open/close 16814 (time 1763345298.37 total 800.31 last 77.53) - open/close 17506 (time 1763345308.38 total 810.32 last 69.12) - open/close 18241 (time 1763345318.39 total 820.33 last 73.41) - open/close 19002 (time 1763345328.40 total 830.34 last 76.04) - open/close 19770 (time 1763345338.40 total 840.34 last 76.79) - open/close 20000 (time 1763345341.19 total 843.14 last 82.34) - open/close 20829 (time 1763345351.20 total 853.15 last 82.82) - open/close 21572 (time 1763345361.21 total 863.15 last 74.24) - open/close 22328 (time 1763345371.22 total 873.16 last 75.57) - open/close 23074 (time 1763345381.22 total 883.16 last 74.58) - open/close 23834 (time 1763345391.22 total 893.16 last 75.99) - open/close 24553 (time 1763345401.23 total 903.17 last 71.85) - open/close 25291 (time 1763345411.23 total 913.17 last 73.80) - open/close 26000 (time 1763345421.25 total 923.19 last 70.74) - open/close 26833 (time 1763345431.25 total 933.20 last 83.26) - open/close 27658 (time 1763345441.27 total 943.21 last 82.37) - open/close 28462 (time 1763345451.27 total 953.21 last 80.39) - open/close 29307 (time 1763345461.27 total 963.22 last 84.48) - open/close 30000 (time 1763345470.45 total 972.39 last 75.51) - open/close 30737 (time 1763345480.45 total 982.39 last 73.70) - open/close 31545 (time 1763345490.46 total 992.40 last 80.72) - open/close 32464 (time 1763345500.46 total 1002.40 last 91.87) - open/close 33263 (time 1763345510.48 total 1012.43 last 79.74) - open/close 34068 (time 1763345520.49 total 1022.43 last 80.43) - open/close 34810 (time 1763345530.50 total 1032.44 last 74.17) - open/close 35532 (time 1763345540.50 total 1042.44 last 72.19) - open/close 36214 (time 1763345550.50 total 1052.45 last 68.16) - open/close 36822 (time 1763345560.52 total 1062.46 last 60.71) - open/close 37494 (time 1763345570.52 total 1072.46 last 67.18) - open/close 38189 (time 1763345580.55 total 1082.49 last 69.33) - open/close 38895 (time 1763345590.55 total 1092.49 last 70.60) - open/close 39647 (time 1763345600.55 total 1102.49 last 75.17) - open/close 40000 (time 1763345605.23 total 1107.17 last 75.43) - open/close 40824 (time 1763345615.24 total 1117.18 last 82.35) - open/close 41498 (time 1763345625.24 total 1127.18 last 67.36) - open/close 42276 (time 1763345635.24 total 1137.19 last 77.79) - open/close 43086 (time 1763345645.25 total 1147.19 last 80.94) - open/close 43876 (time 1763345655.25 total 1157.20 last 78.98) - open/close 44604 (time 1763345665.26 total 1167.20 last 72.77) - open/close 45392 (time 1763345675.27 total 1177.21 last 78.73) - open/close 46138 (time 1763345685.27 total 1187.21 last 74.60) - open/close 46922 (time 1763345695.28 total 1197.22 last 78.30) - open/close 47605 (time 1763345705.28 total 1207.23 last 68.27) - open/close 48308 (time 1763345715.29 total 1217.24 last 70.23) - open/close 49051 (time 1763345725.30 total 1227.24 last 74.27) - open/close 49774 (time 1763345735.31 total 1237.26 last 72.17) total: 50000 open/close in 1240.22 seconds: 40.32 ops/second - unlinked 0 (time 1763345741 ; total 0 ; last 0) - unlinked 10000 (time 1763345886 ; total 145 ; last 145) - unlinked 20000 (time 1763346029 ; total 288 ; last 143) - unlinked 30000 (time 1763346168 ; total 427 ; last 139) - unlinked 40000 (time 1763346304 ; total 563 ; last 136) total: 50000 unlinks in 697 seconds: 71.736008 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg301-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2