== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:22:03 (1763338923) start mds service on oleg410-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing load_modules_local oleg410-server: Loading modules from /home/green/git/lustre-release/lustre oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg410-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg410-server: detected 4 online CPUs by sysfs oleg410-server: MODOPTS_LIBCFS= oleg410-server: Force libcfs to create 2 CPU partitions oleg410-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg410-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg410-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg410-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg410-server: oleg410-server.virtnet: executing set_default_debug -1 all pdsh@oleg410-client: oleg410-server: ssh exited with exit code 1 Started lustre-OST0000 oleg410-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg410-client.virtnet: -o user_xattr,flock 192.168.204.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 213 (time 1763338997.88 total 10.01 last 21.28) - open/close 443 (time 1763339008.40 total 20.52 last 21.87) - open/close 700 (time 1763339018.84 total 30.97 last 24.61) - open/close 919 (time 1763339029.08 total 41.21 last 21.38) - open/close 1135 (time 1763339039.43 total 51.56 last 20.87) - open/close 1308 (time 1763339049.63 total 61.75 last 16.97) - open/close 1522 (time 1763339059.92 total 72.05 last 20.79) - open/close 1703 (time 1763339070.35 total 82.48 last 17.36) - open/close 1907 (time 1763339081.17 total 93.30 last 18.85) - open/close 2063 (time 1763339091.78 total 103.91 last 14.70) - open/close 2243 (time 1763339102.16 total 114.28 last 17.35) - open/close 2431 (time 1763339112.66 total 124.78 last 17.91) - open/close 2650 (time 1763339122.83 total 134.96 last 21.52) - open/close 2823 (time 1763339132.83 total 144.96 last 17.30) - open/close 2993 (time 1763339142.85 total 154.98 last 16.96) - open/close 3177 (time 1763339153.49 total 165.62 last 17.29) - open/close 3363 (time 1763339163.51 total 175.64 last 18.57) - open/close 3572 (time 1763339173.53 total 185.65 last 20.87) - open/close 3749 (time 1763339183.54 total 195.66 last 17.68) - open/close 3962 (time 1763339193.54 total 205.67 last 21.30) - open/close 4162 (time 1763339204.44 total 216.57 last 18.34) - open/close 4360 (time 1763339214.46 total 226.58 last 19.77) - open/close 4554 (time 1763339224.52 total 236.65 last 19.28) - open/close 4756 (time 1763339234.54 total 246.66 last 20.17) - open/close 5010 (time 1763339244.77 total 256.90 last 24.81) - open/close 5193 (time 1763339254.78 total 266.91 last 18.28) - open/close 5477 (time 1763339264.78 total 276.91 last 28.40) - open/close 5666 (time 1763339274.91 total 287.04 last 18.67) - open/close 5866 (time 1763339285.66 total 297.78 last 18.61) - open/close 6094 (time 1763339297.05 total 309.17 last 20.02) - open/close 6333 (time 1763339307.67 total 319.80 last 22.50) - open/close 6577 (time 1763339318.48 total 330.60 last 22.58) - open/close 6804 (time 1763339328.54 total 340.67 last 22.55) - open/close 6993 (time 1763339339.31 total 351.44 last 17.56) - open/close 7248 (time 1763339350.28 total 362.41 last 23.24) - open/close 7433 (time 1763339360.92 total 373.05 last 17.38) - open/close 7630 (time 1763339371.67 total 383.80 last 18.33) - open/close 7861 (time 1763339382.09 total 394.22 last 22.16) - open/close 8026 (time 1763339393.11 total 405.23 last 14.98) - open/close 8239 (time 1763339403.12 total 415.24 last 21.28) - open/close 8402 (time 1763339413.12 total 425.25 last 16.29) - open/close 8657 (time 1763339423.89 total 436.01 last 23.68) - open/close 8815 (time 1763339435.59 total 447.72 last 13.50) - open/close 9000 (time 1763339445.60 total 457.73 last 18.49) - open/close 9182 (time 1763339456.09 total 468.21 last 17.36) - open/close 9322 (time 1763339466.58 total 478.70 last 13.35) - open/close 9455 (time 1763339476.81 total 488.93 last 13.00) - open/close 9616 (time 1763339486.84 total 498.96 last 16.05) - open/close 9793 (time 1763339496.85 total 508.98 last 17.67) - open/close 10000 (time 1763339506.68 total 518.81 last 21.06) - open/close 10186 (time 1763339517.13 total 529.25 last 17.81) - open/close 10356 (time 1763339527.96 total 540.09 last 15.68) - open/close 10573 (time 1763339538.14 total 550.26 last 21.34) - open/close 10814 (time 1763339548.16 total 560.28 last 24.05) - open/close 11012 (time 1763339558.16 total 570.29 last 19.79) - open/close 11198 (time 1763339568.66 total 580.79 last 17.71) - open/close 11385 (time 1763339580.64 total 592.76 last 15.62) - open/close 11587 (time 1763339590.66 total 602.79 last 20.15) - open/close 11802 (time 1763339601.50 total 613.63 last 19.84) - open/close 12038 (time 1763339611.55 total 623.67 last 23.49) - open/close 12248 (time 1763339621.72 total 633.85 last 20.63) - open/close 12493 (time 1763339632.67 total 644.79 last 22.39) - open/close 12695 (time 1763339643.28 total 655.40 last 19.04) - open/close 12843 (time 1763339653.67 total 665.79 last 14.25) - open/close 12965 (time 1763339663.67 total 675.80 last 12.19) - open/close 13162 (time 1763339674.19 total 686.32 last 18.73) - open/close 13422 (time 1763339684.20 total 696.32 last 25.99) - open/close 13626 (time 1763339694.50 total 706.62 last 19.80) - open/close 13843 (time 1763339704.65 total 716.78 last 21.37) - open/close 14073 (time 1763339715.42 total 727.55 last 21.36) - open/close 14233 (time 1763339725.72 total 737.85 last 15.53) - open/close 14428 (time 1763339736.01 total 748.14 last 18.95) - open/close 14610 (time 1763339746.03 total 758.16 last 18.16) - open/close 14789 (time 1763339756.04 total 768.17 last 17.88) - open/close 15010 (time 1763339766.29 total 778.42 last 21.56) - open/close 15236 (time 1763339776.53 total 788.65 last 22.09) - open/close 15407 (time 1763339786.55 total 798.68 last 17.06) - open/close 15951 (time 1763339796.55 total 808.68 last 54.38) - open/close 16667 (time 1763339806.56 total 818.69 last 71.55) - open/close 17464 (time 1763339816.56 total 828.69 last 79.69) - open/close 18187 (time 1763339826.57 total 838.69 last 72.27) - open/close 18900 (time 1763339836.59 total 848.71 last 71.17) - open/close 19596 (time 1763339846.59 total 858.71 last 69.58) - open/close 20000 (time 1763339852.31 total 864.44 last 70.60) - open/close 20683 (time 1763339862.31 total 874.44 last 68.30) - open/close 21376 (time 1763339872.32 total 884.44 last 69.26) - open/close 22049 (time 1763339882.33 total 894.45 last 67.23) - open/close 22747 (time 1763339892.33 total 904.46 last 69.76) - open/close 23412 (time 1763339902.35 total 914.47 last 66.39) - open/close 24125 (time 1763339912.36 total 924.49 last 71.21) - open/close 24801 (time 1763339922.37 total 934.50 last 67.53) - open/close 25527 (time 1763339932.37 total 944.50 last 72.58) - open/close 26210 (time 1763339942.38 total 954.50 last 68.28) - open/close 26924 (time 1763339952.39 total 964.51 last 71.32) - open/close 27597 (time 1763339962.39 total 974.52 last 67.27) - open/close 28298 (time 1763339972.40 total 984.52 last 70.06) - open/close 29049 (time 1763339982.41 total 994.53 last 75.03) - open/close 29736 (time 1763339992.43 total 1004.55 last 68.58) - open/close 30000 (time 1763339996.28 total 1008.41 last 68.41) - open/close 30635 (time 1763340006.28 total 1018.41 last 63.50) - open/close 31358 (time 1763340016.30 total 1028.43 last 72.20) - open/close 32096 (time 1763340026.30 total 1038.43 last 73.78) - open/close 32841 (time 1763340036.30 total 1048.43 last 74.48) - open/close 33575 (time 1763340046.32 total 1058.44 last 73.30) - open/close 34262 (time 1763340056.32 total 1068.45 last 68.66) - open/close 34961 (time 1763340066.33 total 1078.45 last 69.88) - open/close 35641 (time 1763340076.34 total 1088.46 last 67.92) - open/close 36284 (time 1763340086.34 total 1098.47 last 64.27) - open/close 36863 (time 1763340096.36 total 1108.48 last 57.82) - open/close 37498 (time 1763340106.37 total 1118.50 last 63.41) - open/close 38103 (time 1763340116.37 total 1128.50 last 60.48) - open/close 38790 (time 1763340126.39 total 1138.51 last 68.61) - open/close 39457 (time 1763340136.39 total 1148.51 last 66.70) - open/close 40000 (time 1763340143.73 total 1155.85 last 73.99) - open/close 40672 (time 1763340153.74 total 1165.87 last 67.10) - open/close 41368 (time 1763340163.75 total 1175.88 last 69.54) - open/close 42041 (time 1763340173.76 total 1185.89 last 67.23) - open/close 42736 (time 1763340183.76 total 1195.89 last 69.48) - open/close 43449 (time 1763340193.77 total 1205.90 last 71.23) - open/close 44168 (time 1763340203.79 total 1215.91 last 71.80) - open/close 44941 (time 1763340213.80 total 1225.92 last 77.24) - open/close 45512 (time 1763340223.80 total 1235.93 last 57.05) - open/close 46126 (time 1763340233.82 total 1245.95 last 61.29) - open/close 46801 (time 1763340243.83 total 1255.95 last 67.47) - open/close 47492 (time 1763340253.84 total 1265.96 last 69.03) - open/close 48167 (time 1763340263.84 total 1275.97 last 67.49) - open/close 48859 (time 1763340273.85 total 1285.97 last 69.14) - open/close 49546 (time 1763340283.86 total 1295.98 last 68.64) total: 50000 open/close in 1302.46 seconds: 38.39 ops/second - unlinked 0 (time 1763340294 ; total 0 ; last 0) - unlinked 10000 (time 1763340439 ; total 145 ; last 145) - unlinked 20000 (time 1763340591 ; total 297 ; last 152) - unlinked 30000 (time 1763340738 ; total 444 ; last 147) - unlinked 40000 (time 1763340889 ; total 595 ; last 151) total: 50000 unlinks in 744 seconds: 67.204300 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg410-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg410-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg410-server stop mds service on oleg410-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg410-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg410-server' oleg410-server: oleg410-server.virtnet: executing unload_modules_local oleg410-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg410-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-client: ssh exited with exit code 2 pdsh@oleg410-client: oleg410-server: ssh exited with exit code 2