== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:04:22 (1786831462) start mds service on oleg326-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions oleg326-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg326-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg326-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg326-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 85 (time 1786831550.26 total 10.39 last 8.18) - open/close 324 (time 1786831561.02 total 21.15 last 22.22) - open/close 576 (time 1786831571.82 total 31.95 last 23.32) - open/close 854 (time 1786831581.83 total 41.96 last 27.77) - open/close 1156 (time 1786831592.49 total 52.62 last 28.34) - open/close 1391 (time 1786831602.55 total 62.68 last 23.36) - open/close 1624 (time 1786831612.59 total 72.72 last 23.21) - open/close 1905 (time 1786831622.60 total 82.73 last 28.08) - open/close 2160 (time 1786831632.60 total 92.73 last 25.50) - open/close 2378 (time 1786831642.60 total 102.73 last 21.79) - open/close 2585 (time 1786831652.61 total 112.74 last 20.68) - open/close 2799 (time 1786831663.07 total 123.20 last 20.45) - open/close 2960 (time 1786831674.03 total 134.16 last 14.69) - open/close 3133 (time 1786831684.53 total 144.66 last 16.48) - open/close 3361 (time 1786831694.53 total 154.66 last 22.80) - open/close 3625 (time 1786831705.34 total 165.47 last 24.43) - open/close 3884 (time 1786831715.34 total 175.48 last 25.89) - open/close 4016 (time 1786831726.05 total 186.18 last 12.33) - open/close 4215 (time 1786831736.77 total 196.90 last 18.57) - open/close 4456 (time 1786831749.35 total 209.48 last 19.16) - open/close 4624 (time 1786831759.38 total 219.51 last 16.74) - open/close 4763 (time 1786831769.42 total 229.55 last 13.85) - open/close 4939 (time 1786831780.27 total 240.40 last 16.22) - open/close 5093 (time 1786831790.31 total 250.44 last 15.34) - open/close 5320 (time 1786831801.27 total 261.40 last 20.72) - open/close 5530 (time 1786831812.21 total 272.34 last 19.19) - open/close 5762 (time 1786831822.21 total 282.34 last 23.20) - open/close 5990 (time 1786831832.30 total 292.43 last 22.61) - open/close 6253 (time 1786831842.30 total 302.43 last 26.29) - open/close 6439 (time 1786831852.30 total 312.43 last 18.60) - open/close 6673 (time 1786831862.31 total 322.44 last 23.38) - open/close 6959 (time 1786831872.32 total 332.45 last 28.57) - open/close 7098 (time 1786831882.60 total 342.73 last 13.52) - open/close 7282 (time 1786831892.66 total 352.79 last 18.29) - open/close 7462 (time 1786831902.66 total 362.80 last 17.99) - open/close 7613 (time 1786831912.82 total 372.95 last 14.87) - open/close 7788 (time 1786831922.93 total 383.06 last 17.31) - open/close 8015 (time 1786831934.00 total 394.14 last 20.49) - open/close 8231 (time 1786831944.80 total 404.94 last 20.00) - open/close 8472 (time 1786831954.99 total 415.12 last 23.66) - open/close 8748 (time 1786831965.68 total 425.81 last 25.83) - open/close 9099 (time 1786831975.68 total 435.81 last 35.08) - open/close 9332 (time 1786831985.95 total 446.08 last 22.68) - open/close 9537 (time 1786831996.13 total 456.27 last 20.13) - open/close 9774 (time 1786832006.14 total 466.27 last 23.70) - open/close 9873 (time 1786832016.42 total 476.55 last 9.62) - open/close 10000 (time 1786832026.28 total 486.41 last 12.88) - open/close 10097 (time 1786832036.66 total 496.79 last 9.35) - open/close 10260 (time 1786832047.52 total 507.65 last 15.00) - open/close 10445 (time 1786832057.53 total 517.66 last 18.49) - open/close 10679 (time 1786832068.02 total 528.15 last 22.30) - open/close 10833 (time 1786832078.64 total 538.77 last 14.50) - open/close 11088 (time 1786832088.65 total 548.78 last 25.47) - open/close 11323 (time 1786832098.99 total 559.12 last 22.74) - open/close 11518 (time 1786832109.99 total 570.12 last 17.73) - open/close 11721 (time 1786832120.00 total 580.13 last 20.27) - open/close 11982 (time 1786832130.53 total 590.66 last 24.79) - open/close 12182 (time 1786832141.42 total 601.55 last 18.37) - open/close 12423 (time 1786832151.44 total 611.58 last 24.04) - open/close 12693 (time 1786832162.49 total 622.62 last 24.44) - open/close 12827 (time 1786832173.14 total 633.27 last 12.59) - open/close 12950 (time 1786832183.41 total 643.54 last 11.98) - open/close 13170 (time 1786832194.22 total 654.35 last 20.34) - open/close 13366 (time 1786832205.23 total 665.36 last 17.81) - open/close 13489 (time 1786832215.26 total 675.39 last 12.26) - open/close 13728 (time 1786832225.98 total 686.11 last 22.29) - open/close 14010 (time 1786832236.00 total 696.13 last 28.15) - open/close 14281 (time 1786832246.84 total 706.97 last 25.00) - open/close 14468 (time 1786832257.41 total 717.54 last 17.69) - open/close 14697 (time 1786832267.44 total 727.57 last 22.83) - open/close 14899 (time 1786832277.55 total 737.68 last 19.98) - open/close 15176 (time 1786832287.96 total 748.09 last 26.61) - open/close 15411 (time 1786832298.55 total 758.68 last 22.20) - open/close 15577 (time 1786832308.56 total 768.69 last 16.57) - open/close 15786 (time 1786832318.57 total 778.70 last 20.89) - open/close 15993 (time 1786832329.59 total 789.72 last 18.77) - open/close 16251 (time 1786832339.62 total 799.76 last 25.72) - open/close 16498 (time 1786832350.06 total 810.19 last 23.67) - open/close 16717 (time 1786832360.06 total 820.19 last 21.90) - open/close 16999 (time 1786832370.85 total 830.98 last 26.12) - open/close 17251 (time 1786832380.86 total 840.99 last 25.18) - open/close 17431 (time 1786832391.08 total 851.21 last 17.62) - open/close 17591 (time 1786832401.51 total 861.64 last 15.33) - open/close 17834 (time 1786832411.52 total 871.65 last 24.29) - open/close 18060 (time 1786832421.52 total 881.65 last 22.59) - open/close 18213 (time 1786832431.88 total 892.01 last 14.78) - open/close 18286 (time 1786832442.86 total 902.99 last 6.65) - open/close 18502 (time 1786832453.61 total 913.74 last 20.10) - open/close 18778 (time 1786832464.42 total 924.55 last 25.53) - open/close 19024 (time 1786832475.18 total 935.31 last 22.87) - open/close 19737 (time 1786832485.18 total 945.31 last 71.28) - open/close 20000 (time 1786832488.33 total 948.46 last 83.42) - open/close 20827 (time 1786832498.34 total 958.47 last 82.62) - open/close 21526 (time 1786832508.34 total 968.48 last 69.87) - open/close 22217 (time 1786832518.36 total 978.50 last 68.96) - open/close 22889 (time 1786832528.37 total 988.50 last 67.19) - open/close 23581 (time 1786832538.38 total 998.52 last 69.07) - open/close 24309 (time 1786832548.40 total 1008.53 last 72.73) - open/close 25011 (time 1786832558.40 total 1018.53 last 70.19) - open/close 25757 (time 1786832568.40 total 1028.53 last 74.58) - open/close 26597 (time 1786832578.40 total 1038.53 last 84.00) - open/close 27369 (time 1786832588.41 total 1048.54 last 77.15) - open/close 28057 (time 1786832598.43 total 1058.56 last 68.63) - open/close 28631 (time 1786832608.44 total 1068.57 last 57.37) - open/close 29255 (time 1786832618.45 total 1078.58 last 62.34) - open/close 29863 (time 1786832628.45 total 1088.58 last 60.80) - open/close 30000 (time 1786832630.54 total 1090.67 last 65.47) - open/close 30706 (time 1786832640.55 total 1100.68 last 70.56) - open/close 31434 (time 1786832650.57 total 1110.70 last 72.61) - open/close 32141 (time 1786832660.58 total 1120.71 last 70.66) - open/close 32877 (time 1786832670.58 total 1130.71 last 73.57) - open/close 33577 (time 1786832680.58 total 1140.71 last 69.99) - open/close 34286 (time 1786832690.61 total 1150.74 last 70.72) - open/close 34903 (time 1786832700.63 total 1160.76 last 61.59) - open/close 35552 (time 1786832710.63 total 1170.76 last 64.86) - open/close 36205 (time 1786832720.64 total 1180.77 last 65.27) - open/close 37000 (time 1786832730.64 total 1190.77 last 79.44) - open/close 37778 (time 1786832740.65 total 1200.78 last 77.72) - open/close 38559 (time 1786832750.66 total 1210.79 last 78.02) - open/close 39284 (time 1786832760.67 total 1220.80 last 72.45) - open/close 39999 (time 1786832770.71 total 1230.84 last 71.23) - open/close 40740 (time 1786832780.71 total 1240.84 last 74.07) - open/close 41476 (time 1786832790.73 total 1250.86 last 73.49) - open/close 42167 (time 1786832800.73 total 1260.86 last 69.09) - open/close 42868 (time 1786832810.73 total 1270.87 last 70.05) - open/close 43559 (time 1786832820.74 total 1280.87 last 69.09) - open/close 44261 (time 1786832830.74 total 1290.87 last 70.18) - open/close 44914 (time 1786832840.74 total 1300.87 last 65.27) - open/close 45655 (time 1786832850.74 total 1310.87 last 74.10) - open/close 46465 (time 1786832860.75 total 1320.88 last 80.94) - open/close 47238 (time 1786832870.76 total 1330.89 last 77.25) - open/close 47962 (time 1786832880.76 total 1340.89 last 72.37) - open/close 48699 (time 1786832890.77 total 1350.90 last 73.64) - open/close 49381 (time 1786832900.77 total 1360.90 last 68.19) total: 50000 open/close in 1369.75 seconds: 36.50 ops/second - unlinked 0 (time 1786832914 ; total 0 ; last 0) - unlinked 10000 (time 1786833073 ; total 159 ; last 159) - unlinked 20000 (time 1786833215 ; total 301 ; last 142) - unlinked 30000 (time 1786833373 ; total 459 ; last 158) - unlinked 40000 (time 1786833512 ; total 598 ; last 139) total: 50000 unlinks in 743 seconds: 67.294754 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg326-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server unloading modules via unload_modules_local on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing unload_modules_local oleg326-server: modules unloaded. pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2