== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:24:42 (1786829082) start mds service on oleg248-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions oleg248-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg248-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg248-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg248-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 192 (time 1786829170.15 total 10.07 last 19.06) - open/close 405 (time 1786829180.51 total 20.44 last 20.55) - open/close 595 (time 1786829190.52 total 30.44 last 18.99) - open/close 785 (time 1786829201.01 total 40.93 last 18.10) - open/close 964 (time 1786829213.68 total 53.60 last 14.13) - open/close 1137 (time 1786829223.70 total 63.62 last 17.27) - open/close 1229 (time 1786829235.75 total 75.67 last 7.63) - open/close 1348 (time 1786829246.06 total 85.98 last 11.55) - open/close 1548 (time 1786829256.08 total 96.00 last 19.97) - open/close 1787 (time 1786829266.34 total 106.27 last 23.27) - open/close 2012 (time 1786829276.35 total 116.27 last 22.49) - open/close 2233 (time 1786829286.54 total 126.46 last 21.68) - open/close 2357 (time 1786829296.91 total 136.83 last 11.96) - open/close 2542 (time 1786829306.91 total 146.83 last 18.49) - open/close 2759 (time 1786829316.94 total 156.86 last 21.64) - open/close 3024 (time 1786829327.85 total 167.77 last 24.29) - open/close 3245 (time 1786829337.86 total 177.78 last 22.08) - open/close 3416 (time 1786829348.79 total 188.72 last 15.64) - open/close 3481 (time 1786829358.98 total 198.90 last 6.38) - open/close 3638 (time 1786829369.39 total 209.31 last 15.08) - open/close 3856 (time 1786829379.40 total 219.32 last 21.78) - open/close 3988 (time 1786829389.45 total 229.37 last 13.13) - open/close 4163 (time 1786829399.47 total 239.39 last 17.47) - open/close 4351 (time 1786829409.49 total 249.41 last 18.76) - open/close 4564 (time 1786829419.57 total 259.49 last 21.14) - open/close 4793 (time 1786829429.88 total 269.80 last 22.22) - open/close 4976 (time 1786829440.91 total 280.83 last 16.58) - open/close 5150 (time 1786829451.06 total 290.98 last 17.14) - open/close 5406 (time 1786829461.08 total 301.00 last 25.56) - open/close 5492 (time 1786829472.07 total 312.00 last 7.82) - open/close 5651 (time 1786829482.09 total 322.01 last 15.87) - open/close 5818 (time 1786829492.91 total 332.83 last 15.43) - open/close 6023 (time 1786829503.27 total 343.19 last 19.79) - open/close 6253 (time 1786829516.33 total 356.25 last 17.62) - open/close 6490 (time 1786829527.33 total 367.25 last 21.54) - open/close 6716 (time 1786829537.52 total 377.44 last 22.19) - open/close 6864 (time 1786829547.56 total 387.48 last 14.74) - open/close 7007 (time 1786829557.57 total 397.49 last 14.29) - open/close 7255 (time 1786829567.66 total 407.58 last 24.58) - open/close 7446 (time 1786829577.68 total 417.60 last 19.06) - open/close 7648 (time 1786829587.69 total 427.61 last 20.17) - open/close 7843 (time 1786829597.75 total 437.67 last 19.38) - open/close 8091 (time 1786829608.61 total 448.53 last 22.83) - open/close 8252 (time 1786829619.49 total 459.41 last 14.80) - open/close 8490 (time 1786829629.63 total 469.55 last 23.48) - open/close 8784 (time 1786829639.65 total 479.57 last 29.35) - open/close 8936 (time 1786829650.74 total 490.66 last 13.70) - open/close 9067 (time 1786829660.76 total 500.68 last 13.08) - open/close 9229 (time 1786829671.43 total 511.35 last 15.18) - open/close 9360 (time 1786829681.97 total 521.89 last 12.43) - open/close 9582 (time 1786829692.01 total 531.93 last 22.10) - open/close 9820 (time 1786829702.06 total 541.98 last 23.70) - open/close 10000 (time 1786829709.42 total 549.34 last 24.46) - open/close 10227 (time 1786829719.42 total 559.34 last 22.69) - open/close 10470 (time 1786829729.76 total 569.68 last 23.50) - open/close 10662 (time 1786829740.45 total 580.37 last 17.96) - open/close 10891 (time 1786829750.64 total 590.56 last 22.47) - open/close 11083 (time 1786829760.66 total 600.58 last 19.17) - open/close 11300 (time 1786829770.67 total 610.60 last 21.67) - open/close 11463 (time 1786829781.00 total 620.92 last 15.78) - open/close 11744 (time 1786829793.17 total 633.09 last 23.10) - open/close 11954 (time 1786829803.19 total 643.11 last 20.95) - open/close 12209 (time 1786829813.21 total 653.13 last 25.45) - open/close 12317 (time 1786829824.25 total 664.17 last 9.78) - open/close 12441 (time 1786829836.71 total 676.63 last 9.95) - open/close 12621 (time 1786829846.89 total 686.81 last 17.68) - open/close 12887 (time 1786829857.19 total 697.11 last 25.84) - open/close 13073 (time 1786829867.85 total 707.77 last 17.45) - open/close 13307 (time 1786829877.85 total 717.78 last 23.38) - open/close 13512 (time 1786829887.88 total 727.80 last 20.45) - open/close 13630 (time 1786829900.78 total 740.70 last 9.15) - open/close 13822 (time 1786829911.10 total 751.02 last 18.60) - open/close 13919 (time 1786829921.10 total 761.02 last 9.70) - open/close 14121 (time 1786829931.11 total 771.03 last 20.18) - open/close 14364 (time 1786829941.71 total 781.63 last 22.92) - open/close 14517 (time 1786829951.75 total 791.68 last 15.23) - open/close 14637 (time 1786829962.12 total 802.04 last 11.58) - open/close 14799 (time 1786829972.98 total 812.90 last 14.92) - open/close 15024 (time 1786829982.98 total 822.90 last 22.50) - open/close 15255 (time 1786829993.64 total 833.56 last 21.67) - open/close 15514 (time 1786830003.76 total 843.68 last 25.58) - open/close 15605 (time 1786830015.57 total 855.50 last 7.70) - open/close 15842 (time 1786830025.58 total 865.50 last 23.69) - open/close 15939 (time 1786830038.58 total 878.50 last 7.46) - open/close 16176 (time 1786830051.07 total 890.99 last 18.97) - open/close 16432 (time 1786830062.06 total 901.98 last 23.30) - open/close 16649 (time 1786830072.12 total 912.04 last 21.57) - open/close 16927 (time 1786830082.93 total 922.85 last 25.71) - open/close 17160 (time 1786830093.49 total 933.41 last 22.06) - open/close 17379 (time 1786830104.35 total 944.28 last 20.16) - open/close 17562 (time 1786830115.40 total 955.32 last 16.57) - open/close 18179 (time 1786830125.40 total 965.32 last 61.67) - open/close 18802 (time 1786830135.41 total 975.33 last 62.26) - open/close 19456 (time 1786830145.42 total 985.34 last 65.33) - open/close 20000 (time 1786830154.55 total 994.47 last 59.59) - open/close 20642 (time 1786830164.55 total 1004.47 last 64.19) - open/close 21351 (time 1786830174.56 total 1014.48 last 70.82) - open/close 22150 (time 1786830184.57 total 1024.49 last 79.82) - open/close 22891 (time 1786830194.57 total 1034.49 last 74.09) - open/close 23496 (time 1786830204.59 total 1044.52 last 60.38) - open/close 24049 (time 1786830214.60 total 1054.53 last 55.24) - open/close 24720 (time 1786830224.61 total 1064.53 last 67.08) - open/close 25419 (time 1786830234.63 total 1074.55 last 69.77) - open/close 26102 (time 1786830244.63 total 1084.55 last 68.29) - open/close 26871 (time 1786830254.63 total 1094.55 last 76.89) - open/close 27684 (time 1786830264.63 total 1104.56 last 81.26) - open/close 28411 (time 1786830274.66 total 1114.58 last 72.52) - open/close 29122 (time 1786830284.66 total 1124.59 last 71.06) - open/close 29741 (time 1786830294.67 total 1134.59 last 61.85) - open/close 30000 (time 1786830299.49 total 1139.41 last 53.82) - open/close 30692 (time 1786830309.50 total 1149.42 last 69.12) - open/close 31395 (time 1786830319.50 total 1159.42 last 70.25) - open/close 32162 (time 1786830329.50 total 1169.43 last 76.69) - open/close 32847 (time 1786830339.52 total 1179.44 last 68.41) - open/close 33574 (time 1786830349.55 total 1189.47 last 72.50) - open/close 34295 (time 1786830359.56 total 1199.48 last 72.03) - open/close 34965 (time 1786830369.58 total 1209.50 last 66.85) - open/close 35647 (time 1786830379.59 total 1219.51 last 68.13) - open/close 36356 (time 1786830389.60 total 1229.52 last 70.80) - open/close 37042 (time 1786830399.60 total 1239.52 last 68.59) - open/close 37501 (time 1786830409.61 total 1249.53 last 45.86) - open/close 38087 (time 1786830419.62 total 1259.54 last 58.58) - open/close 38677 (time 1786830429.62 total 1269.54 last 58.99) - open/close 39148 (time 1786830439.63 total 1279.55 last 47.02) - open/close 39783 (time 1786830449.64 total 1289.56 last 63.43) - open/close 40000 (time 1786830452.54 total 1292.46 last 75.01) - open/close 40605 (time 1786830462.54 total 1302.46 last 60.50) - open/close 41349 (time 1786830472.54 total 1312.46 last 74.37) - open/close 41950 (time 1786830482.56 total 1322.48 last 60.00) - open/close 42448 (time 1786830492.57 total 1332.49 last 49.72) - open/close 43026 (time 1786830502.58 total 1342.50 last 57.75) - open/close 43734 (time 1786830512.60 total 1352.52 last 70.68) - open/close 44479 (time 1786830522.60 total 1362.52 last 74.49) - open/close 45292 (time 1786830532.61 total 1372.54 last 81.19) - open/close 46090 (time 1786830542.62 total 1382.54 last 79.74) - open/close 46895 (time 1786830552.63 total 1392.56 last 80.40) - open/close 47666 (time 1786830562.64 total 1402.56 last 77.06) - open/close 48369 (time 1786830572.64 total 1412.57 last 70.27) - open/close 48945 (time 1786830582.65 total 1422.57 last 57.59) - open/close 49684 (time 1786830592.66 total 1432.58 last 73.83) total: 50000 open/close in 1436.28 seconds: 34.81 ops/second - unlinked 0 (time 1786830601 ; total 0 ; last 0) - unlinked 10000 (time 1786830748 ; total 147 ; last 147) - unlinked 20000 (time 1786830889 ; total 288 ; last 141) - unlinked 30000 (time 1786831032 ; total 431 ; last 143) - unlinked 40000 (time 1786831171 ; total 570 ; last 139) total: 50000 unlinks in 701 seconds: 71.326675 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server unloading modules via unload_modules_local on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local oleg248-server: modules unloaded. pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2