== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:05:47 (1787832347) start mds service on oleg128-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg128-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg128-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1787832438.82 total 10.54 last 22.20) - open/close 419 (time 1787832448.83 total 20.55 last 18.49) - open/close 614 (time 1787832458.92 total 30.64 last 19.32) - open/close 884 (time 1787832469.99 total 41.71 last 24.38) - open/close 1107 (time 1787832480.01 total 51.72 last 22.27) - open/close 1286 (time 1787832490.15 total 61.87 last 17.64) - open/close 1453 (time 1787832500.71 total 72.43 last 15.82) - open/close 1634 (time 1787832511.21 total 82.92 last 17.25) - open/close 1759 (time 1787832521.21 total 92.93 last 12.49) - open/close 1954 (time 1787832531.22 total 102.94 last 19.49) - open/close 2117 (time 1787832541.23 total 112.95 last 16.28) - open/close 2254 (time 1787832551.98 total 123.69 last 12.75) - open/close 2399 (time 1787832561.98 total 133.69 last 14.50) - open/close 2529 (time 1787832571.98 total 143.70 last 13.00) - open/close 2588 (time 1787832582.91 total 154.63 last 5.40) - open/close 2683 (time 1787832592.94 total 164.66 last 9.47) - open/close 2814 (time 1787832603.38 total 175.10 last 12.55) - open/close 3015 (time 1787832613.41 total 185.13 last 20.05) - open/close 3119 (time 1787832624.11 total 195.83 last 9.72) - open/close 3288 (time 1787832634.91 total 206.63 last 15.65) - open/close 3425 (time 1787832645.16 total 216.87 last 13.38) - open/close 3554 (time 1787832655.61 total 227.33 last 12.34) - open/close 3731 (time 1787832665.63 total 237.35 last 17.67) - open/close 3860 (time 1787832679.90 total 251.62 last 9.04) - open/close 3936 (time 1787832690.19 total 261.90 last 7.39) - open/close 4092 (time 1787832701.22 total 272.94 last 14.14) - open/close 4269 (time 1787832711.55 total 283.27 last 17.13) - open/close 4368 (time 1787832721.62 total 293.33 last 9.83) - open/close 4449 (time 1787832732.26 total 303.98 last 7.61) - open/close 4651 (time 1787832742.85 total 314.57 last 19.07) - open/close 4813 (time 1787832752.86 total 324.58 last 16.19) - open/close 5038 (time 1787832763.44 total 335.16 last 21.26) - open/close 5241 (time 1787832773.45 total 345.17 last 20.29) - open/close 5466 (time 1787832784.21 total 355.93 last 20.91) - open/close 5661 (time 1787832794.40 total 366.12 last 19.13) - open/close 5939 (time 1787832804.43 total 376.15 last 27.72) - open/close 6159 (time 1787832814.44 total 386.15 last 21.99) - open/close 6310 (time 1787832824.45 total 396.16 last 15.09) - open/close 6485 (time 1787832834.47 total 406.19 last 17.45) - open/close 6680 (time 1787832844.48 total 416.20 last 19.48) - open/close 6950 (time 1787832854.49 total 426.21 last 26.98) - open/close 7132 (time 1787832864.82 total 436.54 last 17.62) - open/close 7247 (time 1787832875.32 total 447.04 last 10.95) - open/close 7431 (time 1787832885.33 total 457.05 last 18.38) - open/close 7659 (time 1787832896.05 total 467.77 last 21.28) - open/close 7917 (time 1787832906.13 total 477.85 last 25.59) - open/close 8159 (time 1787832916.15 total 487.86 last 24.16) - open/close 8438 (time 1787832926.15 total 497.86 last 27.89) - open/close 8669 (time 1787832937.01 total 508.72 last 21.27) - open/close 8911 (time 1787832947.88 total 519.59 last 22.27) - open/close 9056 (time 1787832957.88 total 529.59 last 14.50) - open/close 9315 (time 1787832968.43 total 540.15 last 24.53) - open/close 9446 (time 1787832979.34 total 551.05 last 12.01) - open/close 9632 (time 1787832989.68 total 561.40 last 17.98) - open/close 9786 (time 1787833000.24 total 571.96 last 14.59) - open/close 9840 (time 1787833010.25 total 581.96 last 5.40) - open/close 9875 (time 1787833020.53 total 592.25 last 3.40) - open/close 10000 (time 1787833026.30 total 598.01 last 21.67) - open/close 10225 (time 1787833036.34 total 608.06 last 22.40) - open/close 10456 (time 1787833046.92 total 618.63 last 21.84) - open/close 10636 (time 1787833057.65 total 629.36 last 16.78) - open/close 10835 (time 1787833067.65 total 639.36 last 19.89) - open/close 10952 (time 1787833077.66 total 649.37 last 11.69) - open/close 11192 (time 1787833088.23 total 659.95 last 22.70) - open/close 11390 (time 1787833100.13 total 671.85 last 16.63) - open/close 11544 (time 1787833110.51 total 682.23 last 14.84) - open/close 11574 (time 1787833120.82 total 692.53 last 2.91) - open/close 11690 (time 1787833131.77 total 703.48 last 10.59) - open/close 11747 (time 1787833142.31 total 714.02 last 5.41) - open/close 11863 (time 1787833152.69 total 724.41 last 11.17) - open/close 11966 (time 1787833163.78 total 735.50 last 9.29) - open/close 12058 (time 1787833174.21 total 745.93 last 8.82) - open/close 12232 (time 1787833184.53 total 756.24 last 16.87) - open/close 12391 (time 1787833195.51 total 767.23 last 14.47) - open/close 12598 (time 1787833205.53 total 777.25 last 20.66) - open/close 12776 (time 1787833216.58 total 788.29 last 16.12) - open/close 13035 (time 1787833226.87 total 798.58 last 25.17) - open/close 13238 (time 1787833236.89 total 808.60 last 20.26) - open/close 13452 (time 1787833247.47 total 819.18 last 20.23) - open/close 13689 (time 1787833257.72 total 829.44 last 23.11) - open/close 13895 (time 1787833268.87 total 840.58 last 18.49) - open/close 14063 (time 1787833279.75 total 851.47 last 15.43) - open/close 14243 (time 1787833290.80 total 862.52 last 16.29) - open/close 14459 (time 1787833301.51 total 873.23 last 20.17) - open/close 14709 (time 1787833311.52 total 883.23 last 24.98) - open/close 14922 (time 1787833322.03 total 893.75 last 20.26) - open/close 15126 (time 1787833332.66 total 904.38 last 19.19) - open/close 15392 (time 1787833343.08 total 914.80 last 25.53) - open/close 15571 (time 1787833354.04 total 925.75 last 16.33) - open/close 15795 (time 1787833364.05 total 935.76 last 22.38) - open/close 15948 (time 1787833374.07 total 945.78 last 15.27) - open/close 16201 (time 1787833384.51 total 956.23 last 24.23) - open/close 16481 (time 1787833395.53 total 967.24 last 25.41) - open/close 16717 (time 1787833405.53 total 977.25 last 23.59) - open/close 16904 (time 1787833416.45 total 988.17 last 17.12) - open/close 16949 (time 1787833426.60 total 998.31 last 4.44) - open/close 17136 (time 1787833437.91 total 1009.63 last 16.53) - open/close 17253 (time 1787833447.92 total 1019.63 last 11.69) - open/close 17368 (time 1787833458.35 total 1030.06 last 11.03) - open/close 17853 (time 1787833468.36 total 1040.07 last 48.45) - open/close 18385 (time 1787833478.36 total 1050.07 last 53.19) - open/close 19107 (time 1787833488.37 total 1060.09 last 72.09) - open/close 19791 (time 1787833498.38 total 1070.10 last 68.32) - open/close 20000 (time 1787833501.38 total 1073.10 last 69.74) - open/close 20756 (time 1787833511.38 total 1083.10 last 75.58) - open/close 21519 (time 1787833521.39 total 1093.11 last 76.25) - open/close 22315 (time 1787833531.40 total 1103.11 last 79.55) - open/close 23033 (time 1787833541.40 total 1113.12 last 71.75) - open/close 23791 (time 1787833551.41 total 1123.13 last 75.74) - open/close 24546 (time 1787833561.43 total 1133.14 last 75.39) - open/close 25270 (time 1787833571.43 total 1143.15 last 72.37) - open/close 25929 (time 1787833581.43 total 1153.15 last 65.88) - open/close 26603 (time 1787833591.44 total 1163.16 last 67.35) - open/close 27303 (time 1787833601.45 total 1173.17 last 69.93) - open/close 27944 (time 1787833611.46 total 1183.18 last 64.01) - open/close 28647 (time 1787833621.47 total 1193.19 last 70.25) - open/close 29469 (time 1787833631.48 total 1203.19 last 82.16) - open/close 30000 (time 1787833637.63 total 1209.35 last 86.23) - open/close 30804 (time 1787833647.65 total 1219.36 last 80.29) - open/close 31600 (time 1787833657.65 total 1229.36 last 79.60) - open/close 32419 (time 1787833667.66 total 1239.37 last 81.82) - open/close 33155 (time 1787833677.67 total 1249.38 last 73.54) - open/close 33991 (time 1787833687.67 total 1259.39 last 83.52) - open/close 34670 (time 1787833697.67 total 1269.39 last 67.90) - open/close 35275 (time 1787833707.69 total 1279.40 last 60.43) - open/close 35999 (time 1787833717.69 total 1289.40 last 72.39) - open/close 36674 (time 1787833727.69 total 1299.40 last 67.50) - open/close 37325 (time 1787833737.70 total 1309.42 last 64.99) - open/close 37975 (time 1787833747.71 total 1319.42 last 64.97) - open/close 38558 (time 1787833757.71 total 1329.43 last 58.26) - open/close 39415 (time 1787833767.72 total 1339.44 last 85.66) - open/close 40000 (time 1787833775.01 total 1346.72 last 80.29) - open/close 40709 (time 1787833785.05 total 1356.76 last 70.62) - open/close 41422 (time 1787833795.07 total 1366.78 last 71.14) - open/close 42189 (time 1787833805.07 total 1376.78 last 76.70) - open/close 42921 (time 1787833815.07 total 1386.79 last 73.17) - open/close 43685 (time 1787833825.09 total 1396.80 last 76.30) - open/close 44487 (time 1787833835.09 total 1406.81 last 80.16) - open/close 45210 (time 1787833845.10 total 1416.81 last 72.26) - open/close 45995 (time 1787833855.10 total 1426.82 last 78.46) - open/close 46707 (time 1787833865.10 total 1436.82 last 71.18) - open/close 47477 (time 1787833875.11 total 1446.82 last 76.98) - open/close 48183 (time 1787833885.11 total 1456.82 last 70.59) - open/close 48984 (time 1787833895.12 total 1466.84 last 79.99) - open/close 49732 (time 1787833905.13 total 1476.84 last 74.75) total: 50000 open/close in 1479.98 seconds: 33.78 ops/second - unlinked 0 (time 1787833913 ; total 0 ; last 0) - unlinked 10000 (time 1787834057 ; total 144 ; last 144) - unlinked 20000 (time 1787834203 ; total 290 ; last 146) - unlinked 30000 (time 1787834349 ; total 436 ; last 146) - unlinked 40000 (time 1787834492 ; total 579 ; last 143) total: 50000 unlinks in 719 seconds: 69.541031 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server unloading modules via unload_modules_local on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing unload_modules_local oleg128-server: modules unloaded. pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2