== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:21:08 (1786033268) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg152-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 113 (time 1786033370.43 total 10.92 last 10.35) - open/close 231 (time 1786033380.44 total 20.92 last 11.79) - open/close 345 (time 1786033391.37 total 31.85 last 10.43) - open/close 438 (time 1786033403.80 total 44.29 last 7.48) - open/close 516 (time 1786033416.02 total 56.50 last 6.39) - open/close 712 (time 1786033426.11 total 66.60 last 19.42) - open/close 838 (time 1786033436.42 total 76.90 last 12.23) - open/close 939 (time 1786033446.43 total 86.91 last 10.09) - open/close 1029 (time 1786033456.45 total 96.94 last 8.98) - open/close 1217 (time 1786033466.48 total 106.96 last 18.75) - open/close 1349 (time 1786033477.26 total 117.75 last 12.24) - open/close 1433 (time 1786033487.70 total 128.19 last 8.05) - open/close 1653 (time 1786033497.86 total 138.35 last 21.66) - open/close 1879 (time 1786033507.87 total 148.35 last 22.59) - open/close 2058 (time 1786033518.15 total 158.63 last 17.41) - open/close 2204 (time 1786033528.15 total 168.63 last 14.60) - open/close 2389 (time 1786033538.76 total 179.25 last 17.43) - open/close 2467 (time 1786033548.89 total 189.38 last 7.70) - open/close 2635 (time 1786033558.89 total 199.38 last 16.80) - open/close 2859 (time 1786033568.91 total 209.40 last 22.36) - open/close 3075 (time 1786033578.92 total 219.40 last 21.58) - open/close 3249 (time 1786033588.94 total 229.42 last 17.37) - open/close 3406 (time 1786033599.12 total 239.61 last 15.42) - open/close 3580 (time 1786033609.14 total 249.63 last 17.37) - open/close 3755 (time 1786033621.45 total 261.93 last 14.22) - open/close 3944 (time 1786033631.46 total 271.95 last 18.88) - open/close 4105 (time 1786033642.11 total 282.59 last 15.12) - open/close 4337 (time 1786033652.14 total 292.63 last 23.12) - open/close 4420 (time 1786033662.40 total 302.89 last 8.09) - open/close 4584 (time 1786033672.42 total 312.91 last 16.36) - open/close 4716 (time 1786033683.14 total 323.63 last 12.31) - open/close 4794 (time 1786033693.46 total 333.95 last 7.56) - open/close 4924 (time 1786033704.08 total 344.56 last 12.24) - open/close 5098 (time 1786033714.69 total 355.18 last 16.40) - open/close 5217 (time 1786033725.08 total 365.56 last 11.46) - open/close 5292 (time 1786033735.44 total 375.92 last 7.24) - open/close 5491 (time 1786033745.45 total 385.93 last 19.88) - open/close 5625 (time 1786033756.23 total 396.72 last 12.43) - open/close 5735 (time 1786033766.40 total 406.89 last 10.82) - open/close 5859 (time 1786033777.12 total 417.60 last 11.57) - open/close 5977 (time 1786033787.55 total 428.04 last 11.31) - open/close 6082 (time 1786033798.01 total 438.50 last 10.04) - open/close 6226 (time 1786033808.03 total 448.51 last 14.38) - open/close 6382 (time 1786033818.06 total 458.54 last 15.55) - open/close 6471 (time 1786033828.87 total 469.36 last 8.23) - open/close 6661 (time 1786033839.13 total 479.62 last 18.52) - open/close 6800 (time 1786033849.13 total 489.62 last 13.90) - open/close 6907 (time 1786033859.24 total 499.73 last 10.59) - open/close 7158 (time 1786033869.26 total 509.74 last 25.06) - open/close 7406 (time 1786033879.63 total 520.12 last 23.90) - open/close 7582 (time 1786033890.18 total 530.67 last 16.68) - open/close 7763 (time 1786033901.02 total 541.51 last 16.70) - open/close 7893 (time 1786033911.32 total 551.80 last 12.63) - open/close 7999 (time 1786033921.94 total 562.42 last 9.98) - open/close 8084 (time 1786033933.01 total 573.49 last 7.68) - open/close 8206 (time 1786033943.03 total 583.52 last 12.17) - open/close 8334 (time 1786033953.04 total 593.52 last 12.79) - open/close 8498 (time 1786033964.10 total 604.58 last 14.82) - open/close 8656 (time 1786033974.42 total 614.90 last 15.31) - open/close 8744 (time 1786033985.05 total 625.54 last 8.27) - open/close 8882 (time 1786033995.06 total 635.54 last 13.79) - open/close 8983 (time 1786034005.86 total 646.35 last 9.35) - open/close 9135 (time 1786034016.40 total 656.88 last 14.43) - open/close 9249 (time 1786034026.40 total 666.88 last 11.40) - open/close 9407 (time 1786034037.03 total 677.51 last 14.86) - open/close 9607 (time 1786034047.95 total 688.43 last 18.32) - open/close 9843 (time 1786034057.97 total 698.45 last 23.56) - open/close 10000 (time 1786034066.21 total 706.70 last 19.04) - open/close 10079 (time 1786034076.36 total 716.84 last 7.79) - open/close 10246 (time 1786034087.26 total 727.74 last 15.32) - open/close 10471 (time 1786034097.27 total 737.76 last 22.47) - open/close 10709 (time 1786034107.28 total 747.77 last 23.77) - open/close 10886 (time 1786034117.30 total 757.78 last 17.67) - open/close 11095 (time 1786034127.61 total 768.10 last 20.27) - open/close 11351 (time 1786034137.72 total 778.21 last 25.32) - open/close 11590 (time 1786034150.61 total 791.10 last 18.54) - open/close 11726 (time 1786034160.90 total 801.38 last 13.22) - open/close 11919 (time 1786034170.91 total 811.40 last 19.27) - open/close 12080 (time 1786034180.93 total 821.41 last 16.07) - open/close 12272 (time 1786034191.01 total 831.50 last 19.04) - open/close 12385 (time 1786034201.03 total 841.51 last 11.29) - open/close 12577 (time 1786034211.90 total 852.38 last 17.66) - open/close 12775 (time 1786034223.80 total 864.28 last 16.64) - open/close 12884 (time 1786034233.80 total 874.28 last 10.90) - open/close 13026 (time 1786034243.80 total 884.29 last 14.20) - open/close 13182 (time 1786034253.82 total 894.31 last 15.57) - open/close 13346 (time 1786034264.83 total 905.32 last 14.89) - open/close 13581 (time 1786034274.90 total 915.38 last 23.35) - open/close 13697 (time 1786034284.90 total 925.38 last 11.60) - open/close 13965 (time 1786034295.55 total 936.03 last 25.17) - open/close 14130 (time 1786034306.97 total 947.46 last 14.44) - open/close 14387 (time 1786034316.98 total 957.46 last 25.70) - open/close 14590 (time 1786034327.05 total 967.54 last 20.14) - open/close 14776 (time 1786034337.92 total 978.40 last 17.12) - open/close 14949 (time 1786034347.93 total 988.41 last 17.28) - open/close 15148 (time 1786034357.94 total 998.42 last 19.88) - open/close 15255 (time 1786034367.97 total 1008.46 last 10.66) - open/close 15359 (time 1786034378.08 total 1018.56 last 10.29) - open/close 15571 (time 1786034389.07 total 1029.56 last 19.29) - open/close 15747 (time 1786034399.10 total 1039.58 last 17.55) - open/close 15909 (time 1786034409.10 total 1049.59 last 16.19) - open/close 16128 (time 1786034419.29 total 1059.77 last 21.50) - open/close 16576 (time 1786034429.29 total 1069.78 last 44.77) - open/close 17187 (time 1786034439.30 total 1079.79 last 61.03) - open/close 17779 (time 1786034449.31 total 1089.80 last 59.16) - open/close 18475 (time 1786034459.32 total 1099.80 last 69.57) - open/close 19194 (time 1786034469.32 total 1109.80 last 71.87) - open/close 19853 (time 1786034479.33 total 1119.81 last 65.85) - open/close 20000 (time 1786034481.42 total 1121.90 last 70.22) - open/close 20634 (time 1786034491.46 total 1131.95 last 63.12) - open/close 21125 (time 1786034501.47 total 1141.96 last 49.05) - open/close 21845 (time 1786034511.48 total 1151.96 last 71.96) - open/close 22491 (time 1786034521.49 total 1161.97 last 64.54) - open/close 23011 (time 1786034531.50 total 1171.99 last 51.92) - open/close 23801 (time 1786034541.51 total 1182.00 last 78.95) - open/close 24507 (time 1786034551.52 total 1192.01 last 70.51) - open/close 25118 (time 1786034561.53 total 1202.02 last 61.05) - open/close 25714 (time 1786034571.54 total 1212.02 last 59.56) - open/close 26288 (time 1786034581.55 total 1222.04 last 57.32) - open/close 26807 (time 1786034591.56 total 1232.04 last 51.88) - open/close 27360 (time 1786034601.56 total 1242.04 last 55.29) - open/close 27983 (time 1786034611.57 total 1252.05 last 62.25) - open/close 28572 (time 1786034621.58 total 1262.06 last 58.85) - open/close 29168 (time 1786034631.58 total 1272.07 last 59.56) - open/close 29843 (time 1786034641.59 total 1282.07 last 67.45) - open/close 30000 (time 1786034643.97 total 1284.46 last 65.87) - open/close 30636 (time 1786034653.97 total 1294.46 last 63.59) - open/close 31312 (time 1786034663.98 total 1304.46 last 67.57) - open/close 31922 (time 1786034673.99 total 1314.47 last 60.95) - open/close 32451 (time 1786034683.99 total 1324.48 last 52.88) - open/close 32925 (time 1786034693.99 total 1334.48 last 47.39) - open/close 33479 (time 1786034704.00 total 1344.48 last 55.38) - open/close 34164 (time 1786034714.01 total 1354.49 last 68.43) - open/close 34836 (time 1786034724.01 total 1364.50 last 67.16) - open/close 35520 (time 1786034734.02 total 1374.50 last 68.37) - open/close 36255 (time 1786034744.03 total 1384.52 last 73.37) - open/close 36980 (time 1786034754.04 total 1394.52 last 72.48) - open/close 37592 (time 1786034764.05 total 1404.54 last 61.12) - open/close 38098 (time 1786034774.07 total 1414.56 last 50.49) - open/close 38538 (time 1786034784.12 total 1424.60 last 43.80) - open/close 38875 (time 1786034794.13 total 1434.61 last 33.67) - open/close 39360 (time 1786034804.13 total 1444.62 last 48.46) - open/close 39992 (time 1786034814.15 total 1454.63 last 63.14) - open/close 40696 (time 1786034824.15 total 1464.64 last 70.35) - open/close 41316 (time 1786034834.16 total 1474.64 last 61.97) - open/close 41956 (time 1786034844.16 total 1484.64 last 63.98) - open/close 42640 (time 1786034854.16 total 1494.65 last 68.37) - open/close 43324 (time 1786034864.16 total 1504.65 last 68.39) - open/close 44067 (time 1786034874.16 total 1514.65 last 74.30) - open/close 44856 (time 1786034884.17 total 1524.66 last 78.85) - open/close 45592 (time 1786034894.17 total 1534.66 last 73.57) - open/close 46243 (time 1786034904.18 total 1544.66 last 65.07) - open/close 46998 (time 1786034914.19 total 1554.67 last 75.44) - open/close 47632 (time 1786034924.19 total 1564.67 last 63.40) - open/close 48321 (time 1786034934.20 total 1574.68 last 68.83) - open/close 48869 (time 1786034944.20 total 1584.69 last 54.76) - open/close 49454 (time 1786034954.21 total 1594.70 last 58.44) total: 50000 open/close in 1602.80 seconds: 31.20 ops/second - unlinked 0 (time 1786034966 ; total 0 ; last 0) - unlinked 10000 (time 1786035131 ; total 165 ; last 165) - unlinked 20000 (time 1786035286 ; total 320 ; last 155) - unlinked 30000 (time 1786035447 ; total 481 ; last 161) - unlinked 40000 (time 1786035607 ; total 641 ; last 160) total: 50000 unlinks in 794 seconds: 62.972294 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2