== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:33:54 (1776998034) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 122 (time 1776998123.65 total 10.56 last 11.56) - open/close 306 (time 1776998135.03 total 21.93 last 16.18) - open/close 419 (time 1776998145.05 total 31.95 last 11.28) - open/close 603 (time 1776998155.90 total 42.80 last 16.96) - open/close 792 (time 1776998165.93 total 52.83 last 18.84) - open/close 889 (time 1776998178.81 total 65.71 last 7.53) - open/close 1118 (time 1776998188.83 total 75.73 last 22.86) - open/close 1309 (time 1776998200.88 total 87.79 last 15.84) - open/close 1515 (time 1776998212.33 total 99.23 last 17.99) - open/close 1701 (time 1776998223.35 total 110.26 last 16.87) - open/close 1868 (time 1776998234.09 total 121.00 last 15.55) - open/close 1944 (time 1776998244.42 total 131.33 last 7.36) - open/close 2143 (time 1776998254.51 total 141.41 last 19.74) - open/close 2351 (time 1776998265.46 total 152.36 last 18.99) - open/close 2521 (time 1776998276.03 total 162.94 last 16.08) - open/close 2718 (time 1776998286.96 total 173.87 last 18.02) - open/close 2902 (time 1776998296.97 total 183.87 last 18.39) - open/close 3116 (time 1776998309.27 total 196.17 last 17.40) - open/close 3259 (time 1776998320.05 total 206.95 last 13.27) - open/close 3377 (time 1776998330.89 total 217.79 last 10.88) - open/close 3523 (time 1776998341.45 total 228.36 last 13.82) - open/close 3757 (time 1776998352.14 total 239.04 last 21.90) - open/close 3927 (time 1776998366.46 total 253.36 last 11.87) - open/close 4021 (time 1776998376.65 total 263.55 last 9.23) - open/close 4229 (time 1776998386.81 total 273.72 last 20.46) - open/close 4423 (time 1776998396.83 total 283.73 last 19.37) - open/close 4585 (time 1776998406.85 total 293.75 last 16.16) - open/close 4776 (time 1776998417.41 total 304.32 last 18.08) - open/close 4936 (time 1776998427.53 total 314.43 last 15.82) - open/close 5208 (time 1776998438.39 total 325.29 last 25.04) - open/close 5405 (time 1776998448.58 total 335.48 last 19.33) - open/close 5656 (time 1776998459.33 total 346.23 last 23.36) - open/close 5855 (time 1776998469.61 total 356.52 last 19.34) - open/close 6103 (time 1776998480.63 total 367.54 last 22.51) - open/close 6339 (time 1776998490.90 total 377.81 last 22.98) - open/close 6505 (time 1776998501.23 total 388.14 last 16.07) - open/close 6752 (time 1776998514.13 total 401.04 last 19.14) - open/close 6991 (time 1776998524.42 total 411.33 last 23.23) - open/close 7215 (time 1776998535.40 total 422.31 last 20.40) - open/close 7445 (time 1776998546.29 total 433.19 last 21.13) - open/close 7588 (time 1776998556.29 total 443.19 last 14.30) - open/close 7683 (time 1776998566.90 total 453.80 last 8.96) - open/close 7806 (time 1776998576.91 total 463.81 last 12.29) - open/close 7979 (time 1776998586.94 total 473.84 last 17.25) - open/close 8115 (time 1776998597.48 total 484.39 last 12.90) - open/close 8262 (time 1776998608.76 total 495.67 last 13.03) - open/close 8471 (time 1776998618.99 total 505.90 last 20.43) - open/close 8571 (time 1776998629.04 total 515.95 last 9.95) - open/close 8752 (time 1776998639.23 total 526.13 last 17.78) - open/close 8969 (time 1776998649.24 total 536.15 last 21.66) - open/close 9113 (time 1776998659.98 total 546.88 last 13.41) - open/close 9184 (time 1776998670.34 total 557.25 last 6.85) - open/close 9385 (time 1776998680.35 total 567.26 last 20.08) - open/close 9604 (time 1776998690.35 total 577.26 last 21.90) - open/close 9833 (time 1776998700.39 total 587.30 last 22.82) - open/close 10000 (time 1776998710.16 total 597.07 last 17.09) - open/close 10179 (time 1776998720.17 total 607.07 last 17.89) - open/close 10304 (time 1776998730.19 total 617.10 last 12.47) - open/close 10464 (time 1776998742.00 total 628.91 last 13.55) - open/close 10593 (time 1776998752.00 total 638.91 last 12.90) - open/close 10690 (time 1776998762.39 total 649.29 last 9.34) - open/close 10888 (time 1776998772.59 total 659.49 last 19.41) - open/close 11027 (time 1776998782.71 total 669.61 last 13.73) - open/close 11290 (time 1776998793.47 total 680.37 last 24.44) - open/close 11488 (time 1776998803.49 total 690.39 last 19.76) - open/close 11710 (time 1776998813.50 total 700.41 last 22.17) - open/close 11820 (time 1776998823.64 total 710.54 last 10.85) - open/close 11979 (time 1776998834.43 total 721.33 last 14.73) - open/close 12136 (time 1776998844.79 total 731.69 last 15.15) - open/close 12313 (time 1776998855.43 total 742.33 last 16.64) - open/close 12605 (time 1776998868.29 total 755.20 last 22.70) - open/close 12858 (time 1776998878.52 total 765.42 last 24.74) - open/close 13112 (time 1776998888.83 total 775.74 last 24.63) - open/close 13210 (time 1776998899.20 total 786.11 last 9.45) - open/close 13326 (time 1776998909.20 total 796.11 last 11.60) - open/close 13451 (time 1776998919.47 total 806.38 last 12.17) - open/close 13577 (time 1776998929.47 total 816.38 last 12.60) - open/close 13699 (time 1776998939.48 total 826.39 last 12.19) - open/close 13868 (time 1776998949.49 total 836.40 last 16.88) - open/close 14037 (time 1776998959.61 total 846.51 last 16.71) - open/close 14151 (time 1776998969.63 total 856.53 last 11.38) - open/close 14354 (time 1776998979.64 total 866.55 last 20.27) - open/close 14528 (time 1776998989.92 total 876.82 last 16.93) - open/close 14775 (time 1776998999.93 total 886.84 last 24.66) - open/close 14971 (time 1776999009.98 total 896.88 last 19.52) - open/close 15172 (time 1776999020.03 total 906.94 last 19.99) - open/close 15344 (time 1776999030.04 total 916.94 last 17.19) - open/close 15522 (time 1776999040.05 total 926.96 last 17.78) - open/close 15748 (time 1776999050.07 total 936.97 last 22.57) - open/close 15967 (time 1776999060.47 total 947.38 last 21.04) - open/close 16164 (time 1776999071.21 total 958.12 last 18.34) - open/close 16343 (time 1776999084.15 total 971.05 last 13.84) - open/close 16497 (time 1776999094.19 total 981.09 last 15.34) - open/close 16655 (time 1776999105.20 total 992.10 last 14.35) - open/close 16859 (time 1776999115.51 total 1002.42 last 19.78) - open/close 17098 (time 1776999125.55 total 1012.46 last 23.81) - open/close 17300 (time 1776999135.58 total 1022.48 last 20.15) - open/close 17517 (time 1776999146.18 total 1033.08 last 20.47) - open/close 18261 (time 1776999156.18 total 1043.08 last 74.39) - open/close 19015 (time 1776999166.18 total 1053.09 last 75.37) - open/close 19618 (time 1776999176.20 total 1063.11 last 60.18) - open/close 20000 (time 1776999182.75 total 1069.66 last 58.32) - open/close 20765 (time 1776999192.77 total 1079.67 last 76.38) - open/close 21450 (time 1776999202.79 total 1089.70 last 68.34) - open/close 22108 (time 1776999212.81 total 1099.71 last 65.71) - open/close 22940 (time 1776999222.82 total 1109.73 last 83.08) - open/close 23634 (time 1776999232.82 total 1119.73 last 69.40) - open/close 24402 (time 1776999242.82 total 1129.73 last 76.78) - open/close 25201 (time 1776999252.83 total 1139.74 last 79.82) - open/close 26072 (time 1776999262.84 total 1149.74 last 87.06) - open/close 26949 (time 1776999272.87 total 1159.77 last 87.44) - open/close 27809 (time 1776999282.88 total 1169.78 last 85.93) - open/close 28591 (time 1776999292.88 total 1179.79 last 78.16) - open/close 29463 (time 1776999302.90 total 1189.81 last 87.03) - open/close 30000 (time 1776999310.58 total 1197.49 last 69.93) - open/close 30741 (time 1776999320.60 total 1207.50 last 73.97) - open/close 31505 (time 1776999330.60 total 1217.50 last 76.39) - open/close 32282 (time 1776999340.61 total 1227.51 last 77.66) - open/close 33074 (time 1776999350.61 total 1237.52 last 79.16) - open/close 33888 (time 1776999360.62 total 1247.52 last 81.36) - open/close 34619 (time 1776999370.62 total 1257.53 last 73.06) - open/close 35338 (time 1776999380.62 total 1267.53 last 71.89) - open/close 36178 (time 1776999390.64 total 1277.55 last 83.82) - open/close 37023 (time 1776999400.65 total 1287.56 last 84.40) - open/close 37881 (time 1776999410.66 total 1297.56 last 85.76) - open/close 38713 (time 1776999420.66 total 1307.57 last 83.18) - open/close 39573 (time 1776999430.67 total 1317.57 last 85.95) - open/close 40000 (time 1776999435.58 total 1322.49 last 86.92) - open/close 40808 (time 1776999445.59 total 1332.49 last 80.75) - open/close 41569 (time 1776999455.59 total 1342.49 last 76.09) - open/close 42329 (time 1776999465.61 total 1352.51 last 75.86) - open/close 43099 (time 1776999475.61 total 1362.51 last 76.98) - open/close 43864 (time 1776999485.61 total 1372.52 last 76.48) - open/close 44664 (time 1776999495.62 total 1382.52 last 79.96) - open/close 45523 (time 1776999505.63 total 1392.53 last 85.81) - open/close 46388 (time 1776999515.63 total 1402.54 last 86.45) - open/close 47198 (time 1776999525.65 total 1412.55 last 80.88) - open/close 48032 (time 1776999535.66 total 1422.56 last 83.32) - open/close 48876 (time 1776999545.66 total 1432.56 last 84.39) - open/close 49714 (time 1776999555.67 total 1442.57 last 83.73) total: 50000 open/close in 1445.82 seconds: 34.58 ops/second - unlinked 0 (time 1776999562 ; total 0 ; last 0) - unlinked 10000 (time 1776999692 ; total 130 ; last 130) - unlinked 20000 (time 1776999811 ; total 249 ; last 119) - unlinked 30000 (time 1776999929 ; total 367 ; last 118) - unlinked 40000 (time 1777000038 ; total 476 ; last 109) total: 50000 unlinks in 544 seconds: 91.911766 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2