== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:56:33 (1780455393) start mds service on oleg150-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: MODOPTS_LIBCFS= oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg150-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg150-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg150-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1780455503.56 total 11.01 last 16.44) - open/close 443 (time 1780455513.56 total 21.01 last 26.20) - open/close 692 (time 1780455524.46 total 31.92 last 22.83) - open/close 767 (time 1780455536.19 total 43.65 last 6.39) - open/close 1018 (time 1780455547.01 total 54.46 last 23.20) - open/close 1231 (time 1780455557.01 total 64.47 last 21.28) - open/close 1490 (time 1780455567.55 total 75.01 last 24.57) - open/close 1716 (time 1780455578.32 total 85.77 last 21.00) - open/close 1964 (time 1780455588.32 total 95.78 last 24.79) - open/close 2216 (time 1780455599.28 total 106.74 last 22.99) - open/close 2314 (time 1780455609.40 total 116.85 last 9.69) - open/close 2475 (time 1780455620.98 total 128.44 last 13.90) - open/close 2644 (time 1780455631.01 total 138.46 last 16.86) - open/close 2873 (time 1780455641.01 total 148.47 last 22.89) - open/close 3161 (time 1780455651.02 total 158.47 last 28.78) - open/close 3329 (time 1780455661.71 total 169.17 last 15.71) - open/close 3414 (time 1780455672.13 total 179.59 last 8.16) - open/close 3527 (time 1780455682.54 total 190.00 last 10.85) - open/close 3637 (time 1780455693.45 total 200.91 last 10.08) - open/close 3835 (time 1780455703.87 total 211.33 last 19.00) - open/close 4067 (time 1780455714.04 total 221.49 last 22.82) - open/close 4313 (time 1780455724.15 total 231.61 last 24.32) - open/close 4528 (time 1780455734.38 total 241.84 last 21.02) - open/close 4799 (time 1780455744.39 total 251.85 last 27.07) - open/close 4988 (time 1780455755.31 total 262.76 last 17.32) - open/close 5164 (time 1780455765.43 total 272.89 last 17.38) - open/close 5406 (time 1780455775.44 total 282.90 last 24.18) - open/close 5573 (time 1780455785.72 total 293.18 last 16.25) - open/close 5746 (time 1780455795.73 total 303.19 last 17.28) - open/close 5842 (time 1780455805.73 total 313.19 last 9.59) - open/close 6013 (time 1780455816.29 total 323.75 last 16.19) - open/close 6189 (time 1780455826.31 total 333.77 last 17.57) - open/close 6493 (time 1780455837.20 total 344.66 last 27.91) - open/close 6746 (time 1780455850.31 total 357.77 last 19.30) - open/close 6992 (time 1780455861.11 total 368.57 last 22.78) - open/close 7219 (time 1780455871.21 total 378.67 last 22.46) - open/close 7395 (time 1780455882.60 total 390.06 last 15.46) - open/close 7625 (time 1780455892.86 total 400.32 last 22.41) - open/close 7892 (time 1780455903.29 total 410.75 last 25.61) - open/close 8119 (time 1780455913.83 total 421.29 last 21.53) - open/close 8421 (time 1780455924.86 total 432.32 last 27.37) - open/close 8680 (time 1780455934.88 total 442.34 last 25.85) - open/close 8851 (time 1780455945.14 total 452.60 last 16.67) - open/close 9051 (time 1780455955.45 total 462.91 last 19.41) - open/close 9297 (time 1780455965.45 total 472.91 last 24.59) - open/close 9556 (time 1780455975.66 total 483.12 last 25.37) - open/close 9808 (time 1780455985.67 total 493.12 last 25.19) - open/close 10000 (time 1780455992.44 total 499.90 last 28.33) - open/close 10201 (time 1780456002.99 total 510.45 last 19.06) - open/close 10380 (time 1780456013.00 total 520.46 last 17.88) - open/close 10507 (time 1780456023.55 total 531.01 last 12.04) - open/close 10658 (time 1780456034.52 total 541.97 last 13.77) - open/close 10752 (time 1780456044.56 total 552.02 last 9.36) - open/close 10846 (time 1780456056.10 total 563.56 last 8.14) - open/close 11022 (time 1780456066.90 total 574.36 last 16.30) - open/close 11138 (time 1780456076.92 total 584.37 last 11.58) - open/close 11316 (time 1780456087.60 total 595.06 last 16.67) - open/close 11426 (time 1780456098.50 total 605.96 last 10.09) - open/close 11611 (time 1780456109.60 total 617.05 last 16.68) - open/close 11846 (time 1780456119.63 total 627.08 last 23.43) - open/close 12018 (time 1780456129.63 total 637.09 last 17.19) - open/close 12320 (time 1780456140.31 total 647.77 last 28.27) - open/close 12388 (time 1780456151.15 total 658.61 last 6.28) - open/close 12563 (time 1780456161.16 total 668.62 last 17.48) - open/close 12708 (time 1780456171.41 total 678.86 last 14.15) - open/close 12892 (time 1780456181.41 total 688.87 last 18.39) - open/close 13096 (time 1780456191.95 total 699.41 last 19.35) - open/close 13273 (time 1780456201.98 total 709.44 last 17.65) - open/close 13473 (time 1780456212.01 total 719.46 last 19.95) - open/close 13710 (time 1780456222.01 total 729.47 last 23.69) - open/close 13894 (time 1780456232.43 total 739.89 last 17.66) - open/close 14133 (time 1780456243.50 total 750.95 last 21.59) - open/close 14297 (time 1780456254.08 total 761.53 last 15.50) - open/close 14547 (time 1780456264.22 total 771.68 last 24.65) - open/close 14671 (time 1780456274.22 total 781.68 last 12.40) - open/close 14840 (time 1780456284.24 total 791.70 last 16.87) - open/close 15024 (time 1780456294.24 total 801.70 last 18.39) - open/close 15208 (time 1780456305.28 total 812.74 last 16.67) - open/close 15450 (time 1780456316.16 total 823.62 last 22.25) - open/close 15664 (time 1780456326.51 total 833.97 last 20.68) - open/close 15913 (time 1780456337.34 total 844.80 last 22.99) - open/close 16142 (time 1780456347.76 total 855.21 last 21.98) - open/close 16368 (time 1780456357.82 total 865.28 last 22.45) - open/close 16549 (time 1780456367.83 total 875.29 last 18.09) - open/close 16790 (time 1780456377.94 total 885.39 last 23.85) - open/close 17044 (time 1780456388.31 total 895.77 last 24.48) - open/close 17296 (time 1780456398.81 total 906.27 last 23.99) - open/close 17436 (time 1780456409.60 total 917.06 last 12.98) - open/close 17604 (time 1780456419.71 total 927.17 last 16.61) - open/close 17723 (time 1780456429.72 total 937.18 last 11.89) - open/close 17821 (time 1780456440.19 total 947.65 last 9.36) - open/close 17914 (time 1780456450.74 total 958.20 last 8.82) - open/close 18026 (time 1780456461.02 total 968.47 last 10.90) - open/close 18211 (time 1780456473.92 total 981.37 last 14.34) - open/close 18328 (time 1780456483.93 total 991.39 last 11.69) - open/close 18513 (time 1780456494.32 total 1001.78 last 17.80) - open/close 18672 (time 1780456504.88 total 1012.34 last 15.06) - open/close 19311 (time 1780456514.89 total 1022.35 last 63.84) - open/close 20000 (time 1780456524.20 total 1031.66 last 74.04) - open/close 20710 (time 1780456534.20 total 1041.66 last 70.96) - open/close 21563 (time 1780456544.22 total 1051.67 last 85.20) - open/close 22489 (time 1780456554.25 total 1061.71 last 92.29) - open/close 23097 (time 1780456564.26 total 1071.71 last 60.75) - open/close 23831 (time 1780456574.26 total 1081.72 last 73.33) - open/close 24660 (time 1780456584.27 total 1091.73 last 82.86) - open/close 25638 (time 1780456594.27 total 1101.73 last 97.76) - open/close 26564 (time 1780456604.29 total 1111.74 last 92.48) - open/close 27171 (time 1780456614.29 total 1121.75 last 60.68) - open/close 27887 (time 1780456624.30 total 1131.76 last 71.51) - open/close 28517 (time 1780456634.31 total 1141.77 last 62.95) - open/close 29142 (time 1780456644.32 total 1151.78 last 62.45) - open/close 29853 (time 1780456654.34 total 1161.79 last 70.97) - open/close 30558 (time 1780456664.35 total 1171.80 last 70.44) - open/close 31459 (time 1780456674.36 total 1181.82 last 89.96) - open/close 32138 (time 1780456684.37 total 1191.82 last 67.85) - open/close 32800 (time 1780456694.38 total 1201.83 last 66.14) - open/close 33411 (time 1780456704.39 total 1211.85 last 61.03) - open/close 33872 (time 1780456714.40 total 1221.85 last 46.07) - open/close 34712 (time 1780456724.40 total 1231.85 last 83.99) - open/close 35453 (time 1780456734.41 total 1241.87 last 73.99) - open/close 36235 (time 1780456744.42 total 1251.88 last 78.12) - open/close 37073 (time 1780456754.42 total 1261.88 last 83.77) - open/close 37915 (time 1780456764.43 total 1271.88 last 84.19) - open/close 38720 (time 1780456774.43 total 1281.89 last 80.44) - open/close 39535 (time 1780456784.44 total 1291.89 last 81.47) - open/close 40000 (time 1780456790.52 total 1297.97 last 76.49) - open/close 40705 (time 1780456800.52 total 1307.97 last 70.49) - open/close 41485 (time 1780456810.53 total 1317.98 last 77.94) - open/close 42128 (time 1780456820.54 total 1328.00 last 64.21) - open/close 42855 (time 1780456830.54 total 1338.00 last 72.67) - open/close 43637 (time 1780456840.55 total 1348.00 last 78.18) - open/close 44393 (time 1780456850.55 total 1358.01 last 75.55) - open/close 45184 (time 1780456860.55 total 1368.01 last 79.10) - open/close 45839 (time 1780456870.55 total 1378.01 last 65.49) - open/close 46685 (time 1780456880.56 total 1388.01 last 84.59) - open/close 47363 (time 1780456890.57 total 1398.02 last 67.72) - open/close 48196 (time 1780456900.57 total 1408.03 last 83.26) - open/close 48992 (time 1780456910.58 total 1418.03 last 79.56) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1425.41 seconds: 34.82 ops/second - unlinked 0 (time 1780456922 ; total 0 ; last 0) - unlinked 10000 (time 1780457075 ; total 153 ; last 153) - unlinked 20000 (time 1780457214 ; total 292 ; last 139) - unlinked 30000 (time 1780457360 ; total 438 ; last 146) - unlinked 40000 (time 1780457508 ; total 586 ; last 148) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 718 seconds: 69.123955 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server unloading modules via unload_modules_local on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local oleg150-server: modules unloaded. pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2