== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:24:45 (1781234685) start mds service on oleg212-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing load_modules_local oleg212-server: Loading modules from /home/green/git/lustre-release/lustre oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg212-server: detected 4 online CPUs by sysfs oleg212-server: MODOPTS_LIBCFS= oleg212-server: Force libcfs to create 2 CPU partitions oleg212-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg212-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg212-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg212-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg212-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 58 (time 1781234797.07 total 11.01 last 5.27) - open/close 264 (time 1781234807.16 total 21.10 last 20.42) - open/close 542 (time 1781234817.84 total 31.79 last 26.01) - open/close 816 (time 1781234828.81 total 42.76 last 24.98) - open/close 1013 (time 1781234838.83 total 52.77 last 19.68) - open/close 1243 (time 1781234850.81 total 64.75 last 19.20) - open/close 1456 (time 1781234860.99 total 74.94 last 20.92) - open/close 1638 (time 1781234871.99 total 85.94 last 16.54) - open/close 1825 (time 1781234883.21 total 97.16 last 16.67) - open/close 1985 (time 1781234894.79 total 108.74 last 13.81) - open/close 2222 (time 1781234905.40 total 119.35 last 22.34) - open/close 2420 (time 1781234915.45 total 129.39 last 19.72) - open/close 2699 (time 1781234925.47 total 139.42 last 27.83) - open/close 2794 (time 1781234935.49 total 149.44 last 9.48) - open/close 3040 (time 1781234945.72 total 159.67 last 24.06) - open/close 3260 (time 1781234956.66 total 170.61 last 20.10) - open/close 3518 (time 1781234966.67 total 180.62 last 25.78) - open/close 3767 (time 1781234977.22 total 191.17 last 23.61) - open/close 4043 (time 1781234987.22 total 201.17 last 27.59) - open/close 4263 (time 1781234997.23 total 211.18 last 21.98) - open/close 4424 (time 1781235007.25 total 221.20 last 16.07) - open/close 4549 (time 1781235017.48 total 231.42 last 12.23) - open/close 4698 (time 1781235027.89 total 241.83 last 14.31) - open/close 4957 (time 1781235038.45 total 252.39 last 24.52) - open/close 5221 (time 1781235048.45 total 262.40 last 26.39) - open/close 5490 (time 1781235058.47 total 272.42 last 26.84) - open/close 5720 (time 1781235068.49 total 282.44 last 22.95) - open/close 5957 (time 1781235079.35 total 293.29 last 21.84) - open/close 6169 (time 1781235090.36 total 304.30 last 19.26) - open/close 6358 (time 1781235100.36 total 314.30 last 18.90) - open/close 6636 (time 1781235111.22 total 325.17 last 25.58) - open/close 6884 (time 1781235121.27 total 335.22 last 24.68) - open/close 6997 (time 1781235132.17 total 346.12 last 10.37) - open/close 7073 (time 1781235142.26 total 356.21 last 7.53) - open/close 7172 (time 1781235152.51 total 366.46 last 9.65) - open/close 7340 (time 1781235162.88 total 376.83 last 16.20) - open/close 7426 (time 1781235172.94 total 386.89 last 8.55) - open/close 7672 (time 1781235183.29 total 397.24 last 23.78) - open/close 7991 (time 1781235194.03 total 407.97 last 29.70) - open/close 8291 (time 1781235204.04 total 417.99 last 29.95) - open/close 8557 (time 1781235214.87 total 428.81 last 24.58) - open/close 8799 (time 1781235225.42 total 439.36 last 22.94) - open/close 9074 (time 1781235235.50 total 449.45 last 27.26) - open/close 9220 (time 1781235246.01 total 459.95 last 13.90) - open/close 9313 (time 1781235257.22 total 471.16 last 8.30) - open/close 9506 (time 1781235268.01 total 481.96 last 17.88) - open/close 9738 (time 1781235278.33 total 492.27 last 22.50) - open/close 10000 (time 1781235288.11 total 502.06 last 26.77) - open/close 10291 (time 1781235298.12 total 512.06 last 29.09) - open/close 10461 (time 1781235308.62 total 522.56 last 16.19) - open/close 10728 (time 1781235318.64 total 532.59 last 26.63) - open/close 10921 (time 1781235328.64 total 542.59 last 19.30) - open/close 11174 (time 1781235338.81 total 552.75 last 24.89) - open/close 11440 (time 1781235349.78 total 563.72 last 24.25) - open/close 11662 (time 1781235359.81 total 573.76 last 22.12) - open/close 11925 (time 1781235369.82 total 583.76 last 26.28) - open/close 12128 (time 1781235380.15 total 594.10 last 19.64) - open/close 12275 (time 1781235390.15 total 604.10 last 14.70) - open/close 12389 (time 1781235403.06 total 617.01 last 8.83) - open/close 12522 (time 1781235413.07 total 627.01 last 13.29) - open/close 12606 (time 1781235423.32 total 637.26 last 8.19) - open/close 12750 (time 1781235440.25 total 654.20 last 8.50) - open/close 12985 (time 1781235451.00 total 664.95 last 21.86) - open/close 13188 (time 1781235461.02 total 674.97 last 20.27) - open/close 13351 (time 1781235471.03 total 684.98 last 16.29) - open/close 13476 (time 1781235483.64 total 697.59 last 9.91) - open/close 13753 (time 1781235494.40 total 708.35 last 25.75) - open/close 13955 (time 1781235505.48 total 719.43 last 18.23) - open/close 14090 (time 1781235515.49 total 729.44 last 13.49) - open/close 14343 (time 1781235525.50 total 739.45 last 25.28) - open/close 14508 (time 1781235536.25 total 750.20 last 15.34) - open/close 14761 (time 1781235546.26 total 760.20 last 25.30) - open/close 14928 (time 1781235556.59 total 770.54 last 16.16) - open/close 15176 (time 1781235566.99 total 780.94 last 23.84) - open/close 15456 (time 1781235577.48 total 791.43 last 26.70) - open/close 15724 (time 1781235587.49 total 801.44 last 26.77) - open/close 15943 (time 1781235597.49 total 811.44 last 21.89) - open/close 16178 (time 1781235608.51 total 822.46 last 21.33) - open/close 16439 (time 1781235619.08 total 833.02 last 24.70) - open/close 16706 (time 1781235629.46 total 843.41 last 25.70) - open/close 16954 (time 1781235639.47 total 853.42 last 24.78) - open/close 17127 (time 1781235651.79 total 865.74 last 14.04) - open/close 17266 (time 1781235661.81 total 875.76 last 13.87) - open/close 17490 (time 1781235671.98 total 885.93 last 22.02) - open/close 17767 (time 1781235684.40 total 898.34 last 22.31) - open/close 18045 (time 1781235694.40 total 908.34 last 27.80) - open/close 18282 (time 1781235704.85 total 918.80 last 22.67) - open/close 18464 (time 1781235715.07 total 929.02 last 17.81) - open/close 18597 (time 1781235726.07 total 940.01 last 12.10) - open/close 18723 (time 1781235737.02 total 950.96 last 11.51) - open/close 18957 (time 1781235747.85 total 961.79 last 21.61) - open/close 19218 (time 1781235758.44 total 972.38 last 24.64) - open/close 19452 (time 1781235768.44 total 982.38 last 23.40) - open/close 19912 (time 1781235778.44 total 992.39 last 45.99) - open/close 20962 (time 1781235788.44 total 1002.39 last 104.98) - open/close 22000 (time 1781235798.45 total 1012.40 last 103.69) - open/close 22905 (time 1781235808.46 total 1022.41 last 90.43) - open/close 23894 (time 1781235818.47 total 1032.42 last 98.81) - open/close 24861 (time 1781235828.48 total 1042.43 last 96.60) - open/close 25770 (time 1781235838.49 total 1052.43 last 90.83) - open/close 26860 (time 1781235848.50 total 1062.44 last 108.89) - open/close 27952 (time 1781235858.51 total 1072.45 last 109.10) - open/close 29005 (time 1781235868.52 total 1082.46 last 105.17) - open/close 30000 (time 1781235878.35 total 1092.30 last 101.19) - open/close 31096 (time 1781235888.36 total 1102.30 last 109.55) - open/close 31964 (time 1781235898.36 total 1112.31 last 86.78) - open/close 33012 (time 1781235908.37 total 1122.31 last 104.71) - open/close 33847 (time 1781235918.38 total 1132.32 last 83.43) - open/close 34475 (time 1781235928.39 total 1142.34 last 62.69) - open/close 35426 (time 1781235938.40 total 1152.34 last 95.05) - open/close 36278 (time 1781235948.40 total 1162.35 last 85.17) - open/close 36874 (time 1781235958.41 total 1172.36 last 59.54) - open/close 37484 (time 1781235968.44 total 1182.39 last 60.82) - open/close 38397 (time 1781235978.46 total 1192.41 last 91.10) - open/close 39123 (time 1781235988.46 total 1202.41 last 72.58) - open/close 39941 (time 1781235998.47 total 1212.42 last 81.74) - open/close 40779 (time 1781236008.47 total 1222.42 last 83.78) - open/close 41695 (time 1781236018.48 total 1232.43 last 91.52) - open/close 42651 (time 1781236028.49 total 1242.43 last 95.57) - open/close 43642 (time 1781236038.49 total 1252.44 last 99.05) - open/close 44689 (time 1781236048.49 total 1262.44 last 104.69) - open/close 45659 (time 1781236058.50 total 1272.44 last 96.96) - open/close 46548 (time 1781236068.50 total 1282.45 last 88.86) - open/close 47602 (time 1781236078.50 total 1292.45 last 105.36) - open/close 48519 (time 1781236088.51 total 1302.45 last 91.67) - open/close 49399 (time 1781236098.51 total 1312.46 last 87.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1315.35 seconds: 37.73 ops/second - unlinked 0 (time 1781236107 ; total 1 ; last 1) - unlinked 10000 (time 1781236219 ; total 113 ; last 112) - unlinked 20000 (time 1781236333 ; total 227 ; last 114) - unlinked 30000 (time 1781236409 ; total 303 ; last 76) - unlinked 40000 (time 1781236472 ; total 366 ; last 63) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 426 seconds: 116.504692 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg212-server unloading modules via unload_modules_local on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing unload_modules_local oleg212-server: modules unloaded. pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2