== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:29:38 (1788445778) start mds service on oleg248-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions oleg248-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg248-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg248-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg248-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg248-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 220 (time 1788445893.01 total 10.02 last 21.96) - open/close 469 (time 1788445903.30 total 20.31 last 24.20) - open/close 633 (time 1788445913.30 total 30.31 last 16.40) - open/close 849 (time 1788445923.72 total 40.73 last 20.73) - open/close 1056 (time 1788445934.16 total 51.16 last 19.84) - open/close 1258 (time 1788445945.56 total 62.57 last 17.71) - open/close 1467 (time 1788445956.29 total 73.30 last 19.48) - open/close 1727 (time 1788445966.30 total 83.30 last 25.98) - open/close 1932 (time 1788445976.98 total 93.98 last 19.20) - open/close 2147 (time 1788445987.00 total 104.00 last 21.46) - open/close 2357 (time 1788445997.60 total 114.61 last 19.80) - open/close 2642 (time 1788446007.63 total 124.64 last 28.42) - open/close 2917 (time 1788446017.63 total 134.64 last 27.50) - open/close 3185 (time 1788446027.65 total 144.66 last 26.74) - open/close 3449 (time 1788446038.07 total 155.08 last 25.34) - open/close 3749 (time 1788446048.09 total 165.09 last 29.96) - open/close 3940 (time 1788446058.10 total 175.10 last 19.07) - open/close 4244 (time 1788446070.55 total 187.55 last 24.42) - open/close 4412 (time 1788446080.93 total 197.93 last 16.18) - open/close 4575 (time 1788446091.89 total 208.90 last 14.87) - open/close 4802 (time 1788446102.57 total 219.57 last 21.26) - open/close 5023 (time 1788446113.37 total 230.38 last 20.45) - open/close 5199 (time 1788446124.45 total 241.45 last 15.90) - open/close 5528 (time 1788446134.45 total 251.45 last 32.88) - open/close 5718 (time 1788446144.54 total 261.54 last 18.83) - open/close 5920 (time 1788446154.63 total 271.63 last 20.02) - open/close 6125 (time 1788446166.84 total 283.85 last 16.79) - open/close 6378 (time 1788446177.42 total 294.43 last 23.91) - open/close 6670 (time 1788446187.43 total 304.44 last 29.16) - open/close 6870 (time 1788446198.02 total 315.02 last 18.90) - open/close 7047 (time 1788446208.04 total 325.05 last 17.65) - open/close 7321 (time 1788446218.46 total 335.46 last 26.32) - open/close 7523 (time 1788446228.49 total 345.49 last 20.14) - open/close 7760 (time 1788446238.73 total 355.73 last 23.15) - open/close 8042 (time 1788446248.73 total 365.74 last 28.17) - open/close 8316 (time 1788446259.47 total 376.48 last 25.52) - open/close 8615 (time 1788446269.48 total 386.48 last 29.88) - open/close 8845 (time 1788446279.97 total 396.97 last 21.92) - open/close 8977 (time 1788446289.98 total 406.98 last 13.18) - open/close 9251 (time 1788446299.99 total 417.00 last 27.37) - open/close 9530 (time 1788446310.00 total 427.01 last 27.87) - open/close 9809 (time 1788446320.01 total 437.01 last 27.89) - open/close 10000 (time 1788446327.58 total 444.58 last 25.22) - open/close 10222 (time 1788446337.60 total 454.60 last 22.16) - open/close 10467 (time 1788446347.60 total 464.60 last 24.49) - open/close 10630 (time 1788446358.36 total 475.36 last 15.15) - open/close 10844 (time 1788446370.18 total 487.19 last 18.10) - open/close 10979 (time 1788446381.00 total 498.00 last 12.48) - open/close 11148 (time 1788446391.05 total 508.06 last 16.81) - open/close 11356 (time 1788446401.07 total 518.07 last 20.76) - open/close 11579 (time 1788446411.71 total 528.71 last 20.96) - open/close 11836 (time 1788446421.71 total 538.72 last 25.70) - open/close 12065 (time 1788446431.87 total 548.87 last 22.55) - open/close 12311 (time 1788446441.88 total 558.89 last 24.56) - open/close 12578 (time 1788446451.89 total 568.90 last 26.67) - open/close 12860 (time 1788446462.86 total 579.86 last 25.72) - open/close 13034 (time 1788446473.62 total 590.62 last 16.18) - open/close 13243 (time 1788446484.63 total 601.63 last 18.98) - open/close 13496 (time 1788446494.81 total 611.81 last 24.85) - open/close 13741 (time 1788446505.80 total 622.80 last 22.29) - open/close 13985 (time 1788446516.62 total 633.62 last 22.55) - open/close 14232 (time 1788446527.37 total 644.37 last 22.97) - open/close 14459 (time 1788446537.55 total 654.55 last 22.30) - open/close 14711 (time 1788446548.43 total 665.44 last 23.14) - open/close 14811 (time 1788446558.67 total 675.67 last 9.77) - open/close 14936 (time 1788446568.68 total 685.68 last 12.49) - open/close 15184 (time 1788446578.92 total 695.93 last 24.21) - open/close 15358 (time 1788446589.65 total 706.65 last 16.23) - open/close 15539 (time 1788446599.73 total 716.74 last 17.95) - open/close 15689 (time 1788446609.73 total 726.74 last 15.00) - open/close 15893 (time 1788446619.73 total 736.74 last 20.40) - open/close 16190 (time 1788446630.02 total 747.02 last 28.88) - open/close 16448 (time 1788446640.90 total 757.90 last 23.70) - open/close 16714 (time 1788446651.50 total 768.50 last 25.11) - open/close 16887 (time 1788446662.13 total 779.14 last 16.26) - open/close 17053 (time 1788446672.58 total 789.58 last 15.90) - open/close 17172 (time 1788446682.59 total 799.59 last 11.89) - open/close 17379 (time 1788446692.86 total 809.86 last 20.15) - open/close 17599 (time 1788446704.16 total 821.17 last 19.46) - open/close 17803 (time 1788446714.17 total 831.18 last 20.38) - open/close 17957 (time 1788446724.77 total 841.78 last 14.53) - open/close 18156 (time 1788446734.78 total 851.79 last 19.88) - open/close 18486 (time 1788446745.50 total 862.51 last 30.78) - open/close 18749 (time 1788446755.51 total 872.51 last 26.28) - open/close 18947 (time 1788446765.69 total 882.70 last 19.45) - open/close 19204 (time 1788446776.20 total 893.20 last 24.46) - open/close 19492 (time 1788446787.12 total 904.12 last 26.38) - open/close 19709 (time 1788446797.21 total 914.21 last 21.51) - open/close 19936 (time 1788446807.21 total 924.21 last 22.70) - open/close 20000 (time 1788446810.57 total 927.58 last 19.01) - open/close 20262 (time 1788446820.58 total 937.59 last 26.18) - open/close 20482 (time 1788446831.49 total 948.50 last 20.16) - open/close 20644 (time 1788446841.86 total 958.86 last 15.63) - open/close 20818 (time 1788446851.87 total 968.87 last 17.39) - open/close 21248 (time 1788446861.87 total 978.87 last 42.99) - open/close 22137 (time 1788446871.87 total 988.88 last 88.86) - open/close 23053 (time 1788446881.88 total 998.89 last 91.53) - open/close 23874 (time 1788446891.90 total 1008.90 last 81.98) - open/close 24774 (time 1788446901.91 total 1018.91 last 89.90) - open/close 25592 (time 1788446911.91 total 1028.91 last 81.79) - open/close 26431 (time 1788446921.92 total 1038.93 last 83.79) - open/close 27354 (time 1788446931.93 total 1048.94 last 92.21) - open/close 28305 (time 1788446941.94 total 1058.94 last 95.02) - open/close 29174 (time 1788446951.94 total 1068.95 last 86.87) - open/close 30000 (time 1788446960.85 total 1077.86 last 92.74) - open/close 31003 (time 1788446970.86 total 1087.86 last 100.22) - open/close 31833 (time 1788446980.86 total 1097.87 last 82.97) - open/close 32768 (time 1788446990.87 total 1107.87 last 93.48) - open/close 33639 (time 1788447000.87 total 1117.87 last 87.06) - open/close 34608 (time 1788447010.88 total 1127.88 last 96.83) - open/close 35523 (time 1788447020.88 total 1137.88 last 91.48) - open/close 36441 (time 1788447030.90 total 1147.90 last 91.64) - open/close 37278 (time 1788447040.90 total 1157.91 last 83.65) - open/close 38125 (time 1788447050.91 total 1167.92 last 84.61) - open/close 39047 (time 1788447060.92 total 1177.93 last 92.13) - open/close 39910 (time 1788447070.93 total 1187.93 last 86.23) - open/close 40730 (time 1788447080.94 total 1197.94 last 81.94) - open/close 41519 (time 1788447090.95 total 1207.95 last 78.82) - open/close 42195 (time 1788447100.95 total 1217.96 last 67.56) - open/close 42839 (time 1788447110.96 total 1227.97 last 64.33) - open/close 43464 (time 1788447120.97 total 1237.97 last 62.47) - open/close 44487 (time 1788447130.97 total 1247.97 last 102.29) - open/close 45473 (time 1788447140.97 total 1257.98 last 98.56) - open/close 46346 (time 1788447150.98 total 1267.98 last 87.23) - open/close 47256 (time 1788447160.99 total 1277.99 last 90.95) - open/close 48211 (time 1788447171.02 total 1288.03 last 95.14) - open/close 49225 (time 1788447181.03 total 1298.03 last 101.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1301.67 seconds: 38.13 ops/second - unlinked 0 (time 1788447189 ; total 0 ; last 0) - unlinked 10000 (time 1788447317 ; total 128 ; last 128) - unlinked 20000 (time 1788447431 ; total 242 ; last 114) - unlinked 30000 (time 1788447498 ; total 309 ; last 67) - unlinked 40000 (time 1788447570 ; total 381 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 451 seconds: 110.046562 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg248-server unloading modules via unload_modules_local on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local oleg248-server: modules unloaded. pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2