== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:32:19 (1787603539) start mds service on oleg616-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg616-server' oleg616-server: oleg616-server.virtnet: executing load_modules_local oleg616-server: Loading modules from /home/green/git/lustre-release/lustre oleg616-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg616-server: detected 4 online CPUs by sysfs oleg616-server: MODOPTS_LIBCFS= oleg616-server: Force libcfs to create 2 CPU partitions oleg616-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg616-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg616-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg616-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg616-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 303 (time 1787603638.51 total 10.01 last 30.28) - open/close 556 (time 1787603648.73 total 20.23 last 24.76) - open/close 827 (time 1787603658.73 total 30.23 last 27.09) - open/close 1158 (time 1787603668.74 total 40.24 last 33.08) - open/close 1384 (time 1787603679.11 total 50.61 last 21.78) - open/close 1640 (time 1787603689.12 total 60.61 last 25.60) - open/close 1805 (time 1787603699.32 total 70.82 last 16.16) - open/close 2007 (time 1787603710.32 total 81.82 last 18.37) - open/close 2104 (time 1787603720.58 total 92.08 last 9.46) - open/close 2364 (time 1787603731.08 total 102.57 last 24.77) - open/close 2555 (time 1787603741.09 total 112.59 last 19.07) - open/close 2727 (time 1787603751.22 total 122.72 last 16.98) - open/close 3005 (time 1787603761.66 total 133.15 last 26.64) - open/close 3270 (time 1787603771.83 total 143.33 last 26.05) - open/close 3546 (time 1787603782.20 total 153.70 last 26.61) - open/close 3828 (time 1787603793.07 total 164.57 last 25.95) - open/close 4089 (time 1787603803.09 total 174.58 last 26.06) - open/close 4389 (time 1787603813.62 total 185.12 last 28.47) - open/close 4681 (time 1787603824.04 total 195.54 last 28.03) - open/close 4953 (time 1787603834.05 total 205.55 last 27.17) - open/close 5175 (time 1787603846.13 total 217.63 last 18.38) - open/close 5484 (time 1787603856.90 total 228.40 last 28.68) - open/close 5834 (time 1787603867.92 total 239.42 last 31.75) - open/close 6045 (time 1787603878.24 total 249.74 last 20.45) - open/close 6334 (time 1787603889.07 total 260.57 last 26.69) - open/close 6620 (time 1787603899.08 total 270.58 last 28.56) - open/close 6883 (time 1787603909.09 total 280.59 last 26.27) - open/close 7024 (time 1787603919.67 total 291.16 last 13.33) - open/close 7250 (time 1787603929.67 total 301.17 last 22.60) - open/close 7540 (time 1787603939.95 total 311.45 last 28.21) - open/close 7650 (time 1787603950.59 total 322.09 last 10.34) - open/close 7841 (time 1787603961.55 total 333.04 last 17.43) - open/close 7982 (time 1787603971.55 total 343.04 last 14.10) - open/close 8054 (time 1787603981.68 total 353.17 last 7.11) - open/close 8226 (time 1787603992.31 total 363.81 last 16.17) - open/close 8506 (time 1787604002.43 total 373.92 last 27.69) - open/close 8718 (time 1787604013.13 total 384.63 last 19.80) - open/close 9071 (time 1787604023.99 total 395.49 last 32.51) - open/close 9349 (time 1787604034.74 total 406.24 last 25.85) - open/close 9574 (time 1787604044.75 total 416.25 last 22.49) - open/close 9828 (time 1787604055.11 total 426.61 last 24.52) - open/close 10000 (time 1787604061.74 total 433.23 last 25.96) - open/close 10205 (time 1787604072.12 total 443.61 last 19.74) - open/close 10479 (time 1787604083.07 total 454.57 last 25.02) - open/close 10712 (time 1787604093.88 total 465.37 last 21.56) - open/close 11013 (time 1787604104.43 total 475.93 last 28.51) - open/close 11304 (time 1787604114.75 total 486.25 last 28.21) - open/close 11561 (time 1787604127.22 total 498.71 last 20.61) - open/close 11787 (time 1787604138.24 total 509.74 last 20.50) - open/close 11983 (time 1787604149.12 total 520.62 last 18.02) - open/close 12278 (time 1787604160.12 total 531.61 last 26.82) - open/close 12513 (time 1787604170.33 total 541.83 last 23.01) - open/close 12770 (time 1787604180.42 total 551.91 last 25.48) - open/close 13040 (time 1787604190.54 total 562.03 last 26.68) - open/close 13241 (time 1787604200.82 total 572.32 last 19.54) - open/close 13507 (time 1787604211.09 total 582.59 last 25.90) - open/close 13773 (time 1787604221.57 total 593.07 last 25.38) - open/close 14100 (time 1787604231.81 total 603.31 last 31.93) - open/close 14323 (time 1787604241.81 total 613.31 last 22.30) - open/close 14588 (time 1787604252.29 total 623.79 last 25.28) - open/close 14734 (time 1787604262.44 total 633.93 last 14.39) - open/close 15044 (time 1787604273.03 total 644.53 last 29.27) - open/close 15319 (time 1787604283.13 total 654.63 last 27.22) - open/close 15538 (time 1787604293.16 total 664.65 last 21.85) - open/close 15714 (time 1787604303.18 total 674.67 last 17.57) - open/close 15970 (time 1787604313.30 total 684.80 last 25.29) - open/close 16288 (time 1787604323.31 total 694.80 last 31.78) - open/close 16467 (time 1787604333.31 total 704.81 last 17.89) - open/close 16734 (time 1787604343.76 total 715.25 last 25.56) - open/close 17016 (time 1787604353.76 total 725.26 last 28.18) - open/close 17273 (time 1787604363.77 total 735.26 last 25.69) - open/close 17576 (time 1787604374.45 total 745.95 last 28.35) - open/close 17746 (time 1787604384.46 total 755.96 last 16.98) - open/close 18078 (time 1787604395.40 total 766.90 last 30.35) - open/close 18322 (time 1787604405.56 total 777.05 last 24.03) - open/close 18578 (time 1787604415.67 total 787.16 last 25.32) - open/close 18794 (time 1787604426.96 total 798.45 last 19.13) - open/close 19075 (time 1787604439.74 total 811.23 last 21.99) - open/close 19367 (time 1787604450.61 total 822.11 last 26.85) - open/close 19614 (time 1787604460.62 total 832.11 last 24.69) - open/close 19829 (time 1787604470.63 total 842.13 last 21.48) - open/close 19984 (time 1787604480.63 total 852.13 last 15.50) - open/close 20290 (time 1787604491.59 total 863.09 last 27.92) - open/close 20556 (time 1787604502.07 total 873.57 last 25.38) - open/close 20891 (time 1787604512.56 total 884.06 last 31.93) - open/close 21162 (time 1787604523.12 total 894.62 last 25.67) - open/close 21293 (time 1787604533.71 total 905.21 last 12.37) - open/close 21615 (time 1787604543.80 total 915.29 last 31.94) - open/close 22117 (time 1787604553.80 total 925.30 last 50.19) - open/close 23048 (time 1787604563.81 total 935.30 last 93.02) - open/close 24085 (time 1787604573.81 total 945.31 last 103.65) - open/close 25063 (time 1787604583.82 total 955.32 last 97.67) - open/close 26014 (time 1787604593.84 total 965.33 last 94.99) - open/close 27028 (time 1787604603.84 total 975.34 last 101.38) - open/close 27960 (time 1787604613.85 total 985.35 last 93.06) - open/close 28755 (time 1787604623.86 total 995.35 last 79.46) - open/close 29612 (time 1787604633.87 total 1005.36 last 85.63) - open/close 30000 (time 1787604638.73 total 1010.22 last 79.86) - open/close 30735 (time 1787604648.73 total 1020.23 last 73.46) - open/close 31568 (time 1787604658.75 total 1030.24 last 83.17) - open/close 32326 (time 1787604668.75 total 1040.24 last 75.78) - open/close 33258 (time 1787604678.75 total 1050.25 last 93.14) - open/close 34158 (time 1787604688.76 total 1060.25 last 89.99) - open/close 35127 (time 1787604698.76 total 1070.26 last 96.85) - open/close 35788 (time 1787604708.78 total 1080.27 last 66.00) - open/close 36645 (time 1787604718.78 total 1090.28 last 85.63) - open/close 37643 (time 1787604728.80 total 1100.30 last 99.66) - open/close 38543 (time 1787604738.80 total 1110.30 last 89.99) - open/close 39493 (time 1787604748.81 total 1120.30 last 94.94) - open/close 40000 (time 1787604754.74 total 1126.23 last 85.50) - open/close 40952 (time 1787604764.74 total 1136.24 last 95.14) - open/close 41851 (time 1787604774.75 total 1146.25 last 89.83) - open/close 42781 (time 1787604784.76 total 1156.25 last 92.94) - open/close 43833 (time 1787604794.76 total 1166.26 last 105.15) - open/close 44704 (time 1787604804.76 total 1176.26 last 87.09) - open/close 45641 (time 1787604814.76 total 1186.26 last 93.70) - open/close 46588 (time 1787604824.76 total 1196.26 last 94.69) - open/close 47501 (time 1787604834.77 total 1206.26 last 91.28) - open/close 48486 (time 1787604844.77 total 1216.27 last 98.45) - open/close 49465 (time 1787604854.77 total 1226.27 last 97.86) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1227.80 seconds: 40.42 ops/second - unlinked 0 (time 1787604860 ; total 0 ; last 0) - unlinked 10000 (time 1787604968 ; total 108 ; last 108) - unlinked 20000 (time 1787605081 ; total 221 ; last 113) - unlinked 30000 (time 1787605163 ; total 303 ; last 82) - unlinked 40000 (time 1787605225 ; total 365 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 426 seconds: 116.504692 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg616-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg616-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg616-server stop mds service on oleg616-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg616-server stop mds service on oleg616-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg616-server unloading modules via unload_modules_local on: 'oleg616-server' oleg616-server: oleg616-server.virtnet: executing unload_modules_local oleg616-server: modules unloaded. pdsh@oleg616-client: oleg616-client: ssh exited with exit code 2 pdsh@oleg616-client: oleg616-server: ssh exited with exit code 2 pdsh@oleg616-client: oleg616-client: ssh exited with exit code 2 pdsh@oleg616-client: oleg616-server: ssh exited with exit code 2