== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:01:52 (1787702512) start mds service on oleg449-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: MODOPTS_LIBCFS= oleg449-server: Force libcfs to create 2 CPU partitions oleg449-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg449-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg449-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg449-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg449-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1787702637.46 total 10.10 last 14.06) - open/close 222 (time 1787702647.83 total 20.48 last 7.71) - open/close 325 (time 1787702658.26 total 30.90 last 9.88) - open/close 573 (time 1787702668.87 total 41.52 last 23.36) - open/close 748 (time 1787702679.76 total 52.40 last 16.08) - open/close 930 (time 1787702689.77 total 62.42 last 18.17) - open/close 1070 (time 1787702699.77 total 72.42 last 14.00) - open/close 1165 (time 1787702710.08 total 82.72 last 9.22) - open/close 1409 (time 1787702720.08 total 92.73 last 24.39) - open/close 1628 (time 1787702730.25 total 102.89 last 21.54) - open/close 1837 (time 1787702741.12 total 113.77 last 19.22) - open/close 1969 (time 1787702751.39 total 124.04 last 12.86) - open/close 2167 (time 1787702761.43 total 134.07 last 19.73) - open/close 2282 (time 1787702771.73 total 144.37 last 11.17) - open/close 2404 (time 1787702783.42 total 156.07 last 10.43) - open/close 2596 (time 1787702793.45 total 166.09 last 19.15) - open/close 2793 (time 1787702803.46 total 176.10 last 19.68) - open/close 2976 (time 1787702814.91 total 187.56 last 15.97) - open/close 3217 (time 1787702824.96 total 197.61 last 23.98) - open/close 3434 (time 1787702836.04 total 208.69 last 19.59) - open/close 3680 (time 1787702846.05 total 218.70 last 24.56) - open/close 3882 (time 1787702856.38 total 229.02 last 19.57) - open/close 4050 (time 1787702866.83 total 239.47 last 16.07) - open/close 4267 (time 1787702878.48 total 251.13 last 18.62) - open/close 4388 (time 1787702888.50 total 261.14 last 12.08) - open/close 4518 (time 1787702898.62 total 271.26 last 12.85) - open/close 4757 (time 1787702908.62 total 281.27 last 23.89) - open/close 5020 (time 1787702919.50 total 292.14 last 24.19) - open/close 5168 (time 1787702929.93 total 302.58 last 14.18) - open/close 5304 (time 1787702942.36 total 315.00 last 10.95) - open/close 5428 (time 1787702953.27 total 325.92 last 11.36) - open/close 5582 (time 1787702963.29 total 335.94 last 15.37) - open/close 5804 (time 1787702973.51 total 346.16 last 21.72) - open/close 5947 (time 1787702983.53 total 356.17 last 14.28) - open/close 6079 (time 1787702993.54 total 366.19 last 13.18) - open/close 6188 (time 1787703004.60 total 377.24 last 9.86) - open/close 6352 (time 1787703014.71 total 387.36 last 16.21) - open/close 6581 (time 1787703025.40 total 398.05 last 21.42) - open/close 6710 (time 1787703038.41 total 411.06 last 9.91) - open/close 6844 (time 1787703049.35 total 422.00 last 12.25) - open/close 6980 (time 1787703061.46 total 434.10 last 11.24) - open/close 7187 (time 1787703071.90 total 444.54 last 19.82) - open/close 7372 (time 1787703088.65 total 461.29 last 11.05) - open/close 7504 (time 1787703098.66 total 471.30 last 13.19) - open/close 7716 (time 1787703109.19 total 481.83 last 20.13) - open/close 7850 (time 1787703119.21 total 491.86 last 13.37) - open/close 8031 (time 1787703129.22 total 501.86 last 18.09) - open/close 8122 (time 1787703139.23 total 511.88 last 9.09) - open/close 8252 (time 1787703149.23 total 521.88 last 13.00) - open/close 8445 (time 1787703159.23 total 531.88 last 19.30) - open/close 8514 (time 1787703169.25 total 541.90 last 6.89) - open/close 8676 (time 1787703179.64 total 552.29 last 15.59) - open/close 8740 (time 1787703190.31 total 562.95 last 6.00) - open/close 8979 (time 1787703200.97 total 573.61 last 22.42) - open/close 9130 (time 1787703211.43 total 584.08 last 14.43) - open/close 9321 (time 1787703223.04 total 595.68 last 16.46) - open/close 9460 (time 1787703233.92 total 606.57 last 12.77) - open/close 9751 (time 1787703244.49 total 617.14 last 27.53) - open/close 9985 (time 1787703255.24 total 627.88 last 21.77) - open/close 10141 (time 1787703265.47 total 638.12 last 15.25) - open/close 10300 (time 1787703275.58 total 648.23 last 15.73) - open/close 10494 (time 1787703285.59 total 658.24 last 19.37) - open/close 10681 (time 1787703296.50 total 669.15 last 17.14) - open/close 10799 (time 1787703308.42 total 681.07 last 9.90) - open/close 10919 (time 1787703319.28 total 691.93 last 11.05) - open/close 11082 (time 1787703329.93 total 702.58 last 15.30) - open/close 11313 (time 1787703340.93 total 713.58 last 21.01) - open/close 11546 (time 1787703351.75 total 724.40 last 21.53) - open/close 11750 (time 1787703362.11 total 734.75 last 19.70) - open/close 11925 (time 1787703372.12 total 744.77 last 17.47) - open/close 12058 (time 1787703382.41 total 755.06 last 12.93) - open/close 12296 (time 1787703392.42 total 765.06 last 23.78) - open/close 12439 (time 1787703405.39 total 778.03 last 11.03) - open/close 12551 (time 1787703416.01 total 788.65 last 10.55) - open/close 12657 (time 1787703427.28 total 799.92 last 9.41) - open/close 12787 (time 1787703437.34 total 809.99 last 12.92) - open/close 13029 (time 1787703448.01 total 820.65 last 22.69) - open/close 13214 (time 1787703458.56 total 831.20 last 17.54) - open/close 13362 (time 1787703468.56 total 841.20 last 14.80) - open/close 13522 (time 1787703478.92 total 851.57 last 15.44) - open/close 13715 (time 1787703490.01 total 862.66 last 17.40) - open/close 13902 (time 1787703500.02 total 872.67 last 18.68) - open/close 14087 (time 1787703510.04 total 882.68 last 18.48) - open/close 14237 (time 1787703520.57 total 893.22 last 14.24) - open/close 14534 (time 1787703530.57 total 903.22 last 29.69) - open/close 14713 (time 1787703540.74 total 913.38 last 17.61) - open/close 14910 (time 1787703550.76 total 923.41 last 19.65) - open/close 15191 (time 1787703560.77 total 933.42 last 28.07) - open/close 15231 (time 1787703570.91 total 943.56 last 3.94) - open/close 15368 (time 1787703580.92 total 953.56 last 13.69) - open/close 15567 (time 1787703591.31 total 963.95 last 19.16) - open/close 15635 (time 1787703602.07 total 974.72 last 6.32) - open/close 15819 (time 1787703612.92 total 985.57 last 16.96) - open/close 16030 (time 1787703623.18 total 995.83 last 20.57) - open/close 16223 (time 1787703633.19 total 1005.83 last 19.29) - open/close 16371 (time 1787703644.09 total 1016.74 last 13.57) - open/close 16525 (time 1787703654.40 total 1027.04 last 14.94) - open/close 16654 (time 1787703664.40 total 1037.04 last 12.90) - open/close 16833 (time 1787703675.20 total 1047.84 last 16.58) - open/close 17227 (time 1787703685.22 total 1057.87 last 39.31) - open/close 17863 (time 1787703695.23 total 1067.88 last 63.52) - open/close 18573 (time 1787703705.24 total 1077.89 last 70.94) - open/close 19267 (time 1787703715.24 total 1087.89 last 69.37) - open/close 20000 (time 1787703725.19 total 1097.84 last 73.67) - open/close 20575 (time 1787703735.20 total 1107.85 last 57.46) - open/close 21164 (time 1787703745.20 total 1117.85 last 58.90) - open/close 21786 (time 1787703755.20 total 1127.85 last 62.19) - open/close 22397 (time 1787703765.21 total 1137.86 last 61.07) - open/close 23107 (time 1787703775.21 total 1147.86 last 71.00) - open/close 23815 (time 1787703785.21 total 1157.86 last 70.78) - open/close 24565 (time 1787703795.21 total 1167.86 last 74.99) - open/close 25170 (time 1787703805.21 total 1177.86 last 60.50) - open/close 25745 (time 1787703815.22 total 1187.87 last 57.46) - open/close 26178 (time 1787703825.22 total 1197.87 last 43.30) - open/close 26682 (time 1787703835.23 total 1207.87 last 50.38) - open/close 27280 (time 1787703845.23 total 1217.87 last 59.79) - open/close 27903 (time 1787703855.23 total 1227.88 last 62.28) - open/close 28496 (time 1787703865.24 total 1237.89 last 59.25) - open/close 29127 (time 1787703875.25 total 1247.90 last 63.04) - open/close 29748 (time 1787703885.28 total 1257.93 last 61.91) - open/close 30000 (time 1787703889.78 total 1262.42 last 56.06) - open/close 30602 (time 1787703899.79 total 1272.43 last 60.12) - open/close 31295 (time 1787703909.80 total 1282.45 last 69.21) - open/close 31981 (time 1787703919.81 total 1292.46 last 68.54) - open/close 32588 (time 1787703929.85 total 1302.49 last 60.48) - open/close 33255 (time 1787703939.86 total 1312.50 last 66.63) - open/close 33939 (time 1787703949.86 total 1322.50 last 68.40) - open/close 34680 (time 1787703959.87 total 1332.51 last 74.02) - open/close 35343 (time 1787703969.87 total 1342.52 last 66.26) - open/close 35965 (time 1787703979.88 total 1352.53 last 62.16) - open/close 36659 (time 1787703989.90 total 1362.55 last 69.28) - open/close 37310 (time 1787703999.91 total 1372.56 last 65.04) - open/close 37960 (time 1787704009.91 total 1382.56 last 64.98) - open/close 38614 (time 1787704019.92 total 1392.57 last 65.35) - open/close 39379 (time 1787704029.92 total 1402.57 last 76.50) - open/close 40000 (time 1787704038.87 total 1411.52 last 69.36) - open/close 40758 (time 1787704048.89 total 1421.54 last 75.65) - open/close 41470 (time 1787704058.90 total 1431.55 last 71.14) - open/close 42179 (time 1787704068.91 total 1441.56 last 70.84) - open/close 42920 (time 1787704078.91 total 1451.56 last 74.08) - open/close 43682 (time 1787704088.92 total 1461.57 last 76.14) - open/close 44344 (time 1787704098.93 total 1471.57 last 66.17) - open/close 44964 (time 1787704108.93 total 1481.58 last 61.95) - open/close 45644 (time 1787704118.94 total 1491.59 last 67.93) - open/close 46368 (time 1787704128.97 total 1501.61 last 72.23) - open/close 47144 (time 1787704138.97 total 1511.62 last 77.54) - open/close 47877 (time 1787704148.99 total 1521.63 last 73.22) - open/close 48610 (time 1787704158.99 total 1531.64 last 73.25) - open/close 49307 (time 1787704169.00 total 1541.65 last 69.64) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1545.67 seconds: 32.11 ops/second - unlinked 0 (time 1787704177 ; total 0 ; last 0) - unlinked 10000 (time 1787704320 ; total 143 ; last 143) - unlinked 20000 (time 1787704470 ; total 293 ; last 150) - unlinked 30000 (time 1787704612 ; total 435 ; last 142) - unlinked 40000 (time 1787704764 ; total 587 ; last 152) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 727 seconds: 68.268227 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server unloading modules via unload_modules_local on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local oleg449-server: modules unloaded. pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2