== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:31:12 (1785403872) start mds service on oleg221-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg221-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg221-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1785404027.31 total 10.02 last 20.76) - open/close 370 (time 1785404038.30 total 21.00 last 14.75) - open/close 500 (time 1785404048.48 total 31.18 last 12.77) - open/close 600 (time 1785404059.20 total 41.91 last 9.32) - open/close 684 (time 1785404069.71 total 52.42 last 7.99) - open/close 809 (time 1785404080.40 total 63.11 last 11.69) - open/close 971 (time 1785404090.50 total 73.21 last 16.04) - open/close 1127 (time 1785404100.85 total 83.56 last 15.08) - open/close 1253 (time 1785404111.91 total 94.61 last 11.40) - open/close 1336 (time 1785404122.20 total 104.91 last 8.06) - open/close 1470 (time 1785404132.81 total 115.52 last 12.63) - open/close 1623 (time 1785404143.07 total 125.78 last 14.91) - open/close 1760 (time 1785404153.17 total 135.88 last 13.57) - open/close 1875 (time 1785404163.19 total 145.89 last 11.48) - open/close 1968 (time 1785404173.22 total 155.93 last 9.27) - open/close 2110 (time 1785404184.16 total 166.86 last 12.99) - open/close 2174 (time 1785404195.22 total 177.93 last 5.78) - open/close 2354 (time 1785404206.04 total 188.75 last 16.63) - open/close 2419 (time 1785404216.04 total 198.75 last 6.50) - open/close 2579 (time 1785404226.26 total 208.97 last 15.67) - open/close 2721 (time 1785404239.39 total 222.10 last 10.81) - open/close 2904 (time 1785404250.29 total 233.00 last 16.79) - open/close 3003 (time 1785404260.31 total 243.01 last 9.88) - open/close 3151 (time 1785404271.11 total 253.82 last 13.69) - open/close 3261 (time 1785404281.14 total 263.85 last 10.97) - open/close 3419 (time 1785404292.04 total 274.75 last 14.50) - open/close 3538 (time 1785404302.74 total 285.44 last 11.13) - open/close 3667 (time 1785404312.74 total 295.45 last 12.89) - open/close 3796 (time 1785404322.77 total 305.48 last 12.87) - open/close 3943 (time 1785404332.79 total 315.50 last 14.67) - open/close 4122 (time 1785404343.53 total 326.24 last 16.66) - open/close 4255 (time 1785404353.56 total 336.26 last 13.27) - open/close 4460 (time 1785404363.58 total 346.29 last 20.46) - open/close 4597 (time 1785404374.37 total 357.08 last 12.69) - open/close 4782 (time 1785404384.38 total 367.09 last 18.49) - open/close 4948 (time 1785404394.66 total 377.37 last 16.15) - open/close 5120 (time 1785404405.01 total 387.72 last 16.61) - open/close 5275 (time 1785404415.83 total 398.54 last 14.33) - open/close 5431 (time 1785404426.00 total 408.70 last 15.34) - open/close 5589 (time 1785404436.43 total 419.14 last 15.14) - open/close 5718 (time 1785404447.34 total 430.05 last 11.83) - open/close 5842 (time 1785404458.28 total 440.99 last 11.34) - open/close 6029 (time 1785404468.29 total 450.99 last 18.68) - open/close 6170 (time 1785404478.54 total 461.25 last 13.75) - open/close 6272 (time 1785404489.11 total 471.81 last 9.66) - open/close 6369 (time 1785404499.11 total 481.82 last 9.70) - open/close 6459 (time 1785404509.46 total 492.17 last 8.69) - open/close 6600 (time 1785404519.63 total 502.34 last 13.87) - open/close 6702 (time 1785404530.63 total 513.34 last 9.27) - open/close 6807 (time 1785404540.64 total 523.35 last 10.49) - open/close 6940 (time 1785404550.66 total 533.37 last 13.27) - open/close 7102 (time 1785404561.11 total 543.81 last 15.51) - open/close 7244 (time 1785404571.69 total 554.40 last 13.42) - open/close 7377 (time 1785404582.49 total 565.20 last 12.31) - open/close 7464 (time 1785404593.41 total 576.12 last 7.97) - open/close 7619 (time 1785404604.18 total 586.89 last 14.39) - open/close 7724 (time 1785404614.19 total 596.90 last 10.49) - open/close 7873 (time 1785404624.68 total 607.39 last 14.19) - open/close 7994 (time 1785404635.56 total 618.27 last 11.12) - open/close 8147 (time 1785404645.58 total 628.29 last 15.27) - open/close 8285 (time 1785404655.97 total 638.68 last 13.29) - open/close 8479 (time 1785404666.57 total 649.28 last 18.30) - open/close 8592 (time 1785404677.15 total 659.86 last 10.68) - open/close 8678 (time 1785404687.68 total 670.39 last 8.17) - open/close 8819 (time 1785404697.69 total 680.40 last 14.09) - open/close 8996 (time 1785404707.73 total 690.44 last 17.63) - open/close 9153 (time 1785404717.74 total 700.44 last 15.69) - open/close 9251 (time 1785404728.60 total 711.31 last 9.02) - open/close 9430 (time 1785404738.60 total 721.31 last 17.89) - open/close 9521 (time 1785404749.03 total 731.74 last 8.72) - open/close 9678 (time 1785404762.02 total 744.72 last 12.09) - open/close 9768 (time 1785404772.55 total 755.26 last 8.54) - open/close 9883 (time 1785404782.80 total 765.50 last 11.23) - open/close 10000 (time 1785404792.41 total 775.11 last 12.18) - open/close 10047 (time 1785404802.85 total 785.56 last 4.50) - open/close 10176 (time 1785404813.02 total 795.73 last 12.68) - open/close 10300 (time 1785404823.33 total 806.04 last 12.02) - open/close 10446 (time 1785404833.58 total 816.29 last 14.25) - open/close 10640 (time 1785404844.30 total 827.01 last 18.09) - open/close 10741 (time 1785404854.32 total 837.02 last 10.08) - open/close 10918 (time 1785404864.32 total 847.03 last 17.69) - open/close 11035 (time 1785404874.60 total 857.31 last 11.39) - open/close 11229 (time 1785404884.60 total 867.31 last 19.39) - open/close 11343 (time 1785404894.60 total 877.31 last 11.40) - open/close 11468 (time 1785404905.48 total 888.19 last 11.49) - open/close 11618 (time 1785404916.91 total 899.62 last 13.12) - open/close 11712 (time 1785404927.31 total 910.02 last 9.04) - open/close 11830 (time 1785404937.32 total 920.03 last 11.79) - open/close 11989 (time 1785404947.36 total 930.07 last 15.84) - open/close 12128 (time 1785404957.36 total 940.07 last 13.90) - open/close 12309 (time 1785404968.28 total 950.98 last 16.59) - open/close 12409 (time 1785404978.64 total 961.35 last 9.65) - open/close 12507 (time 1785404988.67 total 971.38 last 9.77) - open/close 12591 (time 1785404999.12 total 981.83 last 8.04) - open/close 12759 (time 1785405009.77 total 992.48 last 15.78) - open/close 12873 (time 1785405020.39 total 1003.10 last 10.74) - open/close 13036 (time 1785405030.73 total 1013.44 last 15.76) - open/close 13116 (time 1785405041.29 total 1024.00 last 7.58) - open/close 13234 (time 1785405051.59 total 1034.30 last 11.45) - open/close 13388 (time 1785405061.61 total 1044.32 last 15.38) - open/close 13530 (time 1785405072.30 total 1055.01 last 13.28) - open/close 13655 (time 1785405082.91 total 1065.62 last 11.78) - open/close 13863 (time 1785405093.12 total 1075.83 last 20.37) - open/close 14007 (time 1785405103.77 total 1086.48 last 13.53) - open/close 14146 (time 1785405114.59 total 1097.30 last 12.84) - open/close 14245 (time 1785405124.59 total 1107.30 last 9.90) - open/close 14407 (time 1785405135.35 total 1118.06 last 15.05) - open/close 14501 (time 1785405145.76 total 1128.47 last 9.03) - open/close 14664 (time 1785405156.43 total 1139.14 last 15.27) - open/close 14729 (time 1785405166.46 total 1149.17 last 6.48) - open/close 14819 (time 1785405177.27 total 1159.98 last 8.33) - open/close 14964 (time 1785405187.28 total 1169.99 last 14.48) - open/close 15145 (time 1785405197.31 total 1180.02 last 18.04) - open/close 15378 (time 1785405207.35 total 1190.06 last 23.21) - open/close 15524 (time 1785405217.38 total 1200.08 last 14.56) - open/close 15655 (time 1785405227.40 total 1210.11 last 13.07) - open/close 15789 (time 1785405237.79 total 1220.49 last 12.90) - open/close 15959 (time 1785405248.29 total 1230.99 last 16.19) - open/close 16117 (time 1785405259.37 total 1242.08 last 14.25) - open/close 16259 (time 1785405270.11 total 1252.81 last 13.23) - open/close 16362 (time 1785405280.21 total 1262.92 last 10.19) - open/close 16466 (time 1785405290.25 total 1272.96 last 10.35) - open/close 16651 (time 1785405301.99 total 1284.70 last 15.77) - open/close 16765 (time 1785405312.00 total 1294.71 last 11.38) - open/close 16885 (time 1785405322.02 total 1304.73 last 11.97) - open/close 17042 (time 1785405332.25 total 1314.95 last 15.36) - open/close 17154 (time 1785405342.26 total 1324.97 last 11.19) - open/close 17317 (time 1785405352.79 total 1335.50 last 15.47) - open/close 17460 (time 1785405363.25 total 1345.96 last 13.68) - open/close 17600 (time 1785405373.80 total 1356.51 last 13.27) - open/close 17719 (time 1785405384.20 total 1366.90 last 11.45) - open/close 17840 (time 1785405395.02 total 1377.72 last 11.18) - open/close 17908 (time 1785405406.04 total 1388.75 last 6.17) - open/close 18069 (time 1785405416.05 total 1398.76 last 16.09) - open/close 19021 (time 1785405426.05 total 1408.76 last 95.16) - open/close 19506 (time 1785405436.05 total 1418.76 last 48.50) - open/close 20000 (time 1785405445.41 total 1428.12 last 52.80) - open/close 20395 (time 1785405455.41 total 1438.12 last 39.49) - open/close 21537 (time 1785405465.42 total 1448.12 last 114.13) - open/close 22156 (time 1785405475.43 total 1458.14 last 61.82) - open/close 22658 (time 1785405485.43 total 1468.14 last 50.19) - open/close 23427 (time 1785405495.43 total 1478.14 last 76.89) - open/close 24120 (time 1785405505.44 total 1488.15 last 69.21) - open/close 24689 (time 1785405515.45 total 1498.16 last 56.84) - open/close 25150 (time 1785405525.48 total 1508.19 last 45.97) - open/close 25575 (time 1785405535.50 total 1518.21 last 42.42) - open/close 26113 (time 1785405545.54 total 1528.25 last 53.60) - open/close 26832 (time 1785405555.54 total 1538.25 last 71.86) - open/close 27474 (time 1785405565.55 total 1548.26 last 64.14) - open/close 27907 (time 1785405575.57 total 1558.28 last 43.22) - open/close 28311 (time 1785405585.57 total 1568.28 last 40.40) - open/close 28820 (time 1785405595.59 total 1578.30 last 50.82) - open/close 29307 (time 1785405605.59 total 1588.30 last 48.69) - open/close 29750 (time 1785405615.59 total 1598.30 last 44.29) - open/close 30000 (time 1785405619.97 total 1602.67 last 57.17) - open/close 30481 (time 1785405629.97 total 1612.68 last 48.08) - open/close 31167 (time 1785405639.98 total 1622.69 last 68.50) - open/close 31886 (time 1785405649.99 total 1632.70 last 71.85) - open/close 32695 (time 1785405660.00 total 1642.71 last 80.84) - open/close 33344 (time 1785405670.01 total 1652.72 last 64.81) - open/close 34269 (time 1785405680.01 total 1662.72 last 92.47) - open/close 34942 (time 1785405690.06 total 1672.76 last 67.02) - open/close 35483 (time 1785405700.06 total 1682.77 last 54.05) - open/close 36601 (time 1785405710.07 total 1692.78 last 111.74) - open/close 37215 (time 1785405720.07 total 1702.78 last 61.38) - open/close 37593 (time 1785405730.08 total 1712.79 last 37.78) - open/close 38147 (time 1785405740.08 total 1722.79 last 55.38) - open/close 38878 (time 1785405750.11 total 1732.82 last 72.89) - open/close 39912 (time 1785405760.12 total 1742.82 last 103.35) - open/close 40413 (time 1785405770.14 total 1752.85 last 49.97) - open/close 40984 (time 1785405780.16 total 1762.87 last 56.99) - open/close 41602 (time 1785405790.17 total 1772.88 last 61.76) - open/close 42310 (time 1785405800.17 total 1782.88 last 70.76) - open/close 42957 (time 1785405810.18 total 1792.89 last 64.67) - open/close 43959 (time 1785405820.18 total 1802.89 last 100.15) - open/close 44541 (time 1785405830.20 total 1812.91 last 58.08) - open/close 45089 (time 1785405840.22 total 1822.93 last 54.72) - open/close 45690 (time 1785405850.22 total 1832.93 last 60.09) - open/close 46255 (time 1785405860.23 total 1842.94 last 56.42) - open/close 46833 (time 1785405870.25 total 1852.96 last 57.72) - open/close 47324 (time 1785405880.25 total 1862.96 last 49.09) - open/close 47621 (time 1785405890.27 total 1872.98 last 29.63) - open/close 48004 (time 1785405900.32 total 1883.03 last 38.10) - open/close 48594 (time 1785405910.33 total 1893.04 last 58.95) - open/close 49049 (time 1785405920.33 total 1903.04 last 45.50) - open/close 49487 (time 1785405930.34 total 1913.04 last 43.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1915.33 seconds: 25.91 ops/second - unlinked 0 (time 1785405938 ; total 0 ; last 0) - unlinked 10000 (time 1785406093 ; total 155 ; last 155) - unlinked 20000 (time 1785406261 ; total 323 ; last 168) - unlinked 30000 (time 1785406475 ; total 537 ; last 214) - unlinked 40000 (time 1785406658 ; total 720 ; last 183) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 880 seconds: 56.398865 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: modules unloaded. pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2