== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:08:28 (1786108108) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 120 (time 1786108263.34 total 10.02 last 11.98) - open/close 257 (time 1786108273.39 total 20.06 last 13.64) - open/close 379 (time 1786108283.85 total 30.53 last 11.66) - open/close 491 (time 1786108294.28 total 40.96 last 10.74) - open/close 601 (time 1786108304.96 total 51.64 last 10.30) - open/close 716 (time 1786108315.42 total 62.10 last 11.00) - open/close 814 (time 1786108325.64 total 72.31 last 9.59) - open/close 912 (time 1786108335.80 total 82.48 last 9.64) - open/close 1049 (time 1786108345.80 total 92.48 last 13.70) - open/close 1216 (time 1786108356.31 total 102.98 last 15.89) - open/close 1338 (time 1786108367.18 total 113.86 last 11.22) - open/close 1469 (time 1786108377.88 total 124.55 last 12.25) - open/close 1561 (time 1786108387.88 total 134.56 last 9.19) - open/close 1652 (time 1786108397.90 total 144.57 last 9.09) - open/close 1760 (time 1786108407.90 total 154.58 last 10.80) - open/close 1843 (time 1786108419.52 total 166.19 last 7.14) - open/close 2011 (time 1786108429.56 total 176.23 last 16.74) - open/close 2105 (time 1786108439.58 total 186.26 last 9.37) - open/close 2245 (time 1786108450.59 total 197.27 last 12.72) - open/close 2345 (time 1786108461.64 total 208.32 last 9.05) - open/close 2443 (time 1786108471.66 total 218.33 last 9.79) - open/close 2586 (time 1786108482.39 total 229.06 last 13.32) - open/close 2764 (time 1786108492.40 total 239.07 last 17.78) - open/close 2855 (time 1786108502.86 total 249.54 last 8.70) - open/close 2953 (time 1786108513.03 total 259.71 last 9.64) - open/close 3079 (time 1786108523.34 total 270.02 last 12.22) - open/close 3183 (time 1786108533.36 total 280.04 last 10.38) - open/close 3295 (time 1786108543.36 total 290.04 last 11.20) - open/close 3397 (time 1786108553.38 total 300.05 last 10.19) - open/close 3568 (time 1786108565.95 total 312.63 last 13.60) - open/close 3694 (time 1786108576.20 total 322.87 last 12.30) - open/close 3833 (time 1786108586.39 total 333.06 last 13.64) - open/close 3947 (time 1786108597.39 total 344.06 last 10.36) - open/close 4082 (time 1786108608.27 total 354.94 last 12.41) - open/close 4176 (time 1786108618.28 total 364.95 last 9.39) - open/close 4324 (time 1786108628.97 total 375.65 last 13.84) - open/close 4407 (time 1786108639.24 total 385.91 last 8.09) - open/close 4493 (time 1786108649.68 total 396.35 last 8.24) - open/close 4588 (time 1786108659.69 total 406.36 last 9.49) - open/close 4673 (time 1786108669.70 total 416.38 last 8.49) - open/close 4751 (time 1786108680.40 total 427.07 last 7.29) - open/close 4858 (time 1786108691.43 total 438.11 last 9.70) - open/close 4955 (time 1786108701.46 total 448.13 last 9.68) - open/close 5076 (time 1786108712.09 total 458.76 last 11.38) - open/close 5178 (time 1786108722.95 total 469.63 last 9.39) - open/close 5337 (time 1786108732.96 total 479.64 last 15.88) - open/close 5495 (time 1786108743.49 total 490.16 last 15.01) - open/close 5639 (time 1786108753.54 total 500.22 last 14.32) - open/close 5791 (time 1786108763.99 total 510.67 last 14.54) - open/close 5970 (time 1786108774.00 total 520.67 last 17.89) - open/close 6095 (time 1786108784.06 total 530.74 last 12.42) - open/close 6191 (time 1786108794.97 total 541.65 last 8.80) - open/close 6277 (time 1786108804.98 total 551.66 last 8.59) - open/close 6392 (time 1786108814.99 total 561.66 last 11.49) - open/close 6537 (time 1786108825.01 total 571.69 last 14.47) - open/close 6665 (time 1786108835.47 total 582.14 last 12.24) - open/close 6754 (time 1786108846.23 total 592.90 last 8.27) - open/close 6865 (time 1786108856.32 total 603.00 last 11.00) - open/close 6977 (time 1786108866.63 total 613.31 last 10.86) - open/close 7082 (time 1786108876.65 total 623.32 last 10.48) - open/close 7231 (time 1786108886.67 total 633.34 last 14.87) - open/close 7367 (time 1786108897.49 total 644.17 last 12.56) - open/close 7478 (time 1786108907.50 total 654.17 last 11.09) - open/close 7566 (time 1786108918.50 total 665.18 last 8.00) - open/close 7763 (time 1786108928.51 total 675.18 last 19.69) - open/close 7896 (time 1786108939.52 total 686.19 last 12.08) - open/close 8016 (time 1786108949.72 total 696.39 last 11.76) - open/close 8155 (time 1786108960.56 total 707.24 last 12.82) - open/close 8321 (time 1786108970.97 total 717.64 last 15.95) - open/close 8406 (time 1786108982.44 total 729.12 last 7.41) - open/close 8596 (time 1786108992.53 total 739.21 last 18.83) - open/close 8749 (time 1786109003.20 total 749.88 last 14.33) - open/close 8905 (time 1786109013.64 total 760.31 last 14.95) - open/close 9012 (time 1786109023.74 total 770.42 last 10.59) - open/close 9119 (time 1786109033.75 total 780.42 last 10.69) - open/close 9224 (time 1786109043.75 total 790.43 last 10.50) - open/close 9334 (time 1786109053.89 total 800.57 last 10.85) - open/close 9563 (time 1786109064.98 total 811.66 last 20.65) - open/close 9772 (time 1786109075.59 total 822.26 last 19.70) - open/close 9937 (time 1786109085.89 total 832.57 last 16.01) - open/close 10000 (time 1786109091.41 total 838.08 last 11.43) - open/close 10170 (time 1786109101.41 total 848.09 last 16.99) - open/close 10263 (time 1786109111.57 total 858.24 last 9.16) - open/close 10471 (time 1786109121.58 total 868.26 last 20.77) - open/close 10618 (time 1786109131.61 total 878.29 last 14.66) - open/close 10773 (time 1786109142.16 total 888.83 last 14.70) - open/close 10967 (time 1786109154.14 total 900.81 last 16.19) - open/close 11138 (time 1786109164.81 total 911.49 last 16.03) - open/close 11384 (time 1786109175.65 total 922.32 last 22.70) - open/close 11643 (time 1786109185.67 total 932.34 last 25.85) - open/close 11808 (time 1786109195.67 total 942.35 last 16.49) - open/close 11936 (time 1786109206.20 total 952.87 last 12.16) - open/close 12153 (time 1786109216.76 total 963.43 last 20.55) - open/close 12420 (time 1786109226.76 total 973.44 last 26.69) - open/close 12667 (time 1786109237.06 total 983.73 last 23.99) - open/close 12938 (time 1786109247.88 total 994.56 last 25.05) - open/close 13179 (time 1786109257.90 total 1004.58 last 24.04) - open/close 13331 (time 1786109268.26 total 1014.94 last 14.67) - open/close 13439 (time 1786109278.27 total 1024.95 last 10.79) - open/close 13536 (time 1786109288.53 total 1035.20 last 9.46) - open/close 13689 (time 1786109298.53 total 1045.20 last 15.30) - open/close 13831 (time 1786109308.53 total 1055.20 last 14.20) - open/close 13928 (time 1786109318.79 total 1065.47 last 9.45) - open/close 14026 (time 1786109329.56 total 1076.23 last 9.10) - open/close 14150 (time 1786109340.42 total 1087.10 last 11.41) - open/close 14268 (time 1786109351.12 total 1097.80 last 11.03) - open/close 14447 (time 1786109361.54 total 1108.22 last 17.18) - open/close 14559 (time 1786109371.91 total 1118.59 last 10.80) - open/close 14665 (time 1786109382.12 total 1128.80 last 10.38) - open/close 14730 (time 1786109392.40 total 1139.07 last 6.33) - open/close 14812 (time 1786109403.36 total 1150.04 last 7.48) - open/close 14967 (time 1786109413.85 total 1160.52 last 14.78) - open/close 15080 (time 1786109424.03 total 1170.70 last 11.10) - open/close 15245 (time 1786109434.05 total 1180.72 last 16.47) - open/close 15373 (time 1786109444.59 total 1191.26 last 12.14) - open/close 15438 (time 1786109454.73 total 1201.41 last 6.41) - open/close 15614 (time 1786109465.56 total 1212.23 last 16.26) - open/close 15760 (time 1786109476.05 total 1222.72 last 13.92) - open/close 15945 (time 1786109487.07 total 1233.74 last 16.79) - open/close 16106 (time 1786109497.75 total 1244.42 last 15.07) - open/close 16207 (time 1786109507.77 total 1254.45 last 10.07) - open/close 16294 (time 1786109518.53 total 1265.21 last 8.09) - open/close 16350 (time 1786109528.54 total 1275.22 last 5.60) - open/close 16561 (time 1786109538.55 total 1285.22 last 21.08) - open/close 17351 (time 1786109548.56 total 1295.24 last 78.89) - open/close 18046 (time 1786109558.56 total 1305.24 last 69.50) - open/close 18693 (time 1786109568.57 total 1315.24 last 64.69) - open/close 19396 (time 1786109578.57 total 1325.24 last 70.27) - open/close 20000 (time 1786109587.32 total 1334.00 last 69.00) - open/close 20648 (time 1786109597.33 total 1344.00 last 64.76) - open/close 21417 (time 1786109607.33 total 1354.00 last 76.89) - open/close 22084 (time 1786109617.33 total 1364.01 last 66.67) - open/close 22756 (time 1786109627.35 total 1374.03 last 67.08) - open/close 23412 (time 1786109637.36 total 1384.03 last 65.57) - open/close 24092 (time 1786109647.37 total 1394.04 last 67.93) - open/close 24757 (time 1786109657.38 total 1404.05 last 66.44) - open/close 25455 (time 1786109667.38 total 1414.05 last 69.78) - open/close 26188 (time 1786109677.38 total 1424.06 last 73.27) - open/close 26988 (time 1786109687.39 total 1434.07 last 79.93) - open/close 27813 (time 1786109697.39 total 1444.07 last 82.48) - open/close 28657 (time 1786109707.40 total 1454.08 last 84.31) - open/close 29458 (time 1786109717.42 total 1464.09 last 80.01) - open/close 30000 (time 1786109724.27 total 1470.95 last 79.04) - open/close 30678 (time 1786109734.27 total 1480.95 last 67.78) - open/close 31454 (time 1786109744.29 total 1490.97 last 77.47) - open/close 32236 (time 1786109754.29 total 1500.97 last 78.20) - open/close 33048 (time 1786109764.31 total 1510.98 last 81.07) - open/close 33876 (time 1786109774.31 total 1520.98 last 82.79) - open/close 34637 (time 1786109784.31 total 1530.99 last 76.07) - open/close 35390 (time 1786109794.31 total 1540.99 last 75.28) - open/close 36187 (time 1786109804.32 total 1551.00 last 79.63) - open/close 36911 (time 1786109814.34 total 1561.01 last 72.31) - open/close 37693 (time 1786109824.35 total 1571.02 last 78.11) - open/close 38469 (time 1786109834.36 total 1581.03 last 77.52) - open/close 39305 (time 1786109844.37 total 1591.04 last 83.51) - open/close 40000 (time 1786109852.57 total 1599.24 last 84.75) - open/close 40863 (time 1786109862.57 total 1609.25 last 86.26) - open/close 41643 (time 1786109872.58 total 1619.25 last 77.96) - open/close 42401 (time 1786109882.58 total 1629.25 last 75.78) - open/close 43237 (time 1786109892.58 total 1639.26 last 83.56) - open/close 44059 (time 1786109902.60 total 1649.27 last 82.11) - open/close 44896 (time 1786109912.60 total 1659.27 last 83.70) - open/close 45691 (time 1786109922.61 total 1669.28 last 79.40) - open/close 46503 (time 1786109932.61 total 1679.29 last 81.18) - open/close 47322 (time 1786109942.62 total 1689.29 last 81.86) - open/close 48136 (time 1786109952.62 total 1699.29 last 81.38) - open/close 48912 (time 1786109962.62 total 1709.30 last 77.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1718.06 seconds: 28.89 ops/second - unlinked 0 (time 1786109977 ; total 0 ; last 0) - unlinked 10000 (time 1786110108 ; total 131 ; last 131) - unlinked 20000 (time 1786110241 ; total 264 ; last 133) - unlinked 30000 (time 1786110365 ; total 388 ; last 124) - unlinked 40000 (time 1786110498 ; total 521 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 639 seconds: 77.669800 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2