== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:41:32 (1786693292) start mds service on oleg409-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg409-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg409-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg409-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg409-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 92 (time 1786693426.26 total 10.54 last 8.73) - open/close 199 (time 1786693437.14 total 21.41 last 9.84) - open/close 324 (time 1786693447.51 total 31.78 last 12.05) - open/close 443 (time 1786693458.14 total 42.42 last 11.19) - open/close 572 (time 1786693470.86 total 55.14 last 10.14) - open/close 748 (time 1786693481.06 total 65.33 last 17.26) - open/close 929 (time 1786693492.11 total 76.39 last 16.37) - open/close 1039 (time 1786693502.11 total 86.39 last 11.00) - open/close 1167 (time 1786693514.51 total 98.79 last 10.32) - open/close 1318 (time 1786693524.69 total 108.96 last 14.84) - open/close 1469 (time 1786693534.99 total 119.27 last 14.65) - open/close 1667 (time 1786693545.01 total 129.28 last 19.78) - open/close 1747 (time 1786693555.89 total 140.16 last 7.35) - open/close 1934 (time 1786693566.30 total 150.58 last 17.95) - open/close 2146 (time 1786693576.79 total 161.07 last 20.21) - open/close 2267 (time 1786693587.34 total 171.62 last 11.46) - open/close 2487 (time 1786693597.77 total 182.04 last 21.11) - open/close 2653 (time 1786693607.77 total 192.05 last 16.60) - open/close 2750 (time 1786693618.20 total 202.48 last 9.30) - open/close 2866 (time 1786693629.26 total 213.54 last 10.49) - open/close 3039 (time 1786693639.27 total 223.55 last 17.29) - open/close 3143 (time 1786693649.57 total 233.85 last 10.09) - open/close 3277 (time 1786693660.05 total 244.33 last 12.79) - open/close 3408 (time 1786693670.07 total 254.35 last 13.07) - open/close 3586 (time 1786693680.10 total 264.38 last 17.75) - open/close 3724 (time 1786693691.00 total 275.28 last 12.67) - open/close 3855 (time 1786693702.15 total 286.43 last 11.75) - open/close 4121 (time 1786693712.28 total 296.56 last 26.26) - open/close 4316 (time 1786693722.29 total 306.56 last 19.49) - open/close 4502 (time 1786693732.30 total 316.58 last 18.56) - open/close 4627 (time 1786693742.92 total 327.20 last 11.77) - open/close 4772 (time 1786693754.98 total 339.26 last 12.02) - open/close 4901 (time 1786693765.91 total 350.19 last 11.81) - open/close 5023 (time 1786693776.79 total 361.07 last 11.21) - open/close 5167 (time 1786693787.60 total 371.88 last 13.32) - open/close 5451 (time 1786693797.64 total 381.92 last 28.29) - open/close 5729 (time 1786693808.74 total 393.02 last 25.05) - open/close 6009 (time 1786693819.80 total 404.08 last 25.32) - open/close 6257 (time 1786693829.81 total 414.09 last 24.77) - open/close 6409 (time 1786693840.15 total 424.42 last 14.71) - open/close 6662 (time 1786693850.15 total 434.43 last 25.29) - open/close 6778 (time 1786693860.26 total 444.53 last 11.48) - open/close 6959 (time 1786693870.73 total 455.01 last 17.28) - open/close 7101 (time 1786693880.74 total 465.02 last 14.19) - open/close 7283 (time 1786693890.75 total 475.03 last 18.17) - open/close 7514 (time 1786693901.32 total 485.60 last 21.86) - open/close 7669 (time 1786693913.61 total 497.88 last 12.62) - open/close 7790 (time 1786693923.62 total 507.89 last 12.09) - open/close 7870 (time 1786693934.48 total 518.76 last 7.36) - open/close 8102 (time 1786693944.66 total 528.94 last 22.79) - open/close 8354 (time 1786693954.67 total 538.95 last 25.17) - open/close 8502 (time 1786693965.52 total 549.80 last 13.64) - open/close 8594 (time 1786693975.56 total 559.84 last 9.16) - open/close 8761 (time 1786693988.17 total 572.45 last 13.25) - open/close 8936 (time 1786693998.19 total 582.47 last 17.47) - open/close 9168 (time 1786694009.10 total 593.38 last 21.26) - open/close 9368 (time 1786694019.11 total 603.39 last 19.99) - open/close 9528 (time 1786694029.53 total 613.80 last 15.36) - open/close 9753 (time 1786694040.06 total 624.33 last 21.37) - open/close 10000 (time 1786694049.85 total 634.12 last 25.23) - open/close 10198 (time 1786694060.70 total 644.97 last 18.25) - open/close 10307 (time 1786694071.74 total 656.02 last 9.87) - open/close 10413 (time 1786694082.13 total 666.41 last 10.20) - open/close 10648 (time 1786694094.83 total 679.10 last 18.51) - open/close 10866 (time 1786694107.64 total 691.92 last 17.02) - open/close 11038 (time 1786694118.10 total 702.37 last 16.44) - open/close 11178 (time 1786694128.10 total 712.38 last 13.99) - open/close 11328 (time 1786694138.83 total 723.11 last 13.98) - open/close 11397 (time 1786694149.15 total 733.43 last 6.69) - open/close 11465 (time 1786694159.27 total 743.55 last 6.72) - open/close 11574 (time 1786694169.30 total 753.58 last 10.87) - open/close 11647 (time 1786694180.13 total 764.40 last 6.74) - open/close 11770 (time 1786694191.95 total 776.23 last 10.40) - open/close 11918 (time 1786694202.32 total 786.60 last 14.27) - open/close 12019 (time 1786694212.91 total 797.19 last 9.54) - open/close 12143 (time 1786694223.12 total 807.39 last 12.15) - open/close 12280 (time 1786694233.43 total 817.71 last 13.28) - open/close 12440 (time 1786694244.18 total 828.46 last 14.88) - open/close 12595 (time 1786694255.20 total 839.47 last 14.08) - open/close 12676 (time 1786694268.56 total 852.84 last 6.06) - open/close 12852 (time 1786694279.57 total 863.85 last 15.99) - open/close 12927 (time 1786694289.58 total 873.86 last 7.49) - open/close 13125 (time 1786694300.04 total 884.32 last 18.94) - open/close 13285 (time 1786694310.76 total 895.04 last 14.93) - open/close 13414 (time 1786694320.78 total 905.05 last 12.88) - open/close 13526 (time 1786694332.78 total 917.05 last 9.33) - open/close 13598 (time 1786694343.82 total 928.10 last 6.52) - open/close 13639 (time 1786694354.21 total 938.49 last 3.95) - open/close 13733 (time 1786694364.64 total 948.91 last 9.02) - open/close 13780 (time 1786694374.89 total 959.16 last 4.59) - open/close 13875 (time 1786694385.68 total 969.96 last 8.80) - open/close 14001 (time 1786694396.43 total 980.71 last 11.72) - open/close 14065 (time 1786694407.53 total 991.81 last 5.77) - open/close 14187 (time 1786694418.00 total 1002.28 last 11.65) - open/close 14315 (time 1786694428.70 total 1012.98 last 11.96) - open/close 14412 (time 1786694439.63 total 1023.91 last 8.88) - open/close 14491 (time 1786694451.22 total 1035.49 last 6.82) - open/close 14590 (time 1786694461.85 total 1046.13 last 9.31) - open/close 14688 (time 1786694471.96 total 1056.24 last 9.70) - open/close 14851 (time 1786694481.96 total 1066.24 last 16.30) - open/close 14953 (time 1786694491.96 total 1076.24 last 10.20) - open/close 15081 (time 1786694502.16 total 1086.43 last 12.56) - open/close 15156 (time 1786694512.35 total 1096.63 last 7.35) - open/close 15660 (time 1786694522.35 total 1106.63 last 50.40) - open/close 16182 (time 1786694532.35 total 1116.63 last 52.20) - open/close 16812 (time 1786694542.36 total 1126.64 last 62.94) - open/close 17479 (time 1786694552.38 total 1136.66 last 66.59) - open/close 18091 (time 1786694562.39 total 1146.67 last 61.15) - open/close 18686 (time 1786694572.40 total 1156.67 last 59.46) - open/close 19317 (time 1786694582.40 total 1166.68 last 63.08) - open/close 19961 (time 1786694592.41 total 1176.69 last 64.33) - open/close 20581 (time 1786694602.41 total 1186.69 last 61.99) - open/close 21119 (time 1786694612.41 total 1196.69 last 53.80) - open/close 21609 (time 1786694622.43 total 1206.70 last 48.93) - open/close 22237 (time 1786694632.44 total 1216.72 last 62.70) - open/close 22867 (time 1786694642.46 total 1226.73 last 62.92) - open/close 23518 (time 1786694652.48 total 1236.75 last 64.96) - open/close 24189 (time 1786694662.50 total 1246.77 last 66.97) - open/close 24874 (time 1786694672.50 total 1256.77 last 68.50) - open/close 25556 (time 1786694682.50 total 1266.78 last 68.18) - open/close 26172 (time 1786694692.50 total 1276.78 last 61.58) - open/close 26725 (time 1786694702.53 total 1286.81 last 55.16) - open/close 27305 (time 1786694712.55 total 1296.82 last 57.89) - open/close 27864 (time 1786694722.55 total 1306.83 last 55.87) - open/close 28393 (time 1786694732.57 total 1316.85 last 52.81) - open/close 28970 (time 1786694742.57 total 1326.85 last 57.69) - open/close 29647 (time 1786694752.58 total 1336.86 last 67.63) - open/close 30000 (time 1786694758.18 total 1342.46 last 63.00) - open/close 30648 (time 1786694768.19 total 1352.46 last 64.78) - open/close 31270 (time 1786694778.20 total 1362.47 last 62.15) - open/close 31868 (time 1786694788.20 total 1372.48 last 59.79) - open/close 32503 (time 1786694798.21 total 1382.48 last 63.45) - open/close 33134 (time 1786694808.24 total 1392.52 last 62.87) - open/close 33775 (time 1786694818.24 total 1402.52 last 64.10) - open/close 34315 (time 1786694828.26 total 1412.54 last 53.88) - open/close 34868 (time 1786694838.28 total 1422.56 last 55.19) - open/close 35403 (time 1786694848.30 total 1432.58 last 53.43) - open/close 35862 (time 1786694858.33 total 1442.61 last 45.76) - open/close 36362 (time 1786694868.35 total 1452.63 last 49.90) - open/close 36961 (time 1786694878.35 total 1462.63 last 59.88) - open/close 37591 (time 1786694888.36 total 1472.63 last 62.97) - open/close 38201 (time 1786694898.38 total 1482.66 last 60.86) - open/close 38869 (time 1786694908.39 total 1492.67 last 66.73) - open/close 39567 (time 1786694918.41 total 1502.69 last 69.69) - open/close 40000 (time 1786694924.87 total 1509.15 last 66.97) - open/close 40680 (time 1786694934.89 total 1519.16 last 67.91) - open/close 41319 (time 1786694944.90 total 1529.18 last 63.80) - open/close 41919 (time 1786694954.91 total 1539.19 last 59.97) - open/close 42502 (time 1786694964.92 total 1549.20 last 58.23) - open/close 43087 (time 1786694974.92 total 1559.20 last 58.48) - open/close 43705 (time 1786694984.93 total 1569.21 last 61.76) - open/close 44303 (time 1786694994.94 total 1579.21 last 59.77) - open/close 44977 (time 1786695004.94 total 1589.22 last 67.38) - open/close 45686 (time 1786695014.94 total 1599.22 last 70.88) - open/close 46369 (time 1786695024.95 total 1609.23 last 68.25) - open/close 47054 (time 1786695034.95 total 1619.23 last 68.47) - open/close 47684 (time 1786695044.96 total 1629.23 last 62.98) - open/close 48368 (time 1786695054.98 total 1639.26 last 68.26) - open/close 49061 (time 1786695064.99 total 1649.26 last 69.23) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1657.85 seconds: 29.94 ops/second - unlinked 0 (time 1786695078 ; total 0 ; last 0) - unlinked 10000 (time 1786695231 ; total 153 ; last 153) - unlinked 20000 (time 1786695385 ; total 307 ; last 154) - unlinked 30000 (time 1786695532 ; total 454 ; last 147) - unlinked 40000 (time 1786695685 ; total 607 ; last 153) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 752 seconds: 65.998672 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg409-server unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: modules unloaded. pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2