== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:33:01 (1787866381) start mds service on oleg210-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg210-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg210-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg210-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 162 (time 1787866491.98 total 10.29 last 15.75) - open/close 442 (time 1787866501.99 total 20.30 last 27.98) - open/close 651 (time 1787866512.00 total 30.31 last 20.88) - open/close 892 (time 1787866522.02 total 40.33 last 24.05) - open/close 1117 (time 1787866532.04 total 50.35 last 22.46) - open/close 1291 (time 1787866542.05 total 60.36 last 17.38) - open/close 1533 (time 1787866552.06 total 70.37 last 24.18) - open/close 1747 (time 1787866562.89 total 81.20 last 19.76) - open/close 1896 (time 1787866572.91 total 91.22 last 14.87) - open/close 2129 (time 1787866583.49 total 101.80 last 22.03) - open/close 2401 (time 1787866593.50 total 111.81 last 27.17) - open/close 2576 (time 1787866603.51 total 121.82 last 17.47) - open/close 2750 (time 1787866613.69 total 132.00 last 17.09) - open/close 3041 (time 1787866623.71 total 142.02 last 29.04) - open/close 3192 (time 1787866633.87 total 152.18 last 14.87) - open/close 3455 (time 1787866643.88 total 162.19 last 26.26) - open/close 3646 (time 1787866654.47 total 172.78 last 18.04) - open/close 3892 (time 1787866665.37 total 183.68 last 22.57) - open/close 4033 (time 1787866677.60 total 195.90 last 11.53) - open/close 4261 (time 1787866688.19 total 206.50 last 21.51) - open/close 4438 (time 1787866698.41 total 216.72 last 17.33) - open/close 4601 (time 1787866709.36 total 227.67 last 14.88) - open/close 4756 (time 1787866722.00 total 240.31 last 12.26) - open/close 5000 (time 1787866732.49 total 250.80 last 23.26) - open/close 5247 (time 1787866743.56 total 261.87 last 22.31) - open/close 5505 (time 1787866753.56 total 271.87 last 25.80) - open/close 5739 (time 1787866763.96 total 282.27 last 22.50) - open/close 5813 (time 1787866776.85 total 295.16 last 5.74) - open/close 6076 (time 1787866787.19 total 305.50 last 25.42) - open/close 6295 (time 1787866797.75 total 316.06 last 20.75) - open/close 6516 (time 1787866808.46 total 326.76 last 20.64) - open/close 6755 (time 1787866818.83 total 337.14 last 23.04) - open/close 7017 (time 1787866828.83 total 347.14 last 26.19) - open/close 7250 (time 1787866839.39 total 357.70 last 22.06) - open/close 7388 (time 1787866849.42 total 367.72 last 13.77) - open/close 7633 (time 1787866860.32 total 378.63 last 22.46) - open/close 7736 (time 1787866874.00 total 392.31 last 7.53) - open/close 7833 (time 1787866884.36 total 402.67 last 9.36) - open/close 8089 (time 1787866894.38 total 412.69 last 25.56) - open/close 8299 (time 1787866904.58 total 422.89 last 20.58) - open/close 8508 (time 1787866916.89 total 435.20 last 16.98) - open/close 8607 (time 1787866926.91 total 445.22 last 9.88) - open/close 8821 (time 1787866939.46 total 457.76 last 17.06) - open/close 9049 (time 1787866950.30 total 468.61 last 21.03) - open/close 9204 (time 1787866960.83 total 479.14 last 14.72) - open/close 9413 (time 1787866970.84 total 489.15 last 20.88) - open/close 9618 (time 1787866982.02 total 500.33 last 18.33) - open/close 9851 (time 1787866992.04 total 510.35 last 23.26) - open/close 10000 (time 1787866997.83 total 516.14 last 25.74) - open/close 10204 (time 1787867008.06 total 526.37 last 19.94) - open/close 10409 (time 1787867018.08 total 536.39 last 20.46) - open/close 10621 (time 1787867028.09 total 546.40 last 21.17) - open/close 10767 (time 1787867039.97 total 558.27 last 12.30) - open/close 10947 (time 1787867050.11 total 568.42 last 17.75) - open/close 11152 (time 1787867060.17 total 578.48 last 20.37) - open/close 11361 (time 1787867070.19 total 588.49 last 20.87) - open/close 11540 (time 1787867080.65 total 598.96 last 17.11) - open/close 11687 (time 1787867090.66 total 608.97 last 14.68) - open/close 11775 (time 1787867101.12 total 619.43 last 8.42) - open/close 11952 (time 1787867111.12 total 629.43 last 17.70) - open/close 12119 (time 1787867121.94 total 640.25 last 15.43) - open/close 12246 (time 1787867132.88 total 651.18 last 11.62) - open/close 12427 (time 1787867143.63 total 661.93 last 16.84) - open/close 12509 (time 1787867153.63 total 671.94 last 8.19) - open/close 12702 (time 1787867163.65 total 681.96 last 19.26) - open/close 12919 (time 1787867174.13 total 692.44 last 20.70) - open/close 13052 (time 1787867184.24 total 702.54 last 13.16) - open/close 13327 (time 1787867194.99 total 713.30 last 25.58) - open/close 13451 (time 1787867205.79 total 724.09 last 11.48) - open/close 13555 (time 1787867216.28 total 734.59 last 9.91) - open/close 13805 (time 1787867226.30 total 744.61 last 24.95) - open/close 14004 (time 1787867236.62 total 754.93 last 19.28) - open/close 14179 (time 1787867247.38 total 765.68 last 16.27) - open/close 14287 (time 1787867257.43 total 775.74 last 10.74) - open/close 14466 (time 1787867268.63 total 786.93 last 15.98) - open/close 14724 (time 1787867278.63 total 796.94 last 25.78) - open/close 14934 (time 1787867291.34 total 809.65 last 16.53) - open/close 15113 (time 1787867301.57 total 819.88 last 17.50) - open/close 15252 (time 1787867311.95 total 830.26 last 13.39) - open/close 15390 (time 1787867321.96 total 840.27 last 13.79) - open/close 15546 (time 1787867331.98 total 850.29 last 15.57) - open/close 15799 (time 1787867342.53 total 860.84 last 23.98) - open/close 16057 (time 1787867352.53 total 870.84 last 25.80) - open/close 16348 (time 1787867363.27 total 881.58 last 27.10) - open/close 16500 (time 1787867373.84 total 892.15 last 14.37) - open/close 16711 (time 1787867384.85 total 903.15 last 19.18) - open/close 16824 (time 1787867395.15 total 913.46 last 10.97) - open/close 16953 (time 1787867405.15 total 923.46 last 12.90) - open/close 17076 (time 1787867415.16 total 933.47 last 12.29) - open/close 17224 (time 1787867425.55 total 943.86 last 14.24) - open/close 17421 (time 1787867437.54 total 955.84 last 16.44) - open/close 17547 (time 1787867447.63 total 965.94 last 12.49) - open/close 17722 (time 1787867457.64 total 975.95 last 17.47) - open/close 17945 (time 1787867467.98 total 986.29 last 21.58) - open/close 18125 (time 1787867477.99 total 996.30 last 17.97) - open/close 18263 (time 1787867488.14 total 1006.45 last 13.60) - open/close 18837 (time 1787867498.16 total 1016.46 last 57.32) - open/close 19488 (time 1787867508.16 total 1026.47 last 65.06) - open/close 20000 (time 1787867516.91 total 1035.22 last 58.49) - open/close 20700 (time 1787867526.92 total 1045.23 last 69.96) - open/close 21410 (time 1787867536.93 total 1055.23 last 70.96) - open/close 22118 (time 1787867546.93 total 1065.24 last 70.76) - open/close 22888 (time 1787867556.93 total 1075.24 last 76.99) - open/close 23572 (time 1787867566.94 total 1085.25 last 68.37) - open/close 24267 (time 1787867576.94 total 1095.25 last 69.48) - open/close 25010 (time 1787867586.95 total 1105.25 last 74.26) - open/close 25726 (time 1787867596.98 total 1115.29 last 71.36) - open/close 26381 (time 1787867606.99 total 1125.29 last 65.45) - open/close 26938 (time 1787867616.99 total 1135.30 last 55.67) - open/close 27500 (time 1787867627.00 total 1145.31 last 56.13) - open/close 28069 (time 1787867637.02 total 1155.33 last 56.80) - open/close 28652 (time 1787867647.04 total 1165.35 last 58.19) - open/close 29313 (time 1787867657.05 total 1175.36 last 66.05) - open/close 29910 (time 1787867667.05 total 1185.36 last 59.67) - open/close 30583 (time 1787867677.06 total 1195.36 last 67.27) - open/close 31172 (time 1787867687.06 total 1205.36 last 58.89) - open/close 31905 (time 1787867697.07 total 1215.37 last 73.23) - open/close 32597 (time 1787867707.07 total 1225.38 last 69.16) - open/close 33274 (time 1787867717.08 total 1235.38 last 67.67) - open/close 34004 (time 1787867727.08 total 1245.39 last 72.99) - open/close 34787 (time 1787867737.08 total 1255.39 last 78.28) - open/close 35443 (time 1787867747.11 total 1265.42 last 65.38) - open/close 36077 (time 1787867757.11 total 1275.42 last 63.40) - open/close 36723 (time 1787867767.12 total 1285.43 last 64.56) - open/close 37498 (time 1787867777.13 total 1295.43 last 77.45) - open/close 38175 (time 1787867787.14 total 1305.44 last 67.63) - open/close 38868 (time 1787867797.14 total 1315.45 last 69.28) - open/close 39524 (time 1787867807.14 total 1325.45 last 65.56) - open/close 40000 (time 1787867814.04 total 1332.35 last 68.98) - open/close 40662 (time 1787867824.05 total 1342.36 last 66.13) - open/close 41207 (time 1787867834.07 total 1352.38 last 54.40) - open/close 41698 (time 1787867844.07 total 1362.38 last 49.10) - open/close 42238 (time 1787867854.09 total 1372.40 last 53.91) - open/close 42839 (time 1787867864.09 total 1382.40 last 60.07) - open/close 43423 (time 1787867874.10 total 1392.41 last 58.34) - open/close 43990 (time 1787867884.11 total 1402.42 last 56.67) - open/close 44557 (time 1787867894.12 total 1412.43 last 56.64) - open/close 45105 (time 1787867904.12 total 1422.43 last 54.77) - open/close 45757 (time 1787867914.14 total 1432.45 last 65.07) - open/close 46521 (time 1787867924.16 total 1442.47 last 76.28) - open/close 47107 (time 1787867934.16 total 1452.47 last 58.58) - open/close 47892 (time 1787867944.17 total 1462.48 last 78.44) - open/close 48503 (time 1787867954.18 total 1472.49 last 61.04) - open/close 49186 (time 1787867964.19 total 1482.49 last 68.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1490.91 seconds: 33.29 ops/second - unlinked 0 (time 1787867978 ; total 0 ; last 0) - unlinked 10000 (time 1787868120 ; total 142 ; last 142) - unlinked 20000 (time 1787868264 ; total 286 ; last 144) - unlinked 30000 (time 1787868404 ; total 426 ; last 140) - unlinked 40000 (time 1787868538 ; total 560 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 692 seconds: 71.719650 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local oleg210-server: modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2