== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:15:01 (1787786101) start mds service on oleg437-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg437-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg437-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg437-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 140 (time 1787786202.70 total 12.69 last 11.03) - open/close 299 (time 1787786212.71 total 22.70 last 15.89) - open/close 572 (time 1787786223.37 total 33.36 last 25.60) - open/close 734 (time 1787786234.43 total 44.42 last 14.65) - open/close 922 (time 1787786244.80 total 54.79 last 18.14) - open/close 1152 (time 1787786255.10 total 65.09 last 22.33) - open/close 1393 (time 1787786265.15 total 75.14 last 23.97) - open/close 1679 (time 1787786275.51 total 85.50 last 27.61) - open/close 1964 (time 1787786286.41 total 96.41 last 26.14) - open/close 2227 (time 1787786296.41 total 106.41 last 26.30) - open/close 2499 (time 1787786306.58 total 116.57 last 26.76) - open/close 2705 (time 1787786317.05 total 127.04 last 19.67) - open/close 2908 (time 1787786327.42 total 137.42 last 19.57) - open/close 3150 (time 1787786337.43 total 147.42 last 24.18) - open/close 3346 (time 1787786347.83 total 157.83 last 18.84) - open/close 3624 (time 1787786357.97 total 167.96 last 27.43) - open/close 3895 (time 1787786367.97 total 177.97 last 27.08) - open/close 4101 (time 1787786378.17 total 188.17 last 20.20) - open/close 4355 (time 1787786389.71 total 199.70 last 22.02) - open/close 4644 (time 1787786400.19 total 210.18 last 27.57) - open/close 4947 (time 1787786411.19 total 221.19 last 27.54) - open/close 5228 (time 1787786422.14 total 232.14 last 25.66) - open/close 5518 (time 1787786432.15 total 242.14 last 28.99) - open/close 5764 (time 1787786442.17 total 252.16 last 24.54) - open/close 5964 (time 1787786452.48 total 262.47 last 19.40) - open/close 6224 (time 1787786462.49 total 272.48 last 25.99) - open/close 6491 (time 1787786472.49 total 282.48 last 26.69) - open/close 6748 (time 1787786482.76 total 292.75 last 25.03) - open/close 6999 (time 1787786492.76 total 302.76 last 25.08) - open/close 7226 (time 1787786502.78 total 312.77 last 22.67) - open/close 7319 (time 1787786512.89 total 322.88 last 9.20) - open/close 7523 (time 1787786523.84 total 333.83 last 18.64) - open/close 7707 (time 1787786534.14 total 344.14 last 17.85) - open/close 7806 (time 1787786544.15 total 354.15 last 9.89) - open/close 8104 (time 1787786554.77 total 364.76 last 28.08) - open/close 8311 (time 1787786565.65 total 375.65 last 19.01) - open/close 8504 (time 1787786575.67 total 385.66 last 19.27) - open/close 8685 (time 1787786586.50 total 396.50 last 16.71) - open/close 8958 (time 1787786596.51 total 406.50 last 27.28) - open/close 9236 (time 1787786607.43 total 417.42 last 25.46) - open/close 9470 (time 1787786618.43 total 428.43 last 21.26) - open/close 9748 (time 1787786628.44 total 438.43 last 27.78) - open/close 9993 (time 1787786640.77 total 450.77 last 19.87) - open/close 10236 (time 1787786651.18 total 461.17 last 23.35) - open/close 10462 (time 1787786661.88 total 471.87 last 21.12) - open/close 10679 (time 1787786672.76 total 482.76 last 19.94) - open/close 10883 (time 1787786682.78 total 492.77 last 20.37) - open/close 11177 (time 1787786692.78 total 502.77 last 29.39) - open/close 11340 (time 1787786703.55 total 513.55 last 15.13) - open/close 11572 (time 1787786714.33 total 524.32 last 21.54) - open/close 11831 (time 1787786724.86 total 534.85 last 24.59) - open/close 12073 (time 1787786735.11 total 545.10 last 23.61) - open/close 12336 (time 1787786745.12 total 555.11 last 26.27) - open/close 12585 (time 1787786755.12 total 565.12 last 24.89) - open/close 12840 (time 1787786765.14 total 575.13 last 25.47) - open/close 13105 (time 1787786775.30 total 585.29 last 26.09) - open/close 13366 (time 1787786785.61 total 595.61 last 25.29) - open/close 13577 (time 1787786796.55 total 606.54 last 19.30) - open/close 13656 (time 1787786808.73 total 618.72 last 6.48) - open/close 13953 (time 1787786818.73 total 628.72 last 29.70) - open/close 14207 (time 1787786828.73 total 638.73 last 25.39) - open/close 14487 (time 1787786839.26 total 649.25 last 26.60) - open/close 14786 (time 1787786849.40 total 659.40 last 29.48) - open/close 15033 (time 1787786859.41 total 669.41 last 24.67) - open/close 15254 (time 1787786870.41 total 680.40 last 20.10) - open/close 15550 (time 1787786880.41 total 690.41 last 29.58) - open/close 15605 (time 1787786891.07 total 701.06 last 5.16) - open/close 15744 (time 1787786901.95 total 711.95 last 12.77) - open/close 15885 (time 1787786911.97 total 721.96 last 14.08) - open/close 16219 (time 1787786921.97 total 731.97 last 33.40) - open/close 16451 (time 1787786932.29 total 742.28 last 22.49) - open/close 16751 (time 1787786943.07 total 753.06 last 27.82) - open/close 17053 (time 1787786953.78 total 763.78 last 28.19) - open/close 17327 (time 1787786964.35 total 774.34 last 25.94) - open/close 17592 (time 1787786974.36 total 784.35 last 26.48) - open/close 17700 (time 1787786984.71 total 794.70 last 10.43) - open/close 17760 (time 1787786994.97 total 804.96 last 5.85) - open/close 17987 (time 1787787005.28 total 815.27 last 22.01) - open/close 18182 (time 1787787018.25 total 828.25 last 15.03) - open/close 18388 (time 1787787028.95 total 838.95 last 19.26) - open/close 18603 (time 1787787039.82 total 849.82 last 19.78) - open/close 18784 (time 1787787050.13 total 860.12 last 17.57) - open/close 19084 (time 1787787061.00 total 870.99 last 27.59) - open/close 19365 (time 1787787071.81 total 881.80 last 25.99) - open/close 19633 (time 1787787081.83 total 891.82 last 26.76) - open/close 19904 (time 1787787091.83 total 901.82 last 27.10) - open/close 20000 (time 1787787094.14 total 904.13 last 41.56) - open/close 20251 (time 1787787104.19 total 914.19 last 24.96) - open/close 20608 (time 1787787114.20 total 924.19 last 35.67) - open/close 21457 (time 1787787124.21 total 934.21 last 84.81) - open/close 22313 (time 1787787134.22 total 944.21 last 85.55) - open/close 23243 (time 1787787144.22 total 954.22 last 92.94) - open/close 24087 (time 1787787154.23 total 964.22 last 84.36) - open/close 24943 (time 1787787164.24 total 974.23 last 85.54) - open/close 25789 (time 1787787174.24 total 984.23 last 84.55) - open/close 26591 (time 1787787184.25 total 994.24 last 80.16) - open/close 27320 (time 1787787194.25 total 1004.24 last 72.88) - open/close 28132 (time 1787787204.27 total 1014.26 last 81.06) - open/close 28987 (time 1787787214.27 total 1024.27 last 85.45) - open/close 29870 (time 1787787224.28 total 1034.28 last 88.21) - open/close 30686 (time 1787787234.29 total 1044.28 last 81.55) - open/close 31545 (time 1787787244.30 total 1054.29 last 85.81) - open/close 32419 (time 1787787254.30 total 1064.30 last 87.37) - open/close 33329 (time 1787787264.30 total 1074.30 last 90.99) - open/close 34206 (time 1787787274.31 total 1084.30 last 87.65) - open/close 35033 (time 1787787284.32 total 1094.31 last 82.63) - open/close 35838 (time 1787787294.33 total 1104.32 last 80.43) - open/close 36656 (time 1787787304.33 total 1114.33 last 81.74) - open/close 37520 (time 1787787314.34 total 1124.33 last 86.37) - open/close 38469 (time 1787787324.34 total 1134.34 last 94.84) - open/close 39446 (time 1787787334.36 total 1144.35 last 97.57) - open/close 40000 (time 1787787340.34 total 1150.34 last 92.50) - open/close 40921 (time 1787787350.35 total 1160.35 last 92.02) - open/close 41768 (time 1787787360.38 total 1170.37 last 84.50) - open/close 42648 (time 1787787370.38 total 1180.38 last 87.94) - open/close 43524 (time 1787787380.40 total 1190.39 last 87.47) - open/close 44330 (time 1787787390.41 total 1200.41 last 80.49) - open/close 45200 (time 1787787400.43 total 1210.42 last 86.88) - open/close 46099 (time 1787787410.43 total 1220.43 last 89.83) - open/close 46984 (time 1787787420.44 total 1230.43 last 88.47) - open/close 47835 (time 1787787430.44 total 1240.44 last 85.03) - open/close 48751 (time 1787787440.46 total 1250.45 last 91.50) - open/close 49586 (time 1787787450.46 total 1260.46 last 83.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1261.16 seconds: 39.35 ops/second - unlinked 0 (time 1787787455 ; total 0 ; last 0) - unlinked 10000 (time 1787787568 ; total 113 ; last 113) - unlinked 20000 (time 1787787683 ; total 228 ; last 115) - unlinked 30000 (time 1787787749 ; total 294 ; last 66) - unlinked 40000 (time 1787787817 ; total 362 ; last 68) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 428 seconds: 115.960281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local oleg437-server: modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2