== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:26:59 (1787124419) start mds service on oleg442-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing load_modules_local oleg442-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg442-server: Loading modules from /home/green/git/lustre-release/lustre oleg442-server: detected 4 online CPUs by sysfs oleg442-server: MODOPTS_LIBCFS= oleg442-server: Force libcfs to create 2 CPU partitions oleg442-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg442-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg442-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg442-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg442-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg442-server: oleg442-server.virtnet: executing set_default_debug -1 all pdsh@oleg442-client: oleg442-server: ssh exited with exit code 1 Started lustre-OST0000 oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg442-client.virtnet: -o user_xattr,flock 192.168.204.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1787124532.54 total 10.04 last 23.32) - open/close 456 (time 1787124543.44 total 20.93 last 20.37) - open/close 602 (time 1787124554.47 total 31.96 last 13.24) - open/close 750 (time 1787124564.47 total 41.96 last 14.80) - open/close 1018 (time 1787124576.92 total 54.41 last 21.53) - open/close 1285 (time 1787124587.23 total 64.73 last 25.88) - open/close 1534 (time 1787124597.25 total 74.74 last 24.86) - open/close 1842 (time 1787124608.09 total 85.59 last 28.40) - open/close 2108 (time 1787124618.14 total 95.63 last 26.48) - open/close 2278 (time 1787124628.75 total 106.25 last 16.01) - open/close 2444 (time 1787124638.76 total 116.26 last 16.58) - open/close 2632 (time 1787124650.15 total 127.64 last 16.51) - open/close 2839 (time 1787124660.25 total 137.74 last 20.50) - open/close 3031 (time 1787124670.61 total 148.10 last 18.53) - open/close 3233 (time 1787124681.62 total 159.11 last 18.35) - open/close 3347 (time 1787124691.63 total 169.12 last 11.39) - open/close 3610 (time 1787124701.96 total 179.45 last 25.46) - open/close 3824 (time 1787124711.97 total 189.46 last 21.38) - open/close 3955 (time 1787124721.99 total 199.48 last 13.07) - open/close 4086 (time 1787124732.19 total 209.68 last 12.84) - open/close 4196 (time 1787124742.88 total 220.38 last 10.29) - open/close 4413 (time 1787124753.12 total 230.62 last 21.19) - open/close 4590 (time 1787124763.74 total 241.23 last 16.68) - open/close 4617 (time 1787124773.75 total 251.24 last 2.70) - open/close 4783 (time 1787124785.24 total 262.74 last 14.44) - open/close 4839 (time 1787124796.07 total 273.56 last 5.17) - open/close 5002 (time 1787124806.08 total 283.57 last 16.28) - open/close 5192 (time 1787124816.10 total 293.59 last 18.96) - open/close 5379 (time 1787124827.04 total 304.53 last 17.09) - open/close 5563 (time 1787124837.97 total 315.46 last 16.84) - open/close 5678 (time 1787124847.97 total 325.47 last 11.49) - open/close 5832 (time 1787124858.16 total 335.65 last 15.12) - open/close 5929 (time 1787124868.70 total 346.19 last 9.20) - open/close 5957 (time 1787124878.81 total 356.30 last 2.77) - open/close 6092 (time 1787124889.58 total 367.07 last 12.53) - open/close 6246 (time 1787124900.01 total 377.50 last 14.77) - open/close 6342 (time 1787124910.49 total 387.98 last 9.16) - open/close 6433 (time 1787124920.79 total 398.28 last 8.83) - open/close 6613 (time 1787124932.49 total 409.98 last 15.38) - open/close 6705 (time 1787124943.13 total 420.62 last 8.65) - open/close 6814 (time 1787124953.14 total 430.63 last 10.89) - open/close 6918 (time 1787124963.16 total 440.65 last 10.38) - open/close 7078 (time 1787124973.18 total 450.67 last 15.97) - open/close 7218 (time 1787124984.12 total 461.61 last 12.80) - open/close 7398 (time 1787124994.13 total 471.62 last 17.98) - open/close 7600 (time 1787125005.03 total 482.52 last 18.53) - open/close 7781 (time 1787125015.96 total 493.45 last 16.56) - open/close 7926 (time 1787125026.38 total 503.88 last 13.91) - open/close 8072 (time 1787125036.39 total 513.88 last 14.59) - open/close 8225 (time 1787125046.41 total 523.90 last 15.27) - open/close 8443 (time 1787125056.80 total 534.29 last 20.99) - open/close 8649 (time 1787125069.14 total 546.63 last 16.69) - open/close 8900 (time 1787125079.50 total 557.00 last 24.22) - open/close 9155 (time 1787125090.01 total 567.50 last 24.28) - open/close 9357 (time 1787125102.10 total 579.59 last 16.71) - open/close 9581 (time 1787125112.11 total 589.60 last 22.38) - open/close 9660 (time 1787125122.86 total 600.35 last 7.35) - open/close 9903 (time 1787125133.47 total 610.96 last 22.91) - open/close 10000 (time 1787125138.73 total 616.23 last 18.41) - open/close 10221 (time 1787125149.62 total 627.12 last 20.29) - open/close 10479 (time 1787125160.55 total 638.04 last 23.62) - open/close 10739 (time 1787125170.58 total 648.08 last 25.91) - open/close 10925 (time 1787125181.13 total 658.63 last 17.63) - open/close 11124 (time 1787125191.14 total 668.63 last 19.90) - open/close 11225 (time 1787125201.19 total 678.68 last 10.05) - open/close 11391 (time 1787125211.40 total 688.89 last 16.26) - open/close 11494 (time 1787125221.55 total 699.04 last 10.15) - open/close 11731 (time 1787125232.35 total 709.84 last 21.93) - open/close 11830 (time 1787125242.70 total 720.19 last 9.57) - open/close 12073 (time 1787125254.38 total 731.87 last 20.81) - open/close 12233 (time 1787125268.84 total 746.33 last 11.06) - open/close 12465 (time 1787125281.49 total 758.98 last 18.35) - open/close 12665 (time 1787125291.49 total 768.98 last 19.99) - open/close 12941 (time 1787125301.66 total 779.15 last 27.14) - open/close 13090 (time 1787125311.67 total 789.16 last 14.89) - open/close 13268 (time 1787125321.69 total 799.18 last 17.77) - open/close 13462 (time 1787125332.38 total 809.87 last 18.14) - open/close 13669 (time 1787125342.49 total 819.98 last 20.47) - open/close 13903 (time 1787125352.50 total 830.00 last 23.37) - open/close 14169 (time 1787125362.50 total 840.00 last 26.60) - open/close 14349 (time 1787125372.62 total 850.11 last 17.80) - open/close 14513 (time 1787125382.63 total 860.12 last 16.38) - open/close 14802 (time 1787125392.84 total 870.33 last 28.31) - open/close 15021 (time 1787125402.84 total 880.33 last 21.90) - open/close 15260 (time 1787125412.85 total 890.34 last 23.88) - open/close 15550 (time 1787125423.82 total 901.31 last 26.44) - open/close 15719 (time 1787125435.34 total 912.83 last 14.67) - open/close 15834 (time 1787125445.80 total 923.29 last 11.00) - open/close 16052 (time 1787125456.16 total 933.65 last 21.04) - open/close 16132 (time 1787125467.42 total 944.91 last 7.10) - open/close 16254 (time 1787125477.97 total 955.46 last 11.56) - open/close 16377 (time 1787125488.73 total 966.22 last 11.44) - open/close 16656 (time 1787125498.75 total 976.24 last 27.85) - open/close 16864 (time 1787125509.48 total 986.98 last 19.37) - open/close 17069 (time 1787125519.49 total 996.98 last 20.49) - open/close 17208 (time 1787125530.27 total 1007.76 last 12.90) - open/close 17419 (time 1787125540.75 total 1018.24 last 20.12) - open/close 17579 (time 1787125550.76 total 1028.25 last 15.99) - open/close 17713 (time 1787125560.76 total 1038.25 last 13.40) - open/close 18009 (time 1787125571.80 total 1049.29 last 26.81) - open/close 18523 (time 1787125581.81 total 1059.30 last 51.35) - open/close 19249 (time 1787125591.81 total 1069.30 last 72.59) - open/close 19991 (time 1787125601.84 total 1079.34 last 73.94) - open/close 20752 (time 1787125611.85 total 1089.34 last 76.06) - open/close 21356 (time 1787125621.85 total 1099.35 last 60.37) - open/close 22111 (time 1787125631.86 total 1109.35 last 75.48) - open/close 22942 (time 1787125641.87 total 1119.36 last 83.01) - open/close 23894 (time 1787125651.88 total 1129.38 last 95.03) - open/close 24765 (time 1787125661.89 total 1139.38 last 87.09) - open/close 25770 (time 1787125671.90 total 1149.39 last 100.38) - open/close 26687 (time 1787125681.90 total 1159.39 last 91.69) - open/close 27408 (time 1787125691.92 total 1169.41 last 71.98) - open/close 28276 (time 1787125701.92 total 1179.41 last 86.79) - open/close 29084 (time 1787125711.92 total 1189.41 last 80.80) - open/close 30000 (time 1787125721.83 total 1199.32 last 92.43) - open/close 30951 (time 1787125731.84 total 1209.34 last 94.96) - open/close 31753 (time 1787125741.87 total 1219.36 last 80.00) - open/close 32480 (time 1787125751.87 total 1229.36 last 72.67) - open/close 33443 (time 1787125761.88 total 1239.37 last 96.27) - open/close 34255 (time 1787125771.88 total 1249.37 last 81.19) - open/close 35153 (time 1787125781.88 total 1259.37 last 89.76) - open/close 36115 (time 1787125791.89 total 1269.39 last 96.09) - open/close 37049 (time 1787125801.90 total 1279.39 last 93.36) - open/close 37658 (time 1787125811.90 total 1289.39 last 60.87) - open/close 38454 (time 1787125821.90 total 1299.40 last 79.58) - open/close 39222 (time 1787125831.92 total 1309.41 last 76.66) - open/close 40000 (time 1787125839.68 total 1317.17 last 100.27) - open/close 40717 (time 1787125849.68 total 1327.18 last 71.68) - open/close 41277 (time 1787125859.70 total 1337.19 last 55.92) - open/close 41871 (time 1787125869.70 total 1347.19 last 59.40) - open/close 42493 (time 1787125879.71 total 1357.20 last 62.13) - open/close 43203 (time 1787125889.71 total 1367.20 last 70.99) - open/close 43998 (time 1787125899.71 total 1377.20 last 79.50) - open/close 44789 (time 1787125909.72 total 1387.21 last 79.08) - open/close 45398 (time 1787125919.73 total 1397.22 last 60.83) - open/close 46220 (time 1787125929.73 total 1407.22 last 82.19) - open/close 46992 (time 1787125939.74 total 1417.24 last 77.07) - open/close 47780 (time 1787125949.75 total 1427.24 last 78.79) - open/close 48554 (time 1787125959.75 total 1437.25 last 77.33) - open/close 49330 (time 1787125969.76 total 1447.25 last 77.55) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1450.34 seconds: 34.22 ops/second - unlinked 0 (time 1787125977 ; total 0 ; last 0) - unlinked 10000 (time 1787126117 ; total 140 ; last 140) - unlinked 20000 (time 1787126238 ; total 261 ; last 121) - unlinked 30000 (time 1787126369 ; total 392 ; last 131) - unlinked 40000 (time 1787126493 ; total 516 ; last 124) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 635 seconds: 78.159058 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg442-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg442-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg442-server stop mds service on oleg442-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg442-server unloading modules via unload_modules_local on: 'oleg442-server' oleg442-server: oleg442-server.virtnet: executing unload_modules_local oleg442-server: modules unloaded. pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-client: ssh exited with exit code 2 pdsh@oleg442-client: oleg442-server: ssh exited with exit code 2