== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:39:06 (1787873946) start mds service on oleg448-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg448-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg448-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 81 (time 1787874052.84 total 11.27 last 7.19) - open/close 320 (time 1787874062.84 total 21.27 last 23.90) - open/close 488 (time 1787874072.85 total 31.28 last 16.78) - open/close 689 (time 1787874082.86 total 41.29 last 20.09) - open/close 922 (time 1787874093.87 total 52.30 last 21.16) - open/close 1115 (time 1787874104.34 total 62.77 last 18.44) - open/close 1405 (time 1787874114.59 total 73.02 last 28.29) - open/close 1594 (time 1787874124.63 total 83.06 last 18.83) - open/close 1851 (time 1787874135.70 total 94.13 last 23.21) - open/close 1968 (time 1787874145.88 total 104.31 last 11.49) - open/close 2086 (time 1787874155.89 total 114.31 last 11.79) - open/close 2233 (time 1787874165.90 total 124.32 last 14.69) - open/close 2376 (time 1787874176.66 total 135.08 last 13.29) - open/close 2521 (time 1787874186.67 total 145.10 last 14.48) - open/close 2756 (time 1787874197.28 total 155.70 last 22.16) - open/close 3018 (time 1787874207.28 total 165.71 last 26.20) - open/close 3266 (time 1787874218.29 total 176.71 last 22.53) - open/close 3472 (time 1787874229.08 total 187.51 last 19.08) - open/close 3675 (time 1787874239.26 total 197.69 last 19.94) - open/close 3799 (time 1787874250.06 total 208.49 last 11.48) - open/close 4027 (time 1787874260.76 total 219.19 last 21.32) - open/close 4301 (time 1787874271.84 total 230.26 last 24.74) - open/close 4431 (time 1787874282.64 total 241.07 last 12.03) - open/close 4614 (time 1787874292.65 total 251.08 last 18.28) - open/close 4757 (time 1787874302.66 total 261.09 last 14.28) - open/close 4935 (time 1787874312.69 total 271.12 last 17.75) - open/close 5191 (time 1787874322.70 total 281.13 last 25.57) - open/close 5381 (time 1787874332.95 total 291.38 last 18.54) - open/close 5454 (time 1787874342.95 total 301.38 last 7.30) - open/close 5542 (time 1787874353.75 total 312.18 last 8.15) - open/close 5677 (time 1787874364.37 total 322.80 last 12.71) - open/close 5764 (time 1787874374.95 total 333.37 last 8.23) - open/close 5845 (time 1787874386.35 total 344.78 last 7.10) - open/close 5981 (time 1787874397.10 total 355.52 last 12.66) - open/close 6160 (time 1787874407.26 total 365.69 last 17.60) - open/close 6397 (time 1787874417.29 total 375.71 last 23.65) - open/close 6565 (time 1787874427.88 total 386.30 last 15.86) - open/close 6687 (time 1787874438.06 total 396.48 last 11.99) - open/close 6748 (time 1787874448.28 total 406.71 last 5.96) - open/close 6885 (time 1787874458.64 total 417.07 last 13.22) - open/close 6991 (time 1787874469.48 total 427.91 last 9.78) - open/close 7268 (time 1787874479.49 total 437.91 last 27.69) - open/close 7469 (time 1787874489.49 total 447.91 last 20.10) - open/close 7690 (time 1787874500.19 total 458.61 last 20.66) - open/close 7886 (time 1787874512.15 total 470.57 last 16.39) - open/close 8136 (time 1787874522.18 total 480.61 last 24.92) - open/close 8287 (time 1787874533.62 total 492.05 last 13.20) - open/close 8485 (time 1787874544.04 total 502.47 last 19.00) - open/close 8631 (time 1787874555.25 total 513.68 last 13.02) - open/close 8813 (time 1787874565.26 total 523.69 last 18.19) - open/close 9030 (time 1787874575.85 total 534.28 last 20.49) - open/close 9283 (time 1787874586.71 total 545.14 last 23.30) - open/close 9476 (time 1787874597.95 total 556.38 last 17.17) - open/close 9563 (time 1787874608.29 total 566.71 last 8.42) - open/close 9689 (time 1787874618.62 total 577.05 last 12.19) - open/close 9812 (time 1787874629.47 total 587.90 last 11.34) - open/close 10000 (time 1787874638.71 total 597.14 last 20.33) - open/close 10176 (time 1787874649.69 total 608.11 last 16.04) - open/close 10348 (time 1787874661.60 total 620.03 last 14.44) - open/close 10480 (time 1787874673.83 total 632.26 last 10.79) - open/close 10604 (time 1787874683.84 total 642.27 last 12.39) - open/close 10771 (time 1787874693.85 total 652.28 last 16.68) - open/close 10895 (time 1787874704.73 total 663.15 last 11.41) - open/close 10992 (time 1787874715.29 total 673.71 last 9.19) - open/close 11134 (time 1787874726.58 total 685.00 last 12.58) - open/close 11237 (time 1787874737.57 total 695.99 last 9.37) - open/close 11421 (time 1787874747.58 total 706.01 last 18.38) - open/close 11674 (time 1787874757.59 total 716.02 last 25.27) - open/close 11795 (time 1787874768.46 total 726.89 last 11.13) - open/close 11904 (time 1787874779.25 total 737.68 last 10.10) - open/close 12018 (time 1787874789.27 total 747.69 last 11.38) - open/close 12161 (time 1787874800.16 total 758.59 last 13.12) - open/close 12218 (time 1787874810.20 total 768.63 last 5.68) - open/close 12337 (time 1787874820.51 total 778.94 last 11.54) - open/close 12458 (time 1787874831.39 total 789.81 last 11.13) - open/close 12634 (time 1787874841.94 total 800.37 last 16.68) - open/close 12764 (time 1787874851.99 total 810.42 last 12.93) - open/close 12960 (time 1787874862.24 total 820.67 last 19.12) - open/close 13025 (time 1787874872.25 total 830.68 last 6.49) - open/close 13193 (time 1787874882.85 total 841.27 last 15.85) - open/close 13457 (time 1787874892.86 total 851.29 last 26.36) - open/close 13756 (time 1787874902.88 total 861.31 last 29.85) - open/close 13974 (time 1787874913.45 total 871.88 last 20.61) - open/close 14264 (time 1787874924.05 total 882.48 last 27.35) - open/close 14485 (time 1787874935.19 total 893.62 last 19.85) - open/close 14721 (time 1787874945.36 total 903.79 last 23.20) - open/close 14971 (time 1787874955.54 total 913.97 last 24.56) - open/close 15260 (time 1787874965.55 total 923.98 last 28.88) - open/close 15516 (time 1787874975.61 total 934.03 last 25.46) - open/close 15735 (time 1787874985.61 total 944.04 last 21.89) - open/close 15898 (time 1787874995.61 total 954.04 last 16.29) - open/close 16141 (time 1787875005.77 total 964.20 last 23.91) - open/close 16351 (time 1787875015.83 total 974.25 last 20.89) - open/close 16525 (time 1787875026.85 total 985.28 last 15.78) - open/close 16698 (time 1787875038.43 total 996.85 last 14.95) - open/close 16951 (time 1787875049.06 total 1007.49 last 23.79) - open/close 17211 (time 1787875059.07 total 1017.50 last 25.98) - open/close 17489 (time 1787875069.08 total 1027.50 last 27.78) - open/close 17703 (time 1787875079.09 total 1037.52 last 21.37) - open/close 18154 (time 1787875089.09 total 1047.52 last 45.10) - open/close 18938 (time 1787875099.10 total 1057.52 last 78.34) - open/close 19682 (time 1787875109.10 total 1067.53 last 74.37) - open/close 20000 (time 1787875113.01 total 1071.44 last 81.32) - open/close 20772 (time 1787875123.01 total 1081.44 last 77.17) - open/close 21602 (time 1787875133.02 total 1091.45 last 82.96) - open/close 22378 (time 1787875143.02 total 1101.45 last 77.57) - open/close 23114 (time 1787875153.03 total 1111.45 last 73.56) - open/close 23854 (time 1787875163.03 total 1121.46 last 73.99) - open/close 24622 (time 1787875173.04 total 1131.47 last 76.68) - open/close 25379 (time 1787875183.05 total 1141.48 last 75.64) - open/close 26150 (time 1787875193.06 total 1151.49 last 77.00) - open/close 26872 (time 1787875203.07 total 1161.50 last 72.17) - open/close 27626 (time 1787875213.07 total 1171.50 last 75.38) - open/close 28324 (time 1787875223.08 total 1181.50 last 69.77) - open/close 29003 (time 1787875233.08 total 1191.51 last 67.89) - open/close 29757 (time 1787875243.08 total 1201.51 last 75.36) - open/close 30000 (time 1787875245.81 total 1204.24 last 88.94) - open/close 30583 (time 1787875255.83 total 1214.26 last 58.20) - open/close 30920 (time 1787875265.83 total 1224.26 last 33.69) - open/close 31218 (time 1787875275.89 total 1234.32 last 29.63) - open/close 31487 (time 1787875285.90 total 1244.33 last 26.87) - open/close 31809 (time 1787875295.90 total 1254.33 last 32.20) - open/close 32214 (time 1787875305.91 total 1264.34 last 40.48) - open/close 32681 (time 1787875315.93 total 1274.36 last 46.61) - open/close 33132 (time 1787875325.93 total 1284.36 last 45.07) - open/close 33658 (time 1787875335.94 total 1294.37 last 52.57) - open/close 34242 (time 1787875345.95 total 1304.38 last 58.35) - open/close 34959 (time 1787875355.96 total 1314.39 last 71.64) - open/close 35610 (time 1787875365.96 total 1324.39 last 65.07) - open/close 36314 (time 1787875375.96 total 1334.39 last 70.38) - open/close 36979 (time 1787875385.99 total 1344.41 last 66.37) - open/close 37626 (time 1787875396.00 total 1354.42 last 64.63) - open/close 38305 (time 1787875406.00 total 1364.43 last 67.87) - open/close 38996 (time 1787875416.01 total 1374.44 last 69.02) - open/close 39618 (time 1787875426.01 total 1384.44 last 62.19) - open/close 40000 (time 1787875431.56 total 1389.98 last 68.94) - open/close 40656 (time 1787875441.56 total 1399.99 last 65.55) - open/close 41446 (time 1787875451.57 total 1410.00 last 78.91) - open/close 42240 (time 1787875461.58 total 1420.01 last 79.33) - open/close 42951 (time 1787875471.59 total 1430.02 last 71.07) - open/close 43589 (time 1787875481.59 total 1440.02 last 63.77) - open/close 44285 (time 1787875491.61 total 1450.04 last 69.49) - open/close 44989 (time 1787875501.62 total 1460.05 last 70.34) - open/close 45707 (time 1787875511.62 total 1470.05 last 71.76) - open/close 46422 (time 1787875521.64 total 1480.07 last 71.38) - open/close 46998 (time 1787875531.65 total 1490.08 last 57.55) - open/close 47679 (time 1787875541.66 total 1500.09 last 68.03) - open/close 48379 (time 1787875551.67 total 1510.10 last 69.92) - open/close 49056 (time 1787875561.68 total 1520.11 last 67.64) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1528.71 seconds: 32.47 ops/second - unlinked 0 (time 1787875575 ; total 0 ; last 0) - unlinked 10000 (time 1787875713 ; total 138 ; last 138) - unlinked 20000 (time 1787875848 ; total 273 ; last 135) - unlinked 30000 (time 1787876003 ; total 428 ; last 155) - unlinked 40000 (time 1787876161 ; total 586 ; last 158) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 733 seconds: 67.709412 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded. pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2