== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:17:05 (1786760225) start mds service on oleg407-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg407-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg407-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 121 (time 1786760342.89 total 10.02 last 12.07) - open/close 334 (time 1786760355.56 total 22.69 last 16.81) - open/close 584 (time 1786760365.58 total 32.70 last 24.96) - open/close 782 (time 1786760375.58 total 42.71 last 19.80) - open/close 893 (time 1786760385.59 total 52.71 last 11.09) - open/close 1161 (time 1786760395.60 total 62.73 last 26.77) - open/close 1437 (time 1786760406.39 total 73.52 last 25.57) - open/close 1654 (time 1786760416.61 total 83.74 last 21.23) - open/close 1908 (time 1786760426.63 total 93.76 last 25.36) - open/close 1982 (time 1786760436.64 total 103.77 last 7.39) - open/close 2116 (time 1786760446.64 total 113.77 last 13.40) - open/close 2349 (time 1786760456.70 total 123.82 last 23.18) - open/close 2539 (time 1786760467.63 total 134.75 last 17.38) - open/close 2673 (time 1786760478.47 total 145.59 last 12.36) - open/close 2805 (time 1786760489.51 total 156.64 last 11.95) - open/close 3119 (time 1786760500.49 total 167.62 last 28.59) - open/close 3275 (time 1786760510.58 total 177.70 last 15.47) - open/close 3498 (time 1786760521.01 total 188.14 last 21.37) - open/close 3657 (time 1786760531.67 total 198.80 last 14.92) - open/close 3706 (time 1786760542.11 total 209.23 last 4.69) - open/close 3898 (time 1786760553.02 total 220.15 last 17.59) - open/close 4176 (time 1786760563.03 total 230.16 last 27.76) - open/close 4331 (time 1786760573.72 total 240.85 last 14.50) - open/close 4582 (time 1786760583.72 total 250.85 last 25.10) - open/close 4723 (time 1786760593.73 total 260.86 last 14.09) - open/close 4826 (time 1786760604.01 total 271.14 last 10.02) - open/close 4929 (time 1786760614.52 total 281.65 last 9.80) - open/close 5105 (time 1786760625.50 total 292.63 last 16.03) - open/close 5301 (time 1786760636.13 total 303.26 last 18.43) - open/close 5402 (time 1786760647.35 total 314.48 last 9.01) - open/close 5609 (time 1786760658.15 total 325.28 last 19.16) - open/close 5889 (time 1786760668.17 total 335.29 last 27.96) - open/close 6019 (time 1786760678.69 total 345.81 last 12.36) - open/close 6213 (time 1786760689.63 total 356.76 last 17.72) - open/close 6518 (time 1786760700.00 total 367.12 last 29.43) - open/close 6780 (time 1786760712.31 total 379.44 last 21.28) - open/close 6995 (time 1786760722.31 total 389.44 last 21.49) - open/close 7296 (time 1786760732.90 total 400.02 last 28.44) - open/close 7495 (time 1786760743.72 total 410.84 last 18.39) - open/close 7759 (time 1786760754.15 total 421.28 last 25.29) - open/close 8037 (time 1786760765.01 total 432.14 last 25.60) - open/close 8304 (time 1786760775.04 total 442.16 last 26.64) - open/close 8497 (time 1786760785.05 total 452.17 last 19.28) - open/close 8724 (time 1786760795.05 total 462.18 last 22.68) - open/close 8892 (time 1786760805.10 total 472.23 last 16.72) - open/close 9119 (time 1786760817.58 total 484.70 last 18.20) - open/close 9370 (time 1786760827.87 total 495.00 last 24.38) - open/close 9644 (time 1786760837.88 total 505.00 last 27.39) - open/close 9893 (time 1786760848.27 total 515.40 last 23.95) - open/close 10000 (time 1786760851.01 total 518.14 last 39.02) - open/close 10258 (time 1786760861.09 total 528.22 last 25.59) - open/close 10511 (time 1786760871.12 total 538.24 last 25.24) - open/close 10583 (time 1786760881.13 total 548.26 last 7.19) - open/close 10752 (time 1786760891.50 total 558.63 last 16.30) - open/close 10932 (time 1786760901.55 total 568.68 last 17.92) - open/close 11104 (time 1786760911.88 total 579.01 last 16.65) - open/close 11338 (time 1786760922.15 total 589.28 last 22.78) - open/close 11590 (time 1786760932.63 total 599.76 last 24.05) - open/close 11747 (time 1786760942.95 total 610.08 last 15.22) - open/close 11814 (time 1786760955.70 total 622.83 last 5.25) - open/close 12042 (time 1786760968.03 total 635.15 last 18.50) - open/close 12302 (time 1786760978.27 total 645.40 last 25.38) - open/close 12572 (time 1786760988.53 total 655.66 last 26.31) - open/close 12767 (time 1786760998.63 total 665.75 last 19.32) - open/close 13022 (time 1786761014.15 total 681.28 last 16.43) - open/close 13308 (time 1786761025.05 total 692.18 last 26.23) - open/close 13561 (time 1786761035.43 total 702.55 last 24.38) - open/close 13820 (time 1786761045.44 total 712.57 last 25.87) - open/close 14063 (time 1786761055.46 total 722.59 last 24.25) - open/close 14271 (time 1786761066.25 total 733.37 last 19.28) - open/close 14437 (time 1786761076.25 total 743.38 last 16.59) - open/close 14682 (time 1786761086.25 total 753.38 last 24.49) - open/close 14885 (time 1786761096.80 total 763.93 last 19.24) - open/close 15086 (time 1786761107.87 total 775.00 last 18.15) - open/close 15358 (time 1786761118.81 total 785.94 last 24.86) - open/close 15559 (time 1786761128.82 total 795.95 last 20.09) - open/close 15771 (time 1786761139.81 total 806.94 last 19.29) - open/close 15966 (time 1786761150.57 total 817.69 last 18.13) - open/close 16199 (time 1786761160.58 total 827.71 last 23.26) - open/close 16380 (time 1786761174.30 total 841.43 last 13.19) - open/close 16567 (time 1786761184.44 total 851.56 last 18.45) - open/close 16694 (time 1786761194.45 total 861.58 last 12.68) - open/close 16878 (time 1786761205.36 total 872.49 last 16.86) - open/close 17072 (time 1786761215.38 total 882.51 last 19.36) - open/close 17298 (time 1786761225.38 total 892.51 last 22.59) - open/close 17537 (time 1786761235.38 total 902.51 last 23.90) - open/close 17741 (time 1786761245.73 total 912.86 last 19.71) - open/close 18001 (time 1786761255.75 total 922.87 last 25.96) - open/close 18258 (time 1786761265.75 total 932.88 last 25.69) - open/close 18499 (time 1786761276.06 total 943.18 last 23.38) - open/close 18717 (time 1786761286.51 total 953.63 last 20.87) - open/close 18974 (time 1786761296.51 total 963.64 last 25.69) - open/close 19225 (time 1786761306.51 total 973.64 last 25.09) - open/close 19805 (time 1786761316.51 total 983.64 last 57.99) - open/close 20000 (time 1786761318.99 total 986.11 last 78.91) - open/close 20857 (time 1786761328.99 total 996.12 last 85.65) - open/close 21763 (time 1786761338.99 total 1006.12 last 90.59) - open/close 22703 (time 1786761349.01 total 1016.13 last 93.87) - open/close 23582 (time 1786761359.01 total 1026.14 last 87.86) - open/close 24403 (time 1786761369.02 total 1036.15 last 82.03) - open/close 25262 (time 1786761379.02 total 1046.15 last 85.89) - open/close 26048 (time 1786761389.02 total 1056.15 last 78.58) - open/close 26830 (time 1786761399.03 total 1066.16 last 78.15) - open/close 27651 (time 1786761409.04 total 1076.17 last 82.01) - open/close 28353 (time 1786761419.05 total 1086.18 last 70.14) - open/close 29115 (time 1786761429.05 total 1096.18 last 76.16) - open/close 29835 (time 1786761439.07 total 1106.19 last 71.93) - open/close 30000 (time 1786761441.97 total 1109.09 last 56.87) - open/close 30710 (time 1786761451.99 total 1119.12 last 70.80) - open/close 31341 (time 1786761462.02 total 1129.14 last 62.96) - open/close 31992 (time 1786761472.03 total 1139.16 last 65.03) - open/close 32774 (time 1786761482.05 total 1149.18 last 78.01) - open/close 33438 (time 1786761492.06 total 1159.18 last 66.37) - open/close 34250 (time 1786761502.06 total 1169.19 last 81.17) - open/close 35097 (time 1786761512.06 total 1179.19 last 84.66) - open/close 35968 (time 1786761522.07 total 1189.20 last 87.04) - open/close 36801 (time 1786761532.09 total 1199.21 last 83.16) - open/close 37668 (time 1786761542.10 total 1209.22 last 86.62) - open/close 38495 (time 1786761552.10 total 1219.23 last 82.66) - open/close 39335 (time 1786761562.12 total 1229.25 last 83.81) - open/close 40000 (time 1786761570.39 total 1237.51 last 80.49) - open/close 40900 (time 1786761580.39 total 1247.52 last 89.94) - open/close 41737 (time 1786761590.41 total 1257.53 last 83.58) - open/close 42556 (time 1786761600.41 total 1267.54 last 81.87) - open/close 43374 (time 1786761610.41 total 1277.54 last 81.80) - open/close 44145 (time 1786761620.41 total 1287.54 last 77.08) - open/close 44939 (time 1786761630.42 total 1297.55 last 79.30) - open/close 45767 (time 1786761640.42 total 1307.55 last 82.80) - open/close 46618 (time 1786761650.43 total 1317.56 last 85.05) - open/close 47518 (time 1786761660.45 total 1327.58 last 89.81) - open/close 48387 (time 1786761670.47 total 1337.60 last 86.73) - open/close 49151 (time 1786761680.48 total 1347.61 last 76.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1353.28 seconds: 36.67 ops/second - unlinked 0 (time 1786761691 ; total 0 ; last 0) - unlinked 10000 (time 1786761813 ; total 122 ; last 122) - unlinked 20000 (time 1786761933 ; total 242 ; last 120) - unlinked 30000 (time 1786762053 ; total 362 ; last 120) - unlinked 40000 (time 1786762190 ; total 499 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 613 seconds: 80.964111 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: modules unloaded. pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2