== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:27:52 (1775870872) start mds service on oleg247-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg247-server: detected 4 online CPUs by sysfs oleg247-server: MODOPTS_LIBCFS= oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg247-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg247-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg247-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1775870984.54 total 10.64 last 13.35) - open/close 416 (time 1775870997.60 total 23.70 last 20.98) - open/close 723 (time 1775871007.61 total 33.71 last 30.66) - open/close 867 (time 1775871018.60 total 44.70 last 13.11) - open/close 1149 (time 1775871028.89 total 54.99 last 27.39) - open/close 1370 (time 1775871039.09 total 65.19 last 21.67) - open/close 1542 (time 1775871049.70 total 75.80 last 16.21) - open/close 1762 (time 1775871059.72 total 85.82 last 21.97) - open/close 2042 (time 1775871070.55 total 96.65 last 25.84) - open/close 2283 (time 1775871081.36 total 107.46 last 22.30) - open/close 2473 (time 1775871091.38 total 117.48 last 18.96) - open/close 2687 (time 1775871101.60 total 127.70 last 20.94) - open/close 2859 (time 1775871111.83 total 137.93 last 16.82) - open/close 3039 (time 1775871122.71 total 148.81 last 16.54) - open/close 3170 (time 1775871132.83 total 158.93 last 12.94) - open/close 3325 (time 1775871142.93 total 169.03 last 15.36) - open/close 3465 (time 1775871152.95 total 179.05 last 13.97) - open/close 3725 (time 1775871163.92 total 190.02 last 23.70) - open/close 3978 (time 1775871175.66 total 201.76 last 21.56) - open/close 4089 (time 1775871186.39 total 212.49 last 10.34) - open/close 4380 (time 1775871197.24 total 223.34 last 26.83) - open/close 4566 (time 1775871207.25 total 233.35 last 18.58) - open/close 4678 (time 1775871217.25 total 243.35 last 11.19) - open/close 4782 (time 1775871227.27 total 253.37 last 10.39) - open/close 4901 (time 1775871237.28 total 263.38 last 11.89) - open/close 5004 (time 1775871247.90 total 274.00 last 9.69) - open/close 5085 (time 1775871258.03 total 284.13 last 8.00) - open/close 5300 (time 1775871268.97 total 295.07 last 19.65) - open/close 5502 (time 1775871278.97 total 305.07 last 20.20) - open/close 5767 (time 1775871288.98 total 315.08 last 26.49) - open/close 5939 (time 1775871299.31 total 325.41 last 16.65) - open/close 6038 (time 1775871310.18 total 336.28 last 9.10) - open/close 6182 (time 1775871321.10 total 347.20 last 13.19) - open/close 6428 (time 1775871331.12 total 357.22 last 24.57) - open/close 6674 (time 1775871341.14 total 367.24 last 24.55) - open/close 6837 (time 1775871351.65 total 377.75 last 15.50) - open/close 7130 (time 1775871361.67 total 387.77 last 29.25) - open/close 7287 (time 1775871371.70 total 397.80 last 15.65) - open/close 7516 (time 1775871382.12 total 408.22 last 21.98) - open/close 7652 (time 1775871394.60 total 420.70 last 10.90) - open/close 7870 (time 1775871405.33 total 431.43 last 20.31) - open/close 8081 (time 1775871416.41 total 442.51 last 19.05) - open/close 8245 (time 1775871428.95 total 455.05 last 13.08) - open/close 8439 (time 1775871439.68 total 465.78 last 18.08) - open/close 8721 (time 1775871449.68 total 475.78 last 28.20) - open/close 8923 (time 1775871460.63 total 486.73 last 18.44) - open/close 9168 (time 1775871470.65 total 496.75 last 24.46) - open/close 9371 (time 1775871482.15 total 508.25 last 17.65) - open/close 9644 (time 1775871492.17 total 518.27 last 27.26) - open/close 9865 (time 1775871502.63 total 528.73 last 21.12) - open/close 10000 (time 1775871510.16 total 536.26 last 17.93) - open/close 10206 (time 1775871520.63 total 546.73 last 19.67) - open/close 10465 (time 1775871530.64 total 556.74 last 25.89) - open/close 10697 (time 1775871541.34 total 567.44 last 21.68) - open/close 10909 (time 1775871551.68 total 577.78 last 20.49) - open/close 11094 (time 1775871564.42 total 590.52 last 14.52) - open/close 11156 (time 1775871574.84 total 600.94 last 5.95) - open/close 11296 (time 1775871586.03 total 612.13 last 12.51) - open/close 11463 (time 1775871596.19 total 622.29 last 16.43) - open/close 11696 (time 1775871606.68 total 632.78 last 22.22) - open/close 11845 (time 1775871617.45 total 643.55 last 13.84) - open/close 11961 (time 1775871627.99 total 654.09 last 11.01) - open/close 12090 (time 1775871638.33 total 664.43 last 12.47) - open/close 12234 (time 1775871650.98 total 677.08 last 11.38) - open/close 12318 (time 1775871661.53 total 687.63 last 7.96) - open/close 12432 (time 1775871672.35 total 698.45 last 10.54) - open/close 12578 (time 1775871682.36 total 708.46 last 14.59) - open/close 12658 (time 1775871693.16 total 719.26 last 7.41) - open/close 12812 (time 1775871703.16 total 729.26 last 15.40) - open/close 12969 (time 1775871713.69 total 739.79 last 14.91) - open/close 13169 (time 1775871723.69 total 749.79 last 19.99) - open/close 13426 (time 1775871733.70 total 759.80 last 25.68) - open/close 13665 (time 1775871743.71 total 769.81 last 23.88) - open/close 13926 (time 1775871754.23 total 780.33 last 24.81) - open/close 14172 (time 1775871765.03 total 791.13 last 22.78) - open/close 14429 (time 1775871775.57 total 801.67 last 24.37) - open/close 14726 (time 1775871785.59 total 811.69 last 29.65) - open/close 14878 (time 1775871796.08 total 822.18 last 14.50) - open/close 15035 (time 1775871806.09 total 832.19 last 15.68) - open/close 15237 (time 1775871816.09 total 842.19 last 20.19) - open/close 15387 (time 1775871826.68 total 852.78 last 14.17) - open/close 15448 (time 1775871836.85 total 862.95 last 6.00) - open/close 15563 (time 1775871847.40 total 873.50 last 10.90) - open/close 15705 (time 1775871857.42 total 883.52 last 14.18) - open/close 15933 (time 1775871867.42 total 893.52 last 22.79) - open/close 16071 (time 1775871877.85 total 903.95 last 13.24) - open/close 16193 (time 1775871887.85 total 913.95 last 12.20) - open/close 16387 (time 1775871900.07 total 926.17 last 15.87) - open/close 16506 (time 1775871911.60 total 937.70 last 10.32) - open/close 16651 (time 1775871921.75 total 947.85 last 14.28) - open/close 16717 (time 1775871931.94 total 958.04 last 6.48) - open/close 16838 (time 1775871941.94 total 968.04 last 12.10) - open/close 16980 (time 1775871952.99 total 979.09 last 12.85) - open/close 17294 (time 1775871965.49 total 991.59 last 25.11) - open/close 17523 (time 1775871975.49 total 1001.59 last 22.90) - open/close 17759 (time 1775871985.59 total 1011.69 last 23.36) - open/close 17962 (time 1775871996.45 total 1022.55 last 18.69) - open/close 18279 (time 1775872006.46 total 1032.56 last 31.68) - open/close 19232 (time 1775872016.46 total 1042.56 last 95.26) - open/close 20000 (time 1775872023.47 total 1049.57 last 109.53) - open/close 21003 (time 1775872033.48 total 1059.58 last 100.27) - open/close 22034 (time 1775872043.48 total 1069.58 last 103.10) - open/close 22714 (time 1775872053.48 total 1079.58 last 67.97) - open/close 23495 (time 1775872063.48 total 1089.58 last 78.08) - open/close 24398 (time 1775872073.49 total 1099.59 last 90.27) - open/close 25292 (time 1775872083.49 total 1109.59 last 89.40) - open/close 26311 (time 1775872093.49 total 1119.59 last 101.83) - open/close 27263 (time 1775872103.50 total 1129.60 last 95.16) - open/close 28422 (time 1775872113.50 total 1139.60 last 115.90) - open/close 29426 (time 1775872123.50 total 1149.60 last 100.39) - open/close 30000 (time 1775872129.53 total 1155.63 last 95.26) - open/close 31025 (time 1775872139.53 total 1165.63 last 102.44) - open/close 32031 (time 1775872149.53 total 1175.63 last 100.57) - open/close 32893 (time 1775872159.54 total 1185.64 last 86.13) - open/close 33846 (time 1775872169.55 total 1195.65 last 95.20) - open/close 34614 (time 1775872179.56 total 1205.66 last 76.71) - open/close 35366 (time 1775872189.57 total 1215.67 last 75.16) - open/close 36426 (time 1775872199.58 total 1225.68 last 105.90) - open/close 37367 (time 1775872209.59 total 1235.69 last 94.04) - open/close 38496 (time 1775872219.59 total 1245.69 last 112.89) - open/close 39538 (time 1775872229.59 total 1255.69 last 104.18) - open/close 40000 (time 1775872234.16 total 1260.26 last 101.03) - open/close 40964 (time 1775872244.17 total 1270.27 last 96.29) - open/close 41862 (time 1775872254.18 total 1280.28 last 89.74) - open/close 42594 (time 1775872264.19 total 1290.29 last 73.10) - open/close 43158 (time 1775872274.22 total 1300.32 last 56.25) - open/close 43845 (time 1775872284.23 total 1310.33 last 68.63) - open/close 44494 (time 1775872294.25 total 1320.35 last 64.78) - open/close 45232 (time 1775872304.27 total 1330.37 last 73.65) - open/close 45999 (time 1775872314.27 total 1340.37 last 76.69) - open/close 46557 (time 1775872324.27 total 1350.37 last 55.79) - open/close 47310 (time 1775872334.27 total 1360.37 last 75.28) - open/close 48110 (time 1775872344.28 total 1370.38 last 79.99) - open/close 48917 (time 1775872354.28 total 1380.38 last 80.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1387.81 seconds: 35.76 ops/second - unlinked 0 (time 1775872366 ; total 0 ; last 0) - unlinked 10000 (time 1775872475 ; total 109 ; last 109) - unlinked 20000 (time 1775872570 ; total 204 ; last 95) - unlinked 30000 (time 1775872628 ; total 262 ; last 58) - unlinked 40000 (time 1775872690 ; total 324 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 384 seconds: 129.250000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg247-server unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local oleg247-server: modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2