== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:49:00 (1732430940) start mds service on oleg104-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: detected 4 online CPUs by sysfs oleg104-server: Force libcfs to create 2 CPU partitions oleg104-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg104-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg104-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg104-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 197 (time 1732431034.66 total 10.97 last 17.96) - open/close 413 (time 1732431044.67 total 20.99 last 21.56) - open/close 596 (time 1732431055.48 total 31.79 last 16.94) - open/close 795 (time 1732431065.49 total 41.80 last 19.89) - open/close 996 (time 1732431075.49 total 51.80 last 20.09) - open/close 1175 (time 1732431086.51 total 62.82 last 16.24) - open/close 1329 (time 1732431096.73 total 73.04 last 15.07) - open/close 1469 (time 1732431106.73 total 83.04 last 14.00) - open/close 1659 (time 1732431116.74 total 93.06 last 18.97) - open/close 1870 (time 1732431127.73 total 104.04 last 19.20) - open/close 2116 (time 1732431138.59 total 114.91 last 22.64) - open/close 2354 (time 1732431149.50 total 125.82 last 21.81) - open/close 2560 (time 1732431159.87 total 136.18 last 19.88) - open/close 2764 (time 1732431169.88 total 146.20 last 20.37) - open/close 2905 (time 1732431180.05 total 156.36 last 13.87) - open/close 3096 (time 1732431190.17 total 166.49 last 18.87) - open/close 3292 (time 1732431200.35 total 176.67 last 19.25) - open/close 3541 (time 1732431210.37 total 186.69 last 24.85) - open/close 3731 (time 1732431221.02 total 197.33 last 17.85) - open/close 3939 (time 1732431231.59 total 207.91 last 19.67) - open/close 4132 (time 1732431242.49 total 218.81 last 17.71) - open/close 4312 (time 1732431254.04 total 230.35 last 15.59) - open/close 4529 (time 1732431264.94 total 241.25 last 19.90) - open/close 4723 (time 1732431274.95 total 251.26 last 19.38) - open/close 4918 (time 1732431285.65 total 261.96 last 18.23) - open/close 5142 (time 1732431295.99 total 272.31 last 21.65) - open/close 5356 (time 1732431306.84 total 283.15 last 19.73) - open/close 5543 (time 1732431316.84 total 293.16 last 18.70) - open/close 5755 (time 1732431327.14 total 303.45 last 20.59) - open/close 5956 (time 1732431337.15 total 313.47 last 20.07) - open/close 6111 (time 1732431347.21 total 323.53 last 15.41) - open/close 6259 (time 1732431358.05 total 334.36 last 13.66) - open/close 6503 (time 1732431368.05 total 344.37 last 24.38) - open/close 6666 (time 1732431378.43 total 354.75 last 15.71) - open/close 6854 (time 1732431389.26 total 365.58 last 17.36) - open/close 7055 (time 1732431399.27 total 375.58 last 20.09) - open/close 7267 (time 1732431409.51 total 385.82 last 20.70) - open/close 7459 (time 1732431419.53 total 395.84 last 19.16) - open/close 7657 (time 1732431430.41 total 406.72 last 18.20) - open/close 7850 (time 1732431440.99 total 417.30 last 18.24) - open/close 8047 (time 1732431451.72 total 428.03 last 18.36) - open/close 8241 (time 1732431462.36 total 438.67 last 18.23) - open/close 8434 (time 1732431473.10 total 449.42 last 17.97) - open/close 8624 (time 1732431483.12 total 459.44 last 18.96) - open/close 8816 (time 1732431493.27 total 469.58 last 18.93) - open/close 9061 (time 1732431503.28 total 479.59 last 24.47) - open/close 9282 (time 1732431513.72 total 490.04 last 21.16) - open/close 9514 (time 1732431524.07 total 500.39 last 22.41) - open/close 9685 (time 1732431534.36 total 510.67 last 16.63) - open/close 9904 (time 1732431544.64 total 520.95 last 21.30) - open/close 10000 (time 1732431549.46 total 525.78 last 19.90) - open/close 10181 (time 1732431559.47 total 535.79 last 18.09) - open/close 10383 (time 1732431569.92 total 546.23 last 19.34) - open/close 10574 (time 1732431579.93 total 556.24 last 19.08) - open/close 10755 (time 1732431590.38 total 566.70 last 17.31) - open/close 10945 (time 1732431600.45 total 576.76 last 18.88) - open/close 11173 (time 1732431610.97 total 587.28 last 21.67) - open/close 11408 (time 1732431621.82 total 598.13 last 21.66) - open/close 11596 (time 1732431632.64 total 608.96 last 17.37) - open/close 11737 (time 1732431642.67 total 618.99 last 14.06) - open/close 11938 (time 1732431652.76 total 629.08 last 19.92) - open/close 12130 (time 1732431662.78 total 639.09 last 19.18) - open/close 12311 (time 1732431673.78 total 650.10 last 16.45) - open/close 12516 (time 1732431684.55 total 660.87 last 19.03) - open/close 12707 (time 1732431694.56 total 670.87 last 19.09) - open/close 12895 (time 1732431704.56 total 680.88 last 18.79) - open/close 13084 (time 1732431714.96 total 691.27 last 18.18) - open/close 13315 (time 1732431725.86 total 702.17 last 21.19) - open/close 13517 (time 1732431736.29 total 712.61 last 19.36) - open/close 13710 (time 1732431746.83 total 723.15 last 18.31) - open/close 13908 (time 1732431757.31 total 733.63 last 18.89) - open/close 14087 (time 1732431767.53 total 743.84 last 17.52) - open/close 14324 (time 1732431778.56 total 754.87 last 21.49) - open/close 14576 (time 1732431789.08 total 765.39 last 23.95) - open/close 14763 (time 1732431799.09 total 775.41 last 18.68) - open/close 14964 (time 1732431810.06 total 786.37 last 18.33) - open/close 15145 (time 1732431820.06 total 796.37 last 18.10) - open/close 15489 (time 1732431830.07 total 806.38 last 34.37) - open/close 16185 (time 1732431840.07 total 816.38 last 69.57) - open/close 16824 (time 1732431850.08 total 826.40 last 63.80) - open/close 17429 (time 1732431860.11 total 836.43 last 60.32) - open/close 18091 (time 1732431870.12 total 846.43 last 66.16) - open/close 18698 (time 1732431880.12 total 856.44 last 60.68) - open/close 19334 (time 1732431890.13 total 866.45 last 63.54) - open/close 20000 (time 1732431899.29 total 875.61 last 72.68) - open/close 20708 (time 1732431909.30 total 885.61 last 70.78) - open/close 21512 (time 1732431919.30 total 895.62 last 80.38) - open/close 22267 (time 1732431929.31 total 905.63 last 75.41) - open/close 22923 (time 1732431939.32 total 915.64 last 65.55) - open/close 23573 (time 1732431949.35 total 925.66 last 64.81) - open/close 24204 (time 1732431959.35 total 935.67 last 63.07) - open/close 24888 (time 1732431969.36 total 945.67 last 68.39) - open/close 25604 (time 1732431979.36 total 955.68 last 71.57) - open/close 26349 (time 1732431989.37 total 965.69 last 74.43) - open/close 27015 (time 1732431999.37 total 975.69 last 66.58) - open/close 27779 (time 1732432009.38 total 985.69 last 76.37) - open/close 28468 (time 1732432019.38 total 995.69 last 68.89) - open/close 29137 (time 1732432029.38 total 1005.70 last 66.86) - open/close 29736 (time 1732432039.39 total 1015.71 last 59.86) - open/close 30000 (time 1732432043.36 total 1019.68 last 66.51) - open/close 30630 (time 1732432053.36 total 1029.68 last 62.99) - open/close 31221 (time 1732432063.37 total 1039.69 last 59.04) - open/close 31895 (time 1732432073.38 total 1049.70 last 67.32) - open/close 32483 (time 1732432083.40 total 1059.71 last 58.72) - open/close 33119 (time 1732432093.41 total 1069.73 last 63.52) - open/close 33829 (time 1732432103.42 total 1079.74 last 70.94) - open/close 34401 (time 1732432113.43 total 1089.75 last 57.13) - open/close 34973 (time 1732432123.44 total 1099.76 last 57.15) - open/close 35579 (time 1732432133.45 total 1109.77 last 60.53) - open/close 36242 (time 1732432143.46 total 1119.78 last 66.25) - open/close 36858 (time 1732432153.46 total 1129.78 last 61.57) - open/close 37560 (time 1732432163.47 total 1139.79 last 70.14) - open/close 38202 (time 1732432173.48 total 1149.79 last 64.17) - open/close 38897 (time 1732432183.50 total 1159.81 last 69.35) - open/close 39592 (time 1732432193.50 total 1169.82 last 69.47) - open/close 40000 (time 1732432200.42 total 1176.74 last 58.96) - open/close 40626 (time 1732432210.43 total 1186.75 last 62.56) - open/close 41239 (time 1732432220.45 total 1196.76 last 61.21) - open/close 41868 (time 1732432230.45 total 1206.77 last 62.87) - open/close 42538 (time 1732432240.46 total 1216.78 last 66.94) - open/close 43237 (time 1732432250.47 total 1226.79 last 69.80) - open/close 43868 (time 1732432260.49 total 1236.80 last 63.02) - open/close 44631 (time 1732432270.49 total 1246.81 last 76.27) - open/close 45209 (time 1732432280.51 total 1256.82 last 57.71) - open/close 45843 (time 1732432290.52 total 1266.84 last 63.30) - open/close 46444 (time 1732432300.53 total 1276.84 last 60.07) - open/close 47026 (time 1732432310.53 total 1286.85 last 58.18) - open/close 47620 (time 1732432320.53 total 1296.85 last 59.38) - open/close 48253 (time 1732432330.54 total 1306.86 last 63.26) - open/close 49019 (time 1732432340.55 total 1316.86 last 76.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1325.19 seconds: 37.45 ops/second - unlinked 0 (time 1732432351 ; total 0 ; last 0) - unlinked 10000 (time 1732432516 ; total 165 ; last 165) - unlinked 20000 (time 1732432676 ; total 325 ; last 160) - unlinked 30000 (time 1732432830 ; total 479 ; last 154) - unlinked 40000 (time 1732432991 ; total 640 ; last 161) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 787 seconds: 63.064804 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2