== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:13:45 (1746148425) start mds service on oleg145-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing load_modules_local oleg145-server: Loading modules from /home/green/git/lustre-release/lustre oleg145-server: detected 4 online CPUs by sysfs oleg145-server: MODOPTS_LIBCFS= oleg145-server: Force libcfs to create 2 CPU partitions oleg145-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg145-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg145-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg145-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg145-server: oleg145-server.virtnet: executing set_default_debug -1 all pdsh@oleg145-client: oleg145-server: ssh exited with exit code 1 Started lustre-OST0000 oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg145-client.virtnet: -o user_xattr,flock 192.168.201.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 146 (time 1746148494.44 total 10.00 last 14.60) - open/close 312 (time 1746148505.34 total 20.90 last 15.23) - open/close 446 (time 1746148515.64 total 31.19 last 13.02) - open/close 620 (time 1746148526.29 total 41.85 last 16.33) - open/close 715 (time 1746148536.73 total 52.29 last 9.10) - open/close 869 (time 1746148547.28 total 62.84 last 14.59) - open/close 1019 (time 1746148557.66 total 73.21 last 14.46) - open/close 1157 (time 1746148567.66 total 83.22 last 13.80) - open/close 1289 (time 1746148577.66 total 93.22 last 13.20) - open/close 1543 (time 1746148588.25 total 103.80 last 23.99) - open/close 1726 (time 1746148598.73 total 114.29 last 17.45) - open/close 1891 (time 1746148609.34 total 124.90 last 15.56) - open/close 2033 (time 1746148619.35 total 134.91 last 14.19) - open/close 2168 (time 1746148630.19 total 145.75 last 12.45) - open/close 2360 (time 1746148641.21 total 156.77 last 17.42) - open/close 2589 (time 1746148651.89 total 167.45 last 21.45) - open/close 2806 (time 1746148662.65 total 178.21 last 20.16) - open/close 3011 (time 1746148672.93 total 188.49 last 19.95) - open/close 3140 (time 1746148683.83 total 199.39 last 11.84) - open/close 3312 (time 1746148693.84 total 209.40 last 17.18) - open/close 3391 (time 1746148703.94 total 219.50 last 7.82) - open/close 3525 (time 1746148713.95 total 229.51 last 13.39) - open/close 3620 (time 1746148724.30 total 239.86 last 9.18) - open/close 3767 (time 1746148734.48 total 250.04 last 14.44) - open/close 3988 (time 1746148745.19 total 260.74 last 20.65) - open/close 4215 (time 1746148755.41 total 270.97 last 22.20) - open/close 4389 (time 1746148765.53 total 281.09 last 17.19) - open/close 4535 (time 1746148775.54 total 291.10 last 14.59) - open/close 4644 (time 1746148786.06 total 301.62 last 10.36) - open/close 4772 (time 1746148796.07 total 311.62 last 12.80) - open/close 4954 (time 1746148806.51 total 322.07 last 17.43) - open/close 5119 (time 1746148816.77 total 332.33 last 16.08) - open/close 5313 (time 1746148826.98 total 342.54 last 18.99) - open/close 5452 (time 1746148837.84 total 353.40 last 12.81) - open/close 5592 (time 1746148848.83 total 364.39 last 12.74) - open/close 5729 (time 1746148859.58 total 375.14 last 12.74) - open/close 5891 (time 1746148869.69 total 385.25 last 16.02) - open/close 6168 (time 1746148880.60 total 396.16 last 25.40) - open/close 6384 (time 1746148891.58 total 407.14 last 19.66) - open/close 6506 (time 1746148902.38 total 417.94 last 11.30) - open/close 6644 (time 1746148912.38 total 427.94 last 13.79) - open/close 6790 (time 1746148922.65 total 438.21 last 14.22) - open/close 6942 (time 1746148932.85 total 448.41 last 14.90) - open/close 7088 (time 1746148943.25 total 458.81 last 14.04) - open/close 7198 (time 1746148953.90 total 469.46 last 10.32) - open/close 7332 (time 1746148964.65 total 480.21 last 12.47) - open/close 7498 (time 1746148975.30 total 490.86 last 15.58) - open/close 7627 (time 1746148986.20 total 501.76 last 11.84) - open/close 7766 (time 1746148996.22 total 511.77 last 13.88) - open/close 7954 (time 1746149006.40 total 521.95 last 18.47) - open/close 8183 (time 1746149017.21 total 532.77 last 21.18) - open/close 8380 (time 1746149027.22 total 542.78 last 19.67) - open/close 8590 (time 1746149037.24 total 552.80 last 20.97) - open/close 8813 (time 1746149048.25 total 563.80 last 20.26) - open/close 9044 (time 1746149058.94 total 574.50 last 21.60) - open/close 9212 (time 1746149069.36 total 584.91 last 16.13) - open/close 9333 (time 1746149079.87 total 595.43 last 11.51) - open/close 9472 (time 1746149090.60 total 606.16 last 12.95) - open/close 9552 (time 1746149100.65 total 616.21 last 7.96) - open/close 9996 (time 1746149110.65 total 626.21 last 44.39) - open/close 10889 (time 1746149120.68 total 636.24 last 89.09) - open/close 11943 (time 1746149130.68 total 646.24 last 105.38) - open/close 13007 (time 1746149140.68 total 656.24 last 106.34) - open/close 13978 (time 1746149150.69 total 666.25 last 97.02) - open/close 14934 (time 1746149160.70 total 676.26 last 95.54) - open/close 15851 (time 1746149170.70 total 686.26 last 91.68) - open/close 16860 (time 1746149180.71 total 696.27 last 100.83) - open/close 17833 (time 1746149190.72 total 706.27 last 97.23) - open/close 18730 (time 1746149200.73 total 716.29 last 89.59) - open/close 19602 (time 1746149210.73 total 726.29 last 87.14) - open/close 20000 (time 1746149214.98 total 730.54 last 93.66) - open/close 21131 (time 1746149224.99 total 740.55 last 113.02) - open/close 22297 (time 1746149234.99 total 750.55 last 116.55) - open/close 23143 (time 1746149245.00 total 760.56 last 84.56) - open/close 23999 (time 1746149255.01 total 770.57 last 85.48) - open/close 24898 (time 1746149265.02 total 780.58 last 89.84) - open/close 25878 (time 1746149275.02 total 790.58 last 97.98) - open/close 26827 (time 1746149285.04 total 800.60 last 94.73) - open/close 27616 (time 1746149295.05 total 810.61 last 78.83) - open/close 28309 (time 1746149305.06 total 820.62 last 69.24) - open/close 29020 (time 1746149315.06 total 830.62 last 71.10) - open/close 29836 (time 1746149325.07 total 840.63 last 81.46) - open/close 30000 (time 1746149327.41 total 842.97 last 70.29) - open/close 30585 (time 1746149337.43 total 852.99 last 58.39) - open/close 31252 (time 1746149347.44 total 863.00 last 66.59) - open/close 31908 (time 1746149357.46 total 873.01 last 65.51) - open/close 32781 (time 1746149367.46 total 883.02 last 87.25) - open/close 33761 (time 1746149377.48 total 893.03 last 97.86) - open/close 34501 (time 1746149387.48 total 903.04 last 73.97) - open/close 35147 (time 1746149397.49 total 913.05 last 64.56) - open/close 35783 (time 1746149407.49 total 923.05 last 63.57) - open/close 36442 (time 1746149417.50 total 933.06 last 65.82) - open/close 37226 (time 1746149427.51 total 943.07 last 78.35) - open/close 38007 (time 1746149437.52 total 953.08 last 78.03) - open/close 38719 (time 1746149447.53 total 963.09 last 71.12) - open/close 39539 (time 1746149457.54 total 973.10 last 81.89) - open/close 40000 (time 1746149463.84 total 979.40 last 73.17) - open/close 40720 (time 1746149473.85 total 989.41 last 71.93) - open/close 41606 (time 1746149483.86 total 999.41 last 88.58) - open/close 42398 (time 1746149493.86 total 1009.42 last 79.14) - open/close 43089 (time 1746149503.87 total 1019.43 last 69.07) - open/close 43808 (time 1746149513.87 total 1029.43 last 71.90) - open/close 44580 (time 1746149523.87 total 1039.43 last 77.20) - open/close 45447 (time 1746149533.88 total 1049.44 last 86.62) - open/close 46202 (time 1746149543.88 total 1059.44 last 75.48) - open/close 46894 (time 1746149553.89 total 1069.45 last 69.13) - open/close 47545 (time 1746149563.90 total 1079.46 last 65.05) - open/close 48276 (time 1746149573.91 total 1089.47 last 73.00) - open/close 49024 (time 1746149583.92 total 1099.48 last 74.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1107.58 seconds: 44.81 ops/second - unlinked 0 (time 1746149595 ; total 0 ; last 0) - unlinked 10000 (time 1746149738 ; total 143 ; last 143) - unlinked 20000 (time 1746149884 ; total 289 ; last 146) - unlinked 30000 (time 1746150032 ; total 437 ; last 148) - unlinked 40000 (time 1746150173 ; total 578 ; last 141) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 712 seconds: 69.707863 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg145-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg145-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg145-server stop mds service on oleg145-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg145-server unloading modules via unload_modules_local on: 'oleg145-server' oleg145-server: oleg145-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-client: ssh exited with exit code 2 pdsh@oleg145-client: oleg145-server: ssh exited with exit code 2