== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:11:00 (1746069060) start mds service on oleg121-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg121-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg121-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 272 (time 1746069122.54 total 11.01 last 24.71) - open/close 452 (time 1746069133.29 total 21.76 last 16.74) - open/close 571 (time 1746069143.80 total 32.27 last 11.32) - open/close 759 (time 1746069154.34 total 42.81 last 17.85) - open/close 930 (time 1746069164.43 total 52.91 last 16.94) - open/close 1002 (time 1746069175.44 total 63.91 last 6.54) - open/close 1103 (time 1746069185.54 total 74.01 last 10.00) - open/close 1251 (time 1746069196.02 total 84.49 last 14.12) - open/close 1368 (time 1746069208.49 total 96.97 last 9.38) - open/close 1687 (time 1746069218.90 total 107.37 last 30.66) - open/close 1982 (time 1746069229.49 total 117.96 last 27.85) - open/close 2174 (time 1746069239.84 total 128.31 last 18.55) - open/close 2287 (time 1746069250.20 total 138.67 last 10.91) - open/close 2425 (time 1746069261.13 total 149.61 last 12.62) - open/close 2591 (time 1746069271.69 total 160.16 last 15.73) - open/close 2735 (time 1746069282.42 total 170.89 last 13.42) - open/close 2784 (time 1746069293.04 total 181.51 last 4.62) - open/close 2865 (time 1746069303.04 total 191.51 last 8.10) - open/close 3130 (time 1746069313.80 total 202.27 last 24.64) - open/close 3395 (time 1746069324.76 total 213.23 last 24.17) - open/close 3740 (time 1746069334.76 total 223.23 last 34.50) - open/close 4039 (time 1746069345.11 total 233.58 last 28.89) - open/close 4238 (time 1746069355.28 total 243.75 last 19.56) - open/close 4324 (time 1746069365.29 total 253.76 last 8.59) - open/close 4501 (time 1746069376.06 total 264.54 last 16.43) - open/close 4664 (time 1746069386.07 total 274.54 last 16.29) - open/close 4893 (time 1746069396.24 total 284.71 last 22.52) - open/close 5194 (time 1746069407.14 total 295.61 last 27.62) - open/close 5387 (time 1746069418.08 total 306.55 last 17.64) - open/close 5505 (time 1746069430.64 total 319.11 last 9.40) - open/close 5631 (time 1746069440.98 total 329.46 last 12.18) - open/close 5851 (time 1746069451.02 total 339.49 last 21.92) - open/close 5994 (time 1746069461.28 total 349.75 last 13.94) - open/close 6120 (time 1746069471.28 total 359.75 last 12.60) - open/close 6232 (time 1746069481.39 total 369.86 last 11.08) - open/close 6408 (time 1746069492.12 total 380.59 last 16.40) - open/close 6578 (time 1746069502.96 total 391.43 last 15.69) - open/close 6730 (time 1746069513.01 total 401.48 last 15.13) - open/close 6934 (time 1746069524.45 total 412.92 last 17.83) - open/close 7057 (time 1746069535.08 total 423.56 last 11.57) - open/close 7163 (time 1746069546.54 total 435.01 last 9.26) - open/close 7287 (time 1746069557.36 total 445.83 last 11.45) - open/close 7453 (time 1746069567.87 total 456.35 last 15.79) - open/close 7613 (time 1746069578.48 total 466.95 last 15.09) - open/close 7781 (time 1746069588.95 total 477.42 last 16.05) - open/close 7890 (time 1746069599.49 total 487.97 last 10.33) - open/close 8022 (time 1746069610.32 total 498.79 last 12.19) - open/close 8165 (time 1746069620.42 total 508.90 last 14.15) - open/close 8327 (time 1746069631.06 total 519.53 last 15.24) - open/close 8505 (time 1746069641.70 total 530.18 last 16.72) - open/close 8693 (time 1746069651.84 total 540.31 last 18.55) - open/close 8850 (time 1746069662.06 total 550.53 last 15.36) - open/close 9563 (time 1746069672.06 total 560.54 last 71.26) - open/close 10000 (time 1746069675.65 total 564.12 last 121.99) - open/close 11238 (time 1746069685.65 total 574.12 last 123.78) - open/close 12523 (time 1746069695.65 total 584.13 last 128.43) - open/close 14001 (time 1746069705.66 total 594.13 last 147.70) - open/close 15296 (time 1746069715.66 total 604.13 last 129.49) - open/close 16553 (time 1746069725.66 total 614.14 last 125.69) - open/close 17823 (time 1746069735.67 total 624.15 last 126.86) - open/close 18930 (time 1746069745.68 total 634.15 last 110.68) - open/close 20000 (time 1746069755.23 total 643.70 last 111.98) - open/close 21278 (time 1746069765.23 total 653.71 last 127.76) - open/close 22304 (time 1746069775.24 total 663.71 last 102.51) - open/close 23496 (time 1746069785.25 total 673.72 last 119.15) - open/close 24528 (time 1746069795.25 total 683.73 last 103.14) - open/close 25515 (time 1746069805.26 total 693.73 last 98.62) - open/close 26681 (time 1746069815.26 total 703.74 last 116.57) - open/close 27714 (time 1746069825.27 total 713.74 last 103.21) - open/close 28700 (time 1746069835.29 total 723.76 last 98.45) - open/close 29599 (time 1746069845.29 total 733.76 last 89.88) - open/close 30000 (time 1746069849.76 total 738.23 last 89.67) - open/close 31023 (time 1746069859.77 total 748.24 last 102.22) - open/close 31932 (time 1746069869.77 total 758.24 last 90.89) - open/close 32840 (time 1746069879.79 total 768.27 last 90.59) - open/close 33738 (time 1746069889.80 total 778.27 last 89.76) - open/close 34753 (time 1746069899.80 total 788.27 last 101.46) - open/close 35668 (time 1746069909.81 total 798.28 last 91.45) - open/close 36545 (time 1746069919.81 total 808.28 last 87.67) - open/close 37397 (time 1746069929.82 total 818.29 last 85.15) - open/close 38461 (time 1746069939.82 total 828.30 last 106.31) - open/close 39367 (time 1746069949.83 total 838.30 last 90.57) - open/close 40000 (time 1746069956.59 total 845.06 last 93.57) - open/close 40972 (time 1746069966.60 total 855.08 last 97.10) - open/close 41865 (time 1746069976.60 total 865.08 last 89.28) - open/close 42845 (time 1746069986.61 total 875.08 last 97.97) - open/close 43873 (time 1746069996.61 total 885.08 last 102.78) - open/close 44947 (time 1746070006.61 total 895.08 last 107.37) - open/close 45977 (time 1746070016.61 total 905.09 last 102.98) - open/close 46926 (time 1746070026.62 total 915.09 last 94.88) - open/close 47860 (time 1746070036.62 total 925.09 last 93.39) - open/close 48745 (time 1746070046.62 total 935.10 last 88.43) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 944.48 seconds: 52.55 ops/second - unlinked 0 (time 1746070058 ; total 0 ; last 0) - unlinked 10000 (time 1746070147 ; total 89 ; last 89) - unlinked 20000 (time 1746070212 ; total 154 ; last 65) - unlinked 30000 (time 1746070268 ; total 210 ; last 56) - unlinked 40000 (time 1746070330 ; total 272 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 331 seconds: 149.945618 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2