== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:34:34 (1746066874) start mds service on oleg427-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg427-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 280 (time 1746066934.11 total 10.79 last 25.95) - open/close 408 (time 1746066944.77 total 21.45 last 12.01) - open/close 661 (time 1746066955.72 total 32.41 last 23.09) - open/close 960 (time 1746066965.73 total 42.42 last 29.87) - open/close 1243 (time 1746066976.33 total 53.02 last 26.70) - open/close 1512 (time 1746066986.79 total 63.47 last 25.73) - open/close 1725 (time 1746066997.30 total 73.98 last 20.27) - open/close 1910 (time 1746067007.30 total 83.98 last 18.49) - open/close 2103 (time 1746067017.57 total 94.25 last 18.80) - open/close 2196 (time 1746067029.06 total 105.74 last 8.09) - open/close 2347 (time 1746067041.20 total 117.88 last 12.44) - open/close 2631 (time 1746067051.21 total 127.89 last 28.39) - open/close 2874 (time 1746067061.84 total 138.52 last 22.85) - open/close 3141 (time 1746067071.85 total 148.53 last 26.68) - open/close 3422 (time 1746067081.88 total 158.56 last 28.01) - open/close 3647 (time 1746067092.34 total 169.02 last 21.51) - open/close 3938 (time 1746067102.88 total 179.56 last 27.62) - open/close 4200 (time 1746067113.50 total 190.18 last 24.66) - open/close 4472 (time 1746067124.01 total 200.69 last 25.88) - open/close 4704 (time 1746067134.05 total 210.74 last 23.10) - open/close 4947 (time 1746067144.25 total 220.93 last 23.83) - open/close 5189 (time 1746067154.38 total 231.06 last 23.89) - open/close 5459 (time 1746067164.41 total 241.09 last 26.93) - open/close 5681 (time 1746067174.71 total 251.39 last 21.55) - open/close 5824 (time 1746067185.97 total 262.65 last 12.70) - open/close 6083 (time 1746067196.12 total 272.80 last 25.52) - open/close 6306 (time 1746067206.12 total 282.80 last 22.30) - open/close 6455 (time 1746067216.30 total 292.98 last 14.64) - open/close 6672 (time 1746067226.90 total 303.58 last 20.47) - open/close 6912 (time 1746067237.86 total 314.54 last 21.90) - open/close 7126 (time 1746067248.31 total 324.99 last 20.49) - open/close 7224 (time 1746067259.12 total 335.80 last 9.06) - open/close 7463 (time 1746067270.06 total 346.75 last 21.84) - open/close 7808 (time 1746067280.57 total 357.25 last 32.85) - open/close 8026 (time 1746067291.31 total 367.99 last 20.29) - open/close 8270 (time 1746067301.51 total 378.20 last 23.91) - open/close 8537 (time 1746067312.44 total 389.12 last 24.45) - open/close 8837 (time 1746067322.74 total 399.42 last 29.11) - open/close 9046 (time 1746067332.74 total 409.42 last 20.90) - open/close 9211 (time 1746067343.01 total 419.69 last 16.07) - open/close 10000 (time 1746067352.63 total 429.31 last 82.00) - open/close 12022 (time 1746067362.63 total 439.31 last 202.14) - open/close 13528 (time 1746067372.64 total 449.32 last 150.57) - open/close 15239 (time 1746067382.64 total 459.32 last 171.07) - open/close 16682 (time 1746067392.65 total 469.33 last 144.17) - open/close 17994 (time 1746067402.65 total 479.33 last 131.18) - open/close 19460 (time 1746067412.65 total 489.33 last 146.55) - open/close 20000 (time 1746067415.89 total 492.57 last 166.66) - open/close 21633 (time 1746067425.89 total 502.57 last 163.26) - open/close 22931 (time 1746067435.90 total 512.58 last 129.71) - open/close 23986 (time 1746067445.91 total 522.59 last 105.41) - open/close 25069 (time 1746067455.92 total 532.60 last 108.22) - open/close 26187 (time 1746067465.92 total 542.60 last 111.78) - open/close 27321 (time 1746067475.93 total 552.61 last 113.24) - open/close 28439 (time 1746067485.93 total 562.61 last 111.79) - open/close 29537 (time 1746067495.94 total 572.62 last 109.69) - open/close 30000 (time 1746067500.25 total 576.93 last 107.58) - open/close 31207 (time 1746067510.25 total 586.93 last 120.65) - open/close 32802 (time 1746067520.26 total 596.95 last 159.29) - open/close 34019 (time 1746067530.27 total 606.95 last 121.65) - open/close 35303 (time 1746067540.27 total 616.95 last 128.34) - open/close 36548 (time 1746067550.28 total 626.96 last 124.44) - open/close 37647 (time 1746067560.29 total 636.97 last 109.78) - open/close 38726 (time 1746067570.29 total 646.97 last 107.89) - open/close 39888 (time 1746067580.30 total 656.98 last 116.11) - open/close 41038 (time 1746067590.31 total 666.99 last 114.87) - open/close 42430 (time 1746067600.31 total 676.99 last 139.16) - open/close 43888 (time 1746067610.31 total 686.99 last 145.77) - open/close 45171 (time 1746067620.32 total 697.00 last 128.26) - open/close 46338 (time 1746067630.32 total 707.00 last 116.69) - open/close 47658 (time 1746067640.32 total 717.00 last 131.93) - open/close 49170 (time 1746067650.32 total 727.01 last 151.18) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 730.23 seconds: 67.97 ops/second - unlinked 0 (time 1746067655 ; total 0 ; last 0) - unlinked 10000 (time 1746067699 ; total 44 ; last 44) - unlinked 20000 (time 1746067739 ; total 84 ; last 40) - unlinked 30000 (time 1746067790 ; total 135 ; last 51) - unlinked 40000 (time 1746067835 ; total 180 ; last 45) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 228 seconds: 217.684204 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2