== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:11:54 (1746130314) start mds service on oleg347-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: MODOPTS_LIBCFS= oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg347-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 259 (time 1746130375.48 total 10.81 last 23.96) - open/close 506 (time 1746130385.77 total 21.10 last 24.00) - open/close 722 (time 1746130395.82 total 31.15 last 21.49) - open/close 954 (time 1746130405.83 total 41.16 last 23.19) - open/close 1190 (time 1746130416.62 total 51.94 last 21.88) - open/close 1402 (time 1746130427.04 total 62.37 last 20.33) - open/close 1533 (time 1746130437.97 total 73.30 last 11.99) - open/close 1770 (time 1746130447.97 total 83.30 last 23.70) - open/close 1933 (time 1746130457.98 total 93.31 last 16.28) - open/close 2167 (time 1746130470.34 total 105.67 last 18.93) - open/close 2310 (time 1746130480.71 total 116.04 last 13.80) - open/close 2526 (time 1746130490.71 total 126.04 last 21.60) - open/close 2758 (time 1746130500.87 total 136.20 last 22.83) - open/close 2953 (time 1746130511.03 total 146.36 last 19.18) - open/close 3205 (time 1746130521.45 total 156.78 last 24.19) - open/close 3384 (time 1746130531.88 total 167.21 last 17.17) - open/close 3578 (time 1746130542.67 total 178.00 last 17.97) - open/close 3851 (time 1746130553.13 total 188.46 last 26.09) - open/close 4074 (time 1746130563.27 total 198.60 last 21.99) - open/close 4287 (time 1746130574.07 total 209.40 last 19.74) - open/close 4404 (time 1746130584.07 total 219.40 last 11.69) - open/close 4574 (time 1746130594.45 total 229.78 last 16.39) - open/close 4784 (time 1746130605.18 total 240.51 last 19.56) - open/close 4989 (time 1746130615.72 total 251.05 last 19.46) - open/close 5190 (time 1746130626.20 total 261.53 last 19.18) - open/close 5413 (time 1746130636.20 total 271.53 last 22.29) - open/close 5701 (time 1746130646.80 total 282.13 last 27.19) - open/close 5915 (time 1746130657.57 total 292.90 last 19.86) - open/close 6114 (time 1746130668.41 total 303.74 last 18.37) - open/close 6294 (time 1746130678.94 total 314.27 last 17.09) - open/close 6528 (time 1746130689.40 total 324.73 last 22.36) - open/close 6672 (time 1746130699.97 total 335.30 last 13.62) - open/close 6857 (time 1746130710.56 total 345.89 last 17.47) - open/close 7108 (time 1746130721.02 total 356.35 last 24.01) - open/close 7296 (time 1746130731.99 total 367.32 last 17.13) - open/close 7437 (time 1746130742.55 total 377.88 last 13.36) - open/close 7628 (time 1746130752.68 total 388.01 last 18.85) - open/close 7809 (time 1746130762.77 total 398.09 last 17.95) - open/close 7953 (time 1746130773.43 total 408.76 last 13.50) - open/close 8193 (time 1746130783.81 total 419.13 last 23.14) - open/close 8377 (time 1746130793.81 total 429.14 last 18.39) - open/close 9323 (time 1746130803.82 total 439.15 last 94.52) - open/close 10000 (time 1746130808.62 total 443.95 last 141.06) - open/close 11307 (time 1746130818.62 total 453.95 last 130.70) - open/close 12593 (time 1746130828.63 total 463.96 last 128.47) - open/close 13760 (time 1746130838.63 total 473.96 last 116.68) - open/close 14980 (time 1746130848.63 total 483.96 last 121.94) - open/close 16322 (time 1746130858.64 total 493.97 last 134.15) - open/close 17424 (time 1746130868.64 total 503.97 last 110.17) - open/close 18593 (time 1746130878.64 total 513.97 last 116.83) - open/close 19692 (time 1746130888.65 total 523.98 last 109.86) - open/close 20000 (time 1746130890.96 total 526.29 last 133.36) - open/close 21096 (time 1746130900.96 total 536.29 last 109.57) - open/close 22119 (time 1746130910.96 total 546.29 last 102.29) - open/close 23186 (time 1746130920.97 total 556.30 last 106.65) - open/close 24316 (time 1746130930.97 total 566.30 last 112.91) - open/close 25205 (time 1746130940.98 total 576.31 last 88.87) - open/close 26213 (time 1746130950.98 total 586.31 last 100.80) - open/close 27233 (time 1746130960.98 total 596.31 last 101.96) - open/close 28248 (time 1746130970.98 total 606.31 last 101.48) - open/close 29192 (time 1746130980.99 total 616.32 last 94.33) - open/close 30000 (time 1746130989.87 total 625.20 last 91.00) - open/close 30930 (time 1746130999.87 total 635.20 last 92.99) - open/close 31905 (time 1746131009.88 total 645.21 last 97.46) - open/close 32868 (time 1746131019.88 total 655.21 last 96.23) - open/close 33978 (time 1746131029.88 total 665.21 last 111.00) - open/close 35072 (time 1746131039.88 total 675.21 last 109.40) - open/close 36214 (time 1746131049.90 total 685.22 last 114.06) - open/close 37362 (time 1746131059.90 total 695.23 last 114.71) - open/close 38694 (time 1746131069.91 total 705.23 last 133.18) - open/close 39753 (time 1746131079.91 total 715.24 last 105.81) - open/close 40000 (time 1746131082.24 total 717.57 last 106.23) - open/close 41034 (time 1746131092.24 total 727.57 last 103.36) - open/close 42259 (time 1746131102.24 total 737.57 last 122.49) - open/close 43493 (time 1746131112.25 total 747.58 last 123.33) - open/close 44521 (time 1746131122.25 total 757.58 last 102.75) - open/close 45640 (time 1746131132.26 total 767.58 last 111.89) - open/close 46843 (time 1746131142.26 total 777.59 last 120.29) - open/close 47716 (time 1746131152.26 total 787.59 last 87.23) - open/close 48676 (time 1746131162.28 total 797.61 last 95.83) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 807.41 seconds: 61.47 ops/second - unlinked 0 (time 1746131173 ; total 0 ; last 0) - unlinked 10000 (time 1746131246 ; total 73 ; last 73) - unlinked 20000 (time 1746131297 ; total 124 ; last 51) - unlinked 30000 (time 1746131350 ; total 177 ; last 53) - unlinked 40000 (time 1746131406 ; total 233 ; last 56) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 286 seconds: 173.538467 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules via unload_modules_local on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2