== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:13:10 (1746130390) start mds service on oleg224-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 189 (time 1746130449.77 total 10.01 last 18.88) - open/close 366 (time 1746130460.58 total 20.83 last 16.36) - open/close 625 (time 1746130470.77 total 31.02 last 25.42) - open/close 805 (time 1746130481.44 total 41.68 last 16.88) - open/close 991 (time 1746130492.24 total 52.49 last 17.22) - open/close 1257 (time 1746130502.26 total 62.50 last 26.56) - open/close 1433 (time 1746130512.26 total 72.50 last 17.60) - open/close 1609 (time 1746130522.97 total 83.21 last 16.44) - open/close 1750 (time 1746130533.03 total 93.27 last 14.01) - open/close 1955 (time 1746130543.56 total 103.80 last 19.47) - open/close 2120 (time 1746130554.54 total 114.78 last 15.03) - open/close 2347 (time 1746130564.54 total 124.78 last 22.70) - open/close 2616 (time 1746130575.16 total 135.40 last 25.34) - open/close 2863 (time 1746130585.64 total 145.88 last 23.56) - open/close 3080 (time 1746130596.11 total 156.36 last 20.72) - open/close 3348 (time 1746130606.42 total 166.66 last 26.01) - open/close 3566 (time 1746130617.11 total 177.35 last 20.38) - open/close 3866 (time 1746130627.90 total 188.15 last 27.80) - open/close 4092 (time 1746130638.82 total 199.06 last 20.71) - open/close 4392 (time 1746130649.52 total 209.76 last 28.03) - open/close 4561 (time 1746130660.13 total 220.37 last 15.93) - open/close 4798 (time 1746130670.32 total 230.56 last 23.25) - open/close 5044 (time 1746130681.30 total 241.55 last 22.40) - open/close 5385 (time 1746130691.30 total 251.55 last 34.10) - open/close 5634 (time 1746130702.13 total 262.38 last 22.99) - open/close 5881 (time 1746130712.14 total 272.38 last 24.69) - open/close 6131 (time 1746130724.61 total 284.85 last 20.04) - open/close 6391 (time 1746130734.89 total 295.13 last 25.30) - open/close 6596 (time 1746130745.43 total 305.67 last 19.44) - open/close 6900 (time 1746130758.20 total 318.45 last 23.80) - open/close 7153 (time 1746130768.52 total 328.76 last 24.53) - open/close 7374 (time 1746130779.54 total 339.78 last 20.05) - open/close 7591 (time 1746130789.56 total 349.80 last 21.67) - open/close 7819 (time 1746130800.73 total 360.97 last 20.40) - open/close 8071 (time 1746130810.73 total 370.98 last 25.19) - open/close 8334 (time 1746130821.65 total 381.89 last 24.11) - open/close 8575 (time 1746130831.65 total 391.89 last 24.09) - open/close 8781 (time 1746130842.21 total 402.45 last 19.51) - open/close 9047 (time 1746130852.56 total 412.80 last 25.70) - open/close 9301 (time 1746130863.24 total 423.49 last 23.76) - open/close 9567 (time 1746130873.26 total 433.50 last 26.57) - open/close 9754 (time 1746130883.92 total 444.16 last 17.54) - open/close 10000 (time 1746130890.76 total 451.01 last 35.94) - open/close 11539 (time 1746130900.76 total 461.01 last 153.88) - open/close 13178 (time 1746130910.77 total 471.01 last 163.78) - open/close 14994 (time 1746130920.78 total 481.02 last 181.53) - open/close 16443 (time 1746130930.78 total 491.02 last 144.85) - open/close 18531 (time 1746130940.79 total 501.03 last 208.67) - open/close 20000 (time 1746130949.10 total 509.34 last 176.74) - open/close 21901 (time 1746130959.10 total 519.34 last 190.09) - open/close 23778 (time 1746130969.10 total 529.34 last 187.66) - open/close 25672 (time 1746130979.10 total 539.34 last 189.39) - open/close 27519 (time 1746130989.11 total 549.35 last 184.55) - open/close 29311 (time 1746130999.11 total 559.35 last 179.20) - open/close 30000 (time 1746131002.44 total 562.68 last 207.02) - open/close 31459 (time 1746131012.44 total 572.68 last 145.85) - open/close 32776 (time 1746131022.44 total 582.69 last 131.66) - open/close 34256 (time 1746131032.45 total 592.69 last 147.91) - open/close 35607 (time 1746131042.45 total 602.69 last 135.07) - open/close 37107 (time 1746131052.46 total 612.70 last 149.91) - open/close 38587 (time 1746131062.46 total 622.70 last 147.97) - open/close 39942 (time 1746131072.46 total 632.70 last 135.49) - open/close 41260 (time 1746131082.46 total 642.71 last 131.76) - open/close 42859 (time 1746131092.47 total 652.71 last 159.84) - open/close 45164 (time 1746131102.47 total 662.71 last 230.43) - open/close 46634 (time 1746131112.47 total 672.72 last 146.96) - open/close 47759 (time 1746131122.48 total 682.72 last 112.47) - open/close 48868 (time 1746131132.48 total 692.72 last 110.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 699.26 seconds: 70.98 ops/second - unlinked 0 (time 1746131140 ; total 0 ; last 0) - unlinked 10000 (time 1746131197 ; total 57 ; last 57) - unlinked 20000 (time 1746131244 ; total 104 ; last 47) - unlinked 30000 (time 1746131292 ; total 152 ; last 48) - unlinked 40000 (time 1746131337 ; total 197 ; last 45) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 234 seconds: 212.102570 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2