== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:18:44 (1712359124) start mds service on oleg140-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 728 (time 1712359164.06 total 10.44 last 69.70) - open/close 1625 (time 1712359175.00 total 21.38 last 82.01) - open/close 2417 (time 1712359185.71 total 32.09 last 73.94) - open/close 2926 (time 1712359195.89 total 42.27 last 50.00) - open/close 3575 (time 1712359206.52 total 52.90 last 61.08) - open/close 4195 (time 1712359217.08 total 63.46 last 58.69) - open/close 4701 (time 1712359227.52 total 73.91 last 48.45) - open/close 5183 (time 1712359238.00 total 84.38 last 46.02) - open/close 5695 (time 1712359248.67 total 95.05 last 48.00) - open/close 6290 (time 1712359259.50 total 105.88 last 54.94) - open/close 6865 (time 1712359270.32 total 116.70 last 53.11) - open/close 7423 (time 1712359280.32 total 126.71 last 55.80) - open/close 7921 (time 1712359291.13 total 137.51 last 46.08) - open/close 8441 (time 1712359301.13 total 147.51 last 51.99) - open/close 8949 (time 1712359311.89 total 158.27 last 47.23) - open/close 9472 (time 1712359321.95 total 168.33 last 51.99) - open/close 9927 (time 1712359332.59 total 178.97 last 42.76) - open/close 10409 (time 1712359342.59 total 188.97 last 48.18) - open/close 10914 (time 1712359352.64 total 199.02 last 50.28) - open/close 11465 (time 1712359363.05 total 209.43 last 52.93) - open/close 11871 (time 1712359373.99 total 220.37 last 37.11) - open/close 12309 (time 1712359384.22 total 230.60 last 42.81) - open/close 12795 (time 1712359394.92 total 241.30 last 45.43) - open/close 14636 (time 1712359404.92 total 251.30 last 184.01) - open/close 16329 (time 1712359414.92 total 261.30 last 169.27) - open/close 17776 (time 1712359424.93 total 271.31 last 144.65) - open/close 19455 (time 1712359434.93 total 281.31 last 167.85) - open/close 20000 (time 1712359438.78 total 285.16 last 141.43) - open/close 21262 (time 1712359448.79 total 295.17 last 126.18) - open/close 22305 (time 1712359458.80 total 305.18 last 104.19) - open/close 23631 (time 1712359468.80 total 315.18 last 132.51) - open/close 24693 (time 1712359478.80 total 325.19 last 106.18) - open/close 25817 (time 1712359488.81 total 335.19 last 112.38) - open/close 26824 (time 1712359498.82 total 345.20 last 100.58) - open/close 27814 (time 1712359508.83 total 355.21 last 98.86) - open/close 28637 (time 1712359518.84 total 365.22 last 82.25) - open/close 29451 (time 1712359528.84 total 375.22 last 81.38) - open/close 30000 (time 1712359535.78 total 382.16 last 79.12) - open/close 30785 (time 1712359545.78 total 392.16 last 78.49) - open/close 31582 (time 1712359555.78 total 402.17 last 79.68) - open/close 32382 (time 1712359565.78 total 412.17 last 79.99) - open/close 33122 (time 1712359575.79 total 422.18 last 73.93) - open/close 33912 (time 1712359585.80 total 432.18 last 78.93) - open/close 34609 (time 1712359595.81 total 442.19 last 69.67) - open/close 35357 (time 1712359605.82 total 452.20 last 74.72) - open/close 36191 (time 1712359615.82 total 462.20 last 83.36) - open/close 36976 (time 1712359625.83 total 472.21 last 78.46) - open/close 37662 (time 1712359635.84 total 482.22 last 68.54) - open/close 38280 (time 1712359645.85 total 492.23 last 61.72) - open/close 38875 (time 1712359655.85 total 502.23 last 59.49) - open/close 39574 (time 1712359665.86 total 512.24 last 69.82) - open/close 40000 (time 1712359673.80 total 520.18 last 53.66) - open/close 40612 (time 1712359683.81 total 530.19 last 61.17) - open/close 41120 (time 1712359693.81 total 540.19 last 50.78) - open/close 41614 (time 1712359703.83 total 550.21 last 49.32) - open/close 42097 (time 1712359713.84 total 560.22 last 48.23) - open/close 42579 (time 1712359723.85 total 570.23 last 48.15) - open/close 43138 (time 1712359733.86 total 580.24 last 55.86) - open/close 43707 (time 1712359743.87 total 590.25 last 56.86) - open/close 44296 (time 1712359753.87 total 600.25 last 58.90) - open/close 44845 (time 1712359763.88 total 610.26 last 54.84) - open/close 45391 (time 1712359773.89 total 620.27 last 54.54) - open/close 45932 (time 1712359783.90 total 630.28 last 54.05) - open/close 46431 (time 1712359793.90 total 640.28 last 49.89) - open/close 46945 (time 1712359803.91 total 650.29 last 51.35) - open/close 47429 (time 1712359813.92 total 660.30 last 48.35) - open/close 47909 (time 1712359823.93 total 670.31 last 47.95) - open/close 48408 (time 1712359833.94 total 680.32 last 49.88) - open/close 48941 (time 1712359843.95 total 690.33 last 53.21) - open/close 49438 (time 1712359853.97 total 700.35 last 49.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 704.28 seconds: 70.47 ops/second - unlinked 0 (time 1712359858 ; total 0 ; last 0) - unlinked 10000 (time 1712360038 ; total 180 ; last 180) - unlinked 20000 (time 1712360160 ; total 302 ; last 122) - unlinked 30000 (time 1712360238 ; total 380 ; last 78) - unlinked 40000 (time 1712360284 ; total 426 ; last 46) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 446 seconds: 111.282509 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2