== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:33:52 (1715207632) start mds service on oleg334-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg334-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock oleg334-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 687 (time 1715207676.91 total 10.65 last 64.49) - open/close 1114 (time 1715207687.38 total 21.12 last 40.78) - open/close 1629 (time 1715207698.14 total 31.89 last 47.84) - open/close 2132 (time 1715207708.64 total 42.39 last 47.91) - open/close 2684 (time 1715207718.64 total 52.39 last 55.19) - open/close 2946 (time 1715207728.65 total 62.39 last 26.19) - open/close 3376 (time 1715207739.28 total 73.03 last 40.43) - open/close 3790 (time 1715207749.85 total 83.60 last 39.18) - open/close 4005 (time 1715207760.70 total 94.44 last 19.82) - open/close 4293 (time 1715207773.70 total 107.45 last 22.15) - open/close 4875 (time 1715207784.56 total 118.31 last 53.59) - open/close 5426 (time 1715207794.69 total 128.44 last 54.40) - open/close 5837 (time 1715207804.87 total 138.61 last 40.40) - open/close 6331 (time 1715207815.60 total 149.34 last 46.04) - open/close 6756 (time 1715207825.92 total 159.66 last 41.17) - open/close 7200 (time 1715207836.39 total 170.14 last 42.38) - open/close 7694 (time 1715207847.04 total 180.79 last 46.40) - open/close 8248 (time 1715207857.87 total 191.61 last 51.17) - open/close 8879 (time 1715207868.12 total 201.87 last 61.55) - open/close 9409 (time 1715207879.03 total 212.78 last 48.58) - open/close 9856 (time 1715207889.30 total 223.04 last 43.53) - open/close 10000 (time 1715207893.89 total 227.63 last 31.37) - open/close 10483 (time 1715207904.64 total 238.39 last 44.92) - open/close 10897 (time 1715207915.46 total 249.21 last 38.26) - open/close 11297 (time 1715207926.34 total 260.08 last 36.79) - open/close 11771 (time 1715207936.86 total 270.61 last 45.02) - open/close 12036 (time 1715207946.86 total 280.61 last 26.50) - open/close 12568 (time 1715207956.87 total 290.61 last 53.19) - open/close 12879 (time 1715207967.11 total 300.86 last 30.36) - open/close 14766 (time 1715207977.11 total 310.86 last 188.65) - open/close 17689 (time 1715207987.11 total 320.86 last 292.23) - open/close 20000 (time 1715207995.21 total 328.96 last 285.38) - open/close 23252 (time 1715208005.21 total 338.96 last 325.19) - open/close 25948 (time 1715208015.21 total 348.96 last 269.59) - open/close 28523 (time 1715208025.21 total 358.96 last 257.47) - open/close 30000 (time 1715208030.80 total 364.54 last 264.56) - open/close 32833 (time 1715208040.80 total 374.54 last 283.30) - open/close 35829 (time 1715208050.80 total 384.55 last 299.48) - open/close 40000 (time 1715208059.81 total 393.55 last 463.23) - open/close 42816 (time 1715208069.81 total 403.55 last 281.52) - open/close 45379 (time 1715208079.81 total 413.56 last 256.20) - open/close 47971 (time 1715208089.81 total 423.56 last 259.13) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 429.98 seconds: 115.43 ops/second - unlinked 0 (time 1715208097 ; total 0 ; last 0) - unlinked 10000 (time 1715208120 ; total 23 ; last 23) - unlinked 20000 (time 1715208141 ; total 44 ; last 21) - unlinked 30000 (time 1715208163 ; total 66 ; last 22) - unlinked 40000 (time 1715208185 ; total 88 ; last 22) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 109 seconds: 455.339447 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server unloading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2