== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:10:26 (1746148226) start mds service on oleg213-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg213-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1746148266.99 total 10.56 last 17.99) - open/close 381 (time 1746148277.35 total 20.93 last 18.43) - open/close 507 (time 1746148287.35 total 30.93 last 12.60) - open/close 753 (time 1746148298.25 total 41.82 last 22.59) - open/close 920 (time 1746148308.61 total 52.19 last 16.11) - open/close 1162 (time 1746148318.61 total 62.19 last 24.20) - open/close 1378 (time 1746148328.62 total 72.19 last 21.60) - open/close 1609 (time 1746148339.42 total 82.99 last 21.39) - open/close 1768 (time 1746148349.59 total 93.17 last 15.63) - open/close 1926 (time 1746148359.69 total 103.26 last 15.65) - open/close 2115 (time 1746148370.25 total 113.83 last 17.89) - open/close 2262 (time 1746148380.41 total 123.98 last 14.48) - open/close 2436 (time 1746148390.70 total 134.28 last 16.90) - open/close 2589 (time 1746148400.72 total 144.30 last 15.27) - open/close 2822 (time 1746148411.58 total 155.15 last 21.46) - open/close 3074 (time 1746148421.59 total 165.17 last 25.16) - open/close 3400 (time 1746148432.43 total 176.00 last 30.09) - open/close 3576 (time 1746148442.47 total 186.04 last 17.53) - open/close 3776 (time 1746148453.14 total 196.71 last 18.75) - open/close 3972 (time 1746148463.14 total 206.71 last 19.60) - open/close 4231 (time 1746148473.71 total 217.29 last 24.49) - open/close 4347 (time 1746148483.94 total 227.51 last 11.35) - open/close 4563 (time 1746148493.94 total 237.51 last 21.60) - open/close 4759 (time 1746148504.09 total 247.67 last 19.30) - open/close 5016 (time 1746148514.56 total 258.14 last 24.55) - open/close 5290 (time 1746148525.22 total 268.79 last 25.71) - open/close 5522 (time 1746148537.57 total 281.14 last 18.79) - open/close 5749 (time 1746148547.77 total 291.35 last 22.24) - open/close 6083 (time 1746148558.59 total 302.17 last 30.87) - open/close 6389 (time 1746148569.31 total 312.89 last 28.55) - open/close 6636 (time 1746148580.15 total 323.72 last 22.79) - open/close 6897 (time 1746148590.39 total 333.97 last 25.48) - open/close 7108 (time 1746148601.20 total 344.77 last 19.53) - open/close 7314 (time 1746148611.26 total 354.83 last 20.47) - open/close 7542 (time 1746148621.94 total 365.52 last 21.34) - open/close 7720 (time 1746148631.95 total 375.52 last 17.79) - open/close 7926 (time 1746148642.11 total 385.68 last 20.28) - open/close 8153 (time 1746148652.18 total 395.75 last 22.54) - open/close 8348 (time 1746148664.41 total 407.98 last 15.94) - open/close 8590 (time 1746148674.46 total 418.03 last 24.08) - open/close 9511 (time 1746148684.46 total 428.04 last 92.03) - open/close 10000 (time 1746148688.18 total 431.75 last 131.70) - open/close 11314 (time 1746148698.18 total 441.75 last 131.39) - open/close 12612 (time 1746148708.18 total 451.76 last 129.75) - open/close 14040 (time 1746148718.19 total 461.76 last 142.73) - open/close 15577 (time 1746148728.19 total 471.76 last 153.67) - open/close 16866 (time 1746148738.19 total 481.77 last 128.82) - open/close 18060 (time 1746148748.20 total 491.78 last 119.33) - open/close 19176 (time 1746148758.21 total 501.78 last 111.51) - open/close 20000 (time 1746148765.52 total 509.10 last 112.67) - open/close 21106 (time 1746148775.52 total 519.10 last 110.59) - open/close 22351 (time 1746148785.53 total 529.10 last 124.48) - open/close 23589 (time 1746148795.53 total 539.11 last 123.74) - open/close 24951 (time 1746148805.54 total 549.11 last 136.10) - open/close 26206 (time 1746148815.54 total 559.12 last 125.47) - open/close 27592 (time 1746148825.55 total 569.12 last 138.50) - open/close 28873 (time 1746148835.56 total 579.13 last 127.97) - open/close 30000 (time 1746148844.93 total 588.51 last 120.20) - open/close 31239 (time 1746148854.94 total 598.51 last 123.83) - open/close 32445 (time 1746148864.95 total 608.52 last 120.49) - open/close 33851 (time 1746148874.96 total 618.54 last 140.44) - open/close 35068 (time 1746148884.96 total 628.54 last 121.66) - open/close 36374 (time 1746148894.97 total 638.54 last 130.55) - open/close 37955 (time 1746148904.97 total 648.54 last 158.08) - open/close 39263 (time 1746148914.97 total 658.55 last 130.75) - open/close 40000 (time 1746148921.30 total 664.87 last 116.47) - open/close 41117 (time 1746148931.31 total 674.89 last 111.56) - open/close 42331 (time 1746148941.32 total 684.89 last 121.32) - open/close 43566 (time 1746148951.32 total 694.90 last 123.47) - open/close 45195 (time 1746148961.32 total 704.90 last 162.88) - open/close 46657 (time 1746148971.32 total 714.90 last 146.20) - open/close 48001 (time 1746148981.34 total 724.91 last 134.22) - open/close 49232 (time 1746148991.36 total 734.93 last 122.83) total: 50000 open/close in 740.77 seconds: 67.50 ops/second - unlinked 0 (time 1746148998 ; total 0 ; last 0) - unlinked 10000 (time 1746149054 ; total 56 ; last 56) - unlinked 20000 (time 1746149106 ; total 108 ; last 52) - unlinked 30000 (time 1746149156 ; total 158 ; last 50) - unlinked 40000 (time 1746149209 ; total 211 ; last 53) total: 50000 unlinks in 256 seconds: 195.312500 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2