== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:24:03 (1746127443) start mds service on oleg406-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 136 (time 1746127483.35 total 10.54 last 12.90) - open/close 326 (time 1746127493.82 total 21.02 last 18.14) - open/close 474 (time 1746127503.82 total 31.02 last 14.80) - open/close 766 (time 1746127513.83 total 41.03 last 29.18) - open/close 972 (time 1746127524.00 total 51.20 last 20.25) - open/close 1117 (time 1746127534.24 total 61.44 last 14.16) - open/close 1249 (time 1746127544.42 total 71.62 last 12.97) - open/close 1410 (time 1746127555.35 total 82.55 last 14.72) - open/close 1545 (time 1746127565.95 total 93.15 last 12.74) - open/close 1711 (time 1746127576.21 total 103.41 last 16.19) - open/close 1856 (time 1746127586.26 total 113.46 last 14.43) - open/close 2155 (time 1746127597.01 total 124.21 last 27.80) - open/close 2326 (time 1746127607.99 total 135.19 last 15.58) - open/close 2498 (time 1746127617.99 total 145.19 last 17.19) - open/close 2648 (time 1746127628.72 total 155.92 last 13.98) - open/close 2739 (time 1746127641.23 total 168.43 last 7.28) - open/close 2968 (time 1746127652.07 total 179.27 last 21.12) - open/close 3216 (time 1746127662.33 total 189.53 last 24.18) - open/close 3371 (time 1746127672.34 total 199.54 last 15.48) - open/close 3586 (time 1746127682.63 total 209.83 last 20.90) - open/close 3685 (time 1746127693.01 total 220.21 last 9.53) - open/close 3837 (time 1746127706.01 total 233.21 last 11.69) - open/close 4116 (time 1746127717.02 total 244.22 last 25.36) - open/close 4245 (time 1746127728.44 total 255.64 last 11.30) - open/close 4450 (time 1746127738.82 total 266.02 last 19.74) - open/close 4528 (time 1746127749.79 total 276.99 last 7.11) - open/close 4780 (time 1746127759.79 total 286.99 last 25.19) - open/close 5032 (time 1746127770.00 total 297.20 last 24.69) - open/close 5169 (time 1746127780.78 total 307.98 last 12.71) - open/close 5384 (time 1746127790.98 total 318.18 last 21.06) - open/close 5609 (time 1746127801.54 total 328.74 last 21.32) - open/close 5801 (time 1746127812.00 total 339.20 last 18.35) - open/close 5903 (time 1746127822.64 total 349.84 last 9.59) - open/close 6040 (time 1746127833.40 total 360.60 last 12.73) - open/close 6171 (time 1746127844.08 total 371.28 last 12.26) - open/close 6377 (time 1746127854.37 total 381.57 last 20.02) - open/close 6549 (time 1746127865.34 total 392.54 last 15.68) - open/close 6659 (time 1746127876.75 total 403.95 last 9.64) - open/close 6840 (time 1746127886.87 total 414.07 last 17.89) - open/close 6921 (time 1746127896.99 total 424.19 last 8.00) - open/close 7047 (time 1746127907.76 total 434.96 last 11.70) - open/close 7177 (time 1746127919.52 total 446.72 last 11.06) - open/close 7378 (time 1746127930.31 total 457.51 last 18.62) - open/close 7505 (time 1746127941.36 total 468.56 last 11.49) - open/close 7574 (time 1746127951.37 total 478.57 last 6.90) - open/close 7701 (time 1746127962.00 total 489.20 last 11.94) - open/close 7826 (time 1746127972.82 total 500.02 last 11.55) - open/close 7961 (time 1746127982.83 total 510.03 last 13.49) - open/close 8103 (time 1746127992.99 total 520.19 last 13.97) - open/close 8227 (time 1746128003.90 total 531.10 last 11.37) - open/close 9242 (time 1746128013.91 total 541.11 last 101.43) - open/close 10000 (time 1746128020.68 total 547.88 last 111.91) - open/close 11109 (time 1746128030.68 total 557.88 last 110.87) - open/close 12311 (time 1746128040.68 total 567.88 last 120.18) - open/close 13547 (time 1746128050.69 total 577.89 last 123.56) - open/close 14644 (time 1746128060.69 total 587.89 last 109.62) - open/close 15694 (time 1746128070.70 total 597.90 last 104.89) - open/close 16779 (time 1746128080.71 total 607.91 last 108.47) - open/close 18047 (time 1746128090.71 total 617.91 last 126.78) - open/close 19366 (time 1746128100.71 total 627.91 last 131.83) - open/close 20000 (time 1746128105.45 total 632.64 last 133.99) - open/close 21292 (time 1746128115.45 total 642.65 last 129.19) - open/close 22581 (time 1746128125.45 total 652.65 last 128.89) - open/close 23757 (time 1746128135.45 total 662.65 last 117.57) - open/close 24836 (time 1746128145.45 total 672.65 last 107.85) - open/close 25857 (time 1746128155.46 total 682.66 last 102.05) - open/close 26624 (time 1746128165.47 total 692.66 last 76.66) - open/close 27499 (time 1746128175.47 total 702.67 last 87.46) - open/close 28390 (time 1746128185.47 total 712.67 last 89.06) - open/close 29428 (time 1746128195.48 total 722.68 last 103.74) - open/close 30000 (time 1746128201.71 total 728.91 last 91.84) - open/close 30934 (time 1746128211.71 total 738.91 last 93.37) - open/close 31814 (time 1746128221.71 total 748.91 last 88.00) - open/close 32793 (time 1746128231.72 total 758.92 last 97.84) - open/close 33731 (time 1746128241.72 total 768.92 last 93.74) - open/close 34619 (time 1746128251.74 total 778.94 last 88.62) - open/close 35507 (time 1746128261.76 total 788.96 last 88.67) - open/close 36397 (time 1746128271.77 total 798.97 last 88.89) - open/close 37392 (time 1746128281.78 total 808.98 last 99.37) - open/close 38408 (time 1746128291.79 total 818.99 last 101.53) - open/close 39562 (time 1746128301.79 total 828.99 last 115.40) - open/close 40000 (time 1746128305.61 total 832.81 last 114.60) - open/close 41152 (time 1746128315.62 total 842.81 last 115.18) - open/close 42346 (time 1746128325.62 total 852.82 last 119.38) - open/close 43483 (time 1746128335.63 total 862.83 last 113.58) - open/close 44648 (time 1746128345.63 total 872.83 last 116.50) - open/close 45847 (time 1746128355.63 total 882.83 last 119.84) - open/close 47005 (time 1746128365.64 total 892.84 last 115.73) - open/close 48103 (time 1746128375.64 total 902.84 last 109.74) - open/close 49293 (time 1746128385.65 total 912.85 last 118.94) total: 50000 open/close in 919.32 seconds: 54.39 ops/second - unlinked 0 (time 1746128394 ; total 0 ; last 0) - unlinked 10000 (time 1746128470 ; total 76 ; last 76) - unlinked 20000 (time 1746128527 ; total 133 ; last 57) - unlinked 30000 (time 1746128582 ; total 188 ; last 55) - unlinked 40000 (time 1746128642 ; total 248 ; last 60) total: 50000 unlinks in 312 seconds: 160.256409 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2